Soil data logger telemetry

I have finally found time to build a simple website for my soil data logger with telemetry. The system works as the following:

  1. The data logger consists of a raspberry pi and my SDI-12 USB adapter with a Decagon 5TM soil sensor
  2. The data logger runs the open-source datalogger code I wrote in Python to first get parameters from the user (COM port, SDI-12 address, delay etc.), and then collect data, save to a local .CSV file, and then send the same data to sparkfun’s phant server.
  3. I constructed a web interface to plot the data using Google Charts and download .CSV version from sparkfun’s phant server.

Here is a screen shot:

soil logger webpage

I’ve uploaded the webpage to a server with a link below. The sensor is apparently NOT buried in soil so I can easily take the setup and set it up in different places to test its stability.

Link to the website: Link

Update SDI-12 USB adapter firmware

I have just released an update to the SDI-12 USB adapter. This update makes the adapter recover from errors in communication so it won’t hang your data logging process in case a sensor is broken while you are logging.

To make the firmware update easy, I wrote a Python script to use avrdude.exe to load the firmware to the adapter. You will run the script just like the data logging script or config script:

  1. Install Python 3.5 with Pyserial 3.0 (instruction in the manual).
  2. Unzip the content of the firmware update package in a folder.
  3. Check the properties of avrdude.exe to make sure that it is not blocked from running.
  4. Run the script SDI_12_firmware_update.py in Python environment IDLE.
  5. Select the adapter’s serial port from a list.
  6. Pick the firmware file, usually SDI_12_translator_v1_x.hex (must be stored in the same folder as the script), then wait for it to complete, which takes less than 30 seconds.

Here is the output from Python:

sdi-12 USB adapter firmware update

I had the adapter on COM10 (I have connected lots of Arduino type devices to this computer thus the high COM number).

In the screenshot I also had Blink.hex, the Arduino’s famous Blink code to see if the script was working. You will only see the actual firmware file. Once updated, try connect to it using a terminal program and send zI! (zee-EYE!) and you will see the response has version 1.2 in it.

At the moment it only runs on windows PC. I plan on finding solutions for GNU/Linux and Mac OSX. I’ll update this post once that is done.

(Update:) I have added Raspberry Pi to the list of supported OS for firmware update. I will test Debian Linux 64-bit once I have time.

Back up and clone raspberry pi

In this post, I will explain how to back up and restore or clone a raspberry pi. I am assuming that you have installed Raspbian or Ubuntu mate on your raspberry pi. Since a raspberry pi runs its OS entirely on an sd card, it is easy to clone your system so you can pass along to a friend or create your special distro for others to use. For example, a company that makes compact optical spectrometer, Ocean Optics, has created a spectrometer program that runs on raspberry pi and has its own web interface. You just have to download their raspberry pi image and put it on your card. No need to do multiple build from source and apt-get etc. to get what you need.

If you are already a Linux guru, this post is NOT for you. I am assuming that you don’t have a Linux machine other than raspberry pi or you are still learning Linux.

The “easy” way:

Raspberry pi foundation has raw images of their Raspbian and NOOB on their website. On a windows system, all you need is win32 disk imager. It reads in the image and writes it on an SD card. You can save the image it reads and keep it as a backup or use it to clone a system.

The catch:

Since win32 disk imager only reads and writes in raw format, it actually doesn’t know what it is reading/writing. Say if you want to clone your raspberry pi 2B running Raspbian Jessie on an 8GB Kingston microSD card onto an 8GB SanDisk microSD card, you simply can’t. Although both cards claim to be 8GB, the SanDisk card is about 20MB smaller than the Kingston. Not understanding what it reads and writes, win32 disk imager is unable to shrink even one byte of what it reads (the whole 8GB from Kingston) to try to fit it onto a smaller card. This has pushed people into buying larger cards, such as 16GB. Then when they try to clone a system again, they run into the same issue unless they have bought a few identical SD cards as clone targets. Even that has issues (read the end of the post for details). So they have to step up to 32GB!!!

The solution:

We need a disk/partition reader and writer that understands what it is processing and is able to resize the partitions so they fit. If you are not hosting a lot of large files, you should be OK with an 8GB card or more than OK with a 16GB card. I’ve looked around for quite some time and found my solution: Paragon backup and Recovery 14 Free edition (Home edition for $39,99 has more features not useful for us). It is only for windows so those mac users will either need to become Linux gurus or shell out a small amount of money to get a win 10 netbook.

What the program does is that it is able to back up a whole disk or SD card and restore it on disk or SD card of different size. The reason is it understands most common file systems such as FAT, NTFS, HFS, and EXT. Besides, it also automatically compresses the backup so it doesn’t take more space than it should on your PC.

How to do it:

First you make a backup of your raspberry pi card using back up to VD. It is a virtual hard drive. This backs up both BOOT partition (FAT partition) and your Linux partition (ext4).

backup_to _vd

Next, insert your new SD card (you need to quit the program and restart it). Select restore from VD. It will ask you whether to resize partition and you can choose yes. This way you can even squeeze your clone from a larger card (32GB) to a smaller card (8GB) if your Linux partition has a lot of space.

Another neat trick:

What else you can do is to resize BOOT partition. Although BOOT partition has not much use for most of us, it IS the only partition that windows will recognize. If you are using your pi as a data logger or need to copy files from and to it (without the hassle of SSH and network setting), then the best way to get data is if it is written to the BOOT partition. You turn off raspberry pi and put its SD card on your windows machine. Immediately you have access to BOOT partition. You can copy your data out, you can change some config files of your data logger etc. It is useful to have a large BOOT partition. To do this, you will have to restore one partition at a time. Restore BOOT first. You will be prompted to select the size of the restored partition. Make your selection, say 500MB. Then when the restore is over, use the rest of the space on card to restore your Linux partition (make sure you leave enough space for Linux partition).

Why buying identical cards may not solve clone/restore issue?

Well, say you purchased 10 Kingston 8GB microSD cards, they are all the same size, correct? Wrong! I learned it the hard way. I had a friend that sent me an image he saved from such a card. I have identical cards. But, when I was telling win32 disk imager to write the image on my card, it complained: the target card doesn’t have enough space! Guess what, the target card is one sector less than the image file has?! But how can this happen? It’s simple now that I got my answer: some sectors on the SD card may have become bad either at factory or post purchase. The SD card controller (inside the card there is a controller) has disabled those sectors, making not all “identical” cards identical.

 

How Arduino bootloader works

atmega328P_bootloader

Everyone that uses Arduino can tell you how easy it is to get projects going once they have an Arduino board in hand. Just load up the blink code and press upload. In a matter of seconds, your Arduino LED is blinking ever so confidently. But do you stop and wonder how Arduino receives the blink code? Enter Arduino bootloader. Every Arduino has a bootloader, a small program that is always stored on the Arduino to update the code in the Arduino. It only runs once per reset. It looks for new code to be loaded to the Arduino before starting the existing code.

The bootloader works like this:

On the PC side, to trigger a reset on the RESET pin, the PC (avrdude.exe or GNU/Linux equivalent) opens the serial port to Arduino when your press upload and the code is ready to upload. This causes the Data Transmit Ready (DTR) line of the USB/TTL chip to go LOW. The Arduino board has a capacitor charging circuit that uses this LOW (charging the capacitor) signal to momentarily pull down the RESET line of the ATMEGA328P chip before returning it to HIGH (capacitor charging completes). So Arduino resets each time its serial port is opened.

Upon reset, Arduino enters the bootloader.

The bootloader looks at the source that caused the reset. There are several sources that can cause a reset. If the reset was caused by the RESET pin, then it waits for one second for the PC to send in commands. When it receives valid commands, it will start accepting new Arduino code in HEX format and erase the existing code to load new one. If it doesn’t receive valid commands, it times out after one second and triggers a Watch Dog Timer (WDT) reset.

Once the bootloader runs again it will look at the source of the reset. Once it determines that it was the WDT reset, it immediately jumps to the first line of the actual code. This way if you power up Arduino, it will be able to immediately run your code, instead of waiting in the bootloader to time out 1 second. It’s pretty smart!

The optiboot bootloader is the most recent bootloader used by Arduino UNO, nano, and other boards based on ATMEGA328P and 1284P. It accepts commands or uploads code at 115,200, unlike the previous bootloader that accepts commands or uploads code at 57,600. A number of people have modified this standard bootloader to fit their own needs, such as slower upload speed for a bare-bone system with no crystals and runs at say 100KHz on internal RC oscillator to preserve battery. Others have made Ethernet upload possible. I’ve personally made a bootloader that will refuse to upload code unless a certain EEPROM byte is set to a certain value. This way if the device is a data logger and to be used by a student, the teacher won’t be worried that the student would get “smart” and erase the code. But when the teacher wants to change the code he/she can enter a password in the existing data logger program and unlock the device for upload again.

If you want to deploy a project, such as an artistic installation or a data logger, you can make a bare-bone system without the whole Arduino UNO board. Then you will need an ATMEGA328P chip with preloaded bootloader like this one, and some extra components.

Anyway, if your code doesn’t need to be changed any further, you can decide to get rid of the bootloader altogether. On an Arduino UNO, this will free up 512 bytes of FLASH and the end user of your device won’t have the danger of erasing the code. To load code without a bootloader, you can use Nick Gammon’s HEX uploader. All you need is another Arduino UNO and an SD card (shield format is preferred) and some jumper wires.

Dropbox Python API

Assume that you are building a data logger and need to send your data from your logger to you, one option that will not cost you money or much programming time is to send your data file to your Dropbox. This requires some minimal setup. Once set up, you can proceed to add the file upload feature to your code. I am using Python to do the job. It is quick and easy. Plus, you can port your code to any operating system, such as PC running windows, GNU/Linux, Mac OSX, or Raspberry Pi. I am assuming that you have Python 3.X.

There are two versions of Dropbox Python API, V1 and V2. V2 came out around the end of 2015 and only has minimal tutorial on Dropbox.com. Nevertheless, I will use V2. Installing the API is a snap:

On Windows:

Start a command prompt and enter:

pip install dropbox

On Raspberry Pi (Raspbian Jessie) or Debian PC

sudo pip3 install dropbox

If you still have Raspbian Wheezy, pip3 may tell you that there is a newer version of pip. Don’t attempt to upgrade your pip or pip3. It will break.

Here is a short Python script to upload and download files:

import dropbox
file_name='test_image.jpg'
dropbox_path='/'
dbx=dropbox.Dropbox('Your access token')
with open(file_name, 'rb') as f:
    dbx.files_upload(f,dropbox_path+file_name,mute=True)

dbx.files_download_to_file('Copy of '+file_name,dropbox_path+file_name)

This was easy. There are only three functions that I used, first an authentication, then upload, followed by download. Your access token is generated by Dropbox. This post has the details.
The function dropbox.Dropbox() returns an object. You can use this object to upload or download files and more.
The files_upload() function does the upload. You need to first open the file on your local computer with open(). I am assuming the file is stored in the same folder as your Python script. This returns a file handler f. Then pass f to the files_upload(). This is the first argument. The next argument is the path you want the file to be uploaded to dropbox. It has to start with ‘/’, then the file name (including additional path). You don’t have to preserve file name. The third argument is useful. When you sent mute to True, you won’t get notification for the file upload. If you are making a camera trap, you don’t want your PC flooded with Dropbox notifications just because a bunny decides to visit your backyard.
The files_download_to_file requires first a local computer name, then the Dropbox file name. Again you don’t have to preserve file name. It’s useful to first check if the file exists on your local computer and decide what to do (overwrite, or add prefix/postfix to new file).
This is it! How you use it is up to you.

If you are interested, you can explore the rest of the API, such as creating folders, moving files, listing folders, etc. by reading the documentation (alert: document is very dry)

https://www.dropbox.com/developers/documentation/python

In my next post, I will make a simple Raspberry Pi camera logger to activate only during several intervals of time of the day.

Connect to Dropbox

If you are making a data logger, or camera trap, this is definitely good news. With the right approach, you can sit in your home and data/pictures just flow to your desktop. All you need is internet connection and some Dropbox programming. With a raspberry pi or a pc, you can have your data logger or camera logger automatically upload data to your Dropbox. On a windows PC, this is nothing more than saving your data file to a folder inside your Dropbox folder. Still, constantly writing to a Drpobox file is not the best way to make use of it. You won’t be able to have a meaningful history of the file since the your data logger updates the file too many times. Also, raspberry pi doesn’t have dropbox client. So it makes sense to programmatically upload your file to Dropbox using its API or application programming interface.

I am starting a series or posts to detail how to upload files to Dropbox from within your code. This post only explains how to set things up.

Using Python, it is extremely easy to upload files to dropbox. There is only one catch, you have to set up as an app developer. Here is their logic thus how to get set up:

You have some good ideas for an app. You want to store your app’s data files on dropbox so your user can access them on any device they are using. Good idea, but how to set it up?

First you need a Dropbox account yourself. Once you’re set, log on with your web browser. You’ll see three circles on the bottom left of your window:

dropbox dev-1

Clicking the dots brings up a menu. Select Developers:

dropbox dev-2

 

Click “Create app” to create a new app. I’ve already created a couple of similar apps. It looks like I’ve been trying this camera logger idea for one too many times😀

dropbox dev-3

Make sure that you select the right options. Select “App folder” access so all your actions are contained in a folder inside the App folder. It used to be called sandbox access. You may not want this app to access your entire Dropbox folder and accidentally delete files it is not intended to touch.

dropbox dev-4

Once done, you will be able to edit your project. Scroll down to find “Generated access token”. This will give you a very long alpha-numeric sequence that grants access to your Dropbox app folder, or the entire Dropbox if you chose “Full Dropbox”. Safeguard this string. If you need to ask questions about your code, remove/black out this string. Anyone with this string can do anything they want with your Dropbox app folder, no password needed. alternatively, you may try the App key and App secret path. That is a longer path than we need to take, just to show how things work.

dropbox dev-5

Now you are all set to develop this app, which is probably a program that uploads data/pictures to your dropbox. Read my next post about how to access Dropbox using Python. Python is one of the least preferred language I use but there are benefits of using it. It’s quick to see results. I am a big fan of Arduino. Unfortunately, it is almost impossible to upload to Dropbox using just an Arduino (except Arduino Yun).

SDI-12 USB adapter on Raspberry pi

I have successfully developed Python code to run the SDI-12 USB adapter data logger on Raspberry PI. It was quite a learning experience for me but I was able to boil down the steps into a tutorial on how to install the latest Python on Debian/Raspbian here.

Here are some screen shots of the data logger:

2016-02-03-171037_1680x1050_scrot

2016-02-05-100346_1680x1050_scrotI was using a Decagon 5TM soil temperature and humidity sensor (hanging free in air and sometimes touching my table).

I have translated the Tera Term scripts into Python so it would run on all operating systems, including Win 10, Linux (Debian), and Raspberry PI (Raspbian). I have not got my Mac fixed but don’t expect any issues.

Now you have two Python scripts:

The configuration script detects the SDI-12 sensor’s address, prints the name of the sensor, and you can change it to a different address.

The data logger script lists all available serial ports to let you choose the correct one, then asks for sensor address, total data points, delays between data points, and also what time (GMT or local) to use. You can expand the script to take data from multiple probes and upload data to a server. I’ll add more info on the server upload to sparkfun’s phant data server.

While developing the code, I also discovered that the serial port console was very useful. You can set up auto login and run the data logger script upon auto login. This way the data logger starts logging data once the RPI boots up and auto logs on. No need to log in or use VNC (not good if your logger is using 4G hotspot for internet). Just a command prompt is all that is needed. Next step is to add a config file so the data logger will no longer ask for user inputs when this file exists and takes parameters from the file. This config file will reside in the FAT partition so that the user can easily update the parameters without having to boot into raspberry pi. Just remove the sd card and change parameters on a PC.

Install python on raspberry pi or debian

I have just started learning the python programming language. It is very different than C/C++ that I’m used to. So to Arduino and Java folks it will take some getting used to. But the reason that I am learning is the promise of cross-platform portability and powerful libraries/modules. If you could send an HTTP request in a few lines, or imagine that you write and test a program on PC and run it on a raspberry pi with no/minimal modification, would you stop complaining about weird syntax or loose data types and just learn the darn language? Well, I did.

So in order to use Python, you need to install it. Since I am interested in using raspberry pi with an arduino, I need the best serial port support that I can find. My conclusion is to use Python 3.5.1 and pyserial 3.0.1 (the latest of both as of the blog post).

On a windows PC, this would be the easiest. Download the installers and install the program.

https://www.python.org/downloads/release/python-351/

https://pypi.python.org/pypi/pyserial

On a linux PC or raspberry pi, python is included but is a lower version. The tutorial helps you install python on linux machines and raspberry pi’s.

This tutorial is based on this tutorial and pieces of other tutorials and information online:

http://www.extellisys.com/articles/python-on-debian-wheezy

To install the latest version of python, you need to build it from source code.

  1. Download the source code and unzip it in a folder.

https://www.python.org/downloads/source/

  1. Update apt-get first

$ sudo apt-get update

  1. Install required tools to build Python from source. Debian/Raspbian Jessie will install libdb5.3-dev while Wheezy will install libdb5.1-dev. You need to install tk-dev to make Python’s IDE IDLE3.5 work

$ sudo apt-get install build-essential

$ sudo apt-get install tk-dev

$ sudo apt-get install libncurses5-dev libncursesw5-dev libreadline6-dev

$ sudo apt-get install libdb5.1-dev libgdbm-dev libsqlite3-dev libssl-dev

$ sudo apt-get install libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev

  1. Prepare for Pip (I didn’t do it but the tutorial I followed mentioned this, which seems a bit old)

$ mkdir -p ~/.pip/cache

$ echo ‘[global]’ > ~/.pip/pip.conf

$ echo ‘download_cache = ~/.pip/cache’ >> ~/.pip/pip.conf

  1. Make python in the unzipped source code folder (Python-3.5.1), install it in a folder that will not overwrite the current python version, such as in /usr/local/opt, delete upzipped source folder after done

$ cd Python-3.5.1

$ ./configure –prefix=/usr/local/opt/python-3.5.1

$ make

$ sudo make install

$ sudo mkdir /usr/local/opt/python-3.5.1

  1. Download pyserial source code and unzip in a folder.
  2. In the unzipped folder, run python 3.5 to install the module. If you used an alias to call python, then the install is in the default python version (3.4 on RPI) so don’t use an alias.

$ sudo /usr/local/opt/python-3.5.1/bin./python3.5 setup.py install

  1. Make aliases to run python 3.5 and idle more easily. You have to log out and back in after making them.

$ echo ‘alias python35=”/usr/local/opt/python3.5.1/bin/python3.5″‘ >> .bashrc

$ echo ‘alias idle35=”/usr/local/opt/python3.5.1/bin/idle3.5″‘ >> .bashrc

Now you are done! I forgot to mention that if you are installing Python on raspberry pi 1 or zero, the build process will take quite some time to complete. Log out and back in. In a terminal, type idle35 and you will be able to run your new Python.

SDI-12 USB adapter

After some delay, the SDI-12 USB adapter is finally here:

2015-10-03 16.16.57

This adapter is extremely easy to use. Just connect it to your PC and SDI-12 sensor. Then you can use any serial monitor or terminal emulator program to talk with your sensor. Just open the serial port at 9600 Baud rate. You can start by sending device identification command such as ?!. You will see a response from your sensor, which is the one-character address of your sensor. If you have not set its address, it is most likely to be zero (0). Then you can use 0I! to find out the manufacture and model of your sensor, before getting measurements from it. Getting measurement is easy as pie. First send 0M!, then wait for response. Then send 0D0! to fetch the measurements.

For PC users, I even wrote a data logger script that can automatically log data using the popular Tera Term program. You can choose sensor address, total number of data points, delay between points, and time zone when logging, then the program will keep logging data. The following is a screen grab of Tera Term. The sensor is a Decagon 5TM soil temperature and moisture sensor. The address is one (1) and the returned values are relative dielectric permittivity and then temperature in Celsius.

Data logger

Once you get data logging going, you can import the .CSV file into your Excel and plot it. You can choose a proper refresh rate so your data and plot are up to date when you look at them.

Plot

I’m still ordering more circuit boards but should be able to sell these on my inmojo.com store starting now. There is even a quantity discount if you need 10 or more.

Inmojo store sales page

Arduino board install problems

Arduino board is darn close to plug and play. However, there are many arduino compatible boards that come with different USB chips, making the play part just a bit further away from the plug part. So, if you have got your new arduino (or compatible), how do you install the USB chip driver? The following are the steps to identify what driver you need, where to find them, and how to install them.

There are several USB chips different Arduino compatible boards use. An incomplete list of them is supplied at the end of the post.

First of all, if you have good eye sights, you should locate the USB chip on your arduino, which is usually near the USB socket, read its markings. Then find the driver from the list at the end of the post.

On Linux, many of these drivers are included in the distribution so you don’t notice anything except that you now have a new serial port, such as TTYUSB0. On windows, however, you need more work to get the driver installed.

If you are using windows, and you can’t determine (markings too faint) what chip set you have, do the following after plugging in your arduino board:

Go to device manager and find the device with an exclamation mark under “Other devices” as an “Unknown device” (mine actually is already recognized as COM47, yeah, lots of Arduinos).

unknown device

Select properties and view the Details tab. Select under Property “Hardware Ids”. Look at the value.

USB serial port propertiesMine has VID=0403 and PID=6001. It is the FTDI FT232RL. This tells me that I would need the FTDI driver in case the driver is not already installed.

Install the correct driver, remove the arduino, reinsert it and wait for the driver to properly associate with the board. In case that didn’t happen automatically, you want to then right click the unknown device and update driver, then choose “browse my computer”. Then point to the right driver location. This step is necessary for all ATMEGAXXuX chips. The driver is in Arduino IDE under Drivers folder.

Update software driver

The following is a working list of chips, VID, PID and download links. I don’t guarantee the links are current or virus free but I did post official links.

Chip VID PID Board Link Note
Atmel ATMEGA16U2 2341 003D 003F 0042 0043 0044 ADK, DUE, MEGA2560 R3, UNO R3, ADK R3, clones Included in Arduino software under drivers
Atmel ATMEGA32U4 2341 8036 etc. Leonardo, micro, clones Included in Arduino software under drivers
FTDI FT232RL 0403 6001 Nano, Duemilanove, MEGA, clones http://www.ftdichip.com/Drivers/VCP.htm Included in Arduino software under drivers. Many fakes exist so avoid buying cheap Arduino compatible board with this chip.
WCH CH34X 1A86 5523 7523 etc. Many clone boards http://www.wch.cn/download/CH341SER_ZIP.html Supports win 10. Seems best choice for Arduino compatible boards.
Prolific PL2303 10CE Many clone boards http://www.prolific.com.tw/US/ShowProduct.aspx?p_id=225&pcid=41 The company claims that many older PL2303 models on the market are fakes so it has stopped supplying drivers for win 7 and up on these models.
Silicon Labs CP210X 11F6 Many clone boards https://www.silabs.com/products/mcu/Pages/USBtoUARTBridgeVCPDrivers.aspx The driver may not work with win 10 so if you just upgraded to win 10 and your board stops working, …

 

Follow

Get every new post delivered to your Inbox.

Join 61 other followers

%d bloggers like this: