SDI-12 USB adapter manual updated

I have just finished an update to the SDI-12 USB adapter documentation.

Updates:

  1. I added a document to the manual: Sensor connection guide. It lists a number of SDI-12 and other sensor wiring information from various manufacturers. Find it in the SDI-12 USB adapter page under documentation.
  2. All sections are now numbered now for easy reference.
  3. Two new sensors have been tested by customers to work perfectly with the adapter: Campbell Scientific CS650 and CS655 soil sensors.
  4. I added an appendix to explain how to use external power for your SDI-12 sensors, in case they don’t work with the 5V power supplied by the SDI-12 USB adapter. A wiring diagram is included.
  5. I added an appendix to explain how to easily connect a Decagon SDI-12 sensor with a stereo plug with a picture.

SDI-12 USD adapter with external power SDI-12 USD adapter with stereo adapter

Arduino Nano for battery-powered projects

Since Arduino Nano is very small in size and cheap, it might be a good choice for a battery-powered project.
Clones are abundant on ebay and other websites for around 4USD and mostly come with CH340 series USB/TTL adapter, instead of the original FT232.

To see how effective it is as a battery-powered platform, I made some measurements and would like to share my results.

First, to save power, you will need a boost converter (or DC-DC converter) instead of using siz AA batteries or 9V batteries on VIN pin. The on-board converter is a 1117-5.0 linear regulator that is not very efficient. I recommend something like this:

https://www.sparkfun.com/products/10968

You can also find some similar boost converters on ebay for less.

You can use two AA batteries to power the converter and get 5V output. Supply it to 5V on Nano. I measured currents on the battery side (@3V), not the 5V side, since I was trying to estimate battery life. I also measured currents on the USB side in case I use a power bank.

When Arduino is sleeping with POWER_DOWN sleep mode, the whole board consumes 15.5mA @3V from the battery. If you are using USB to power it (say you use a power bank), it consumes about 18.5mA. This is definitely not good enough for batteries. Even with a gigantic 20,000mAh power bank, the maximal time is 45 days. The efficiency of the boost converter and activities (other than sleeping) will reduce this to just a handful of days.

Next, the power indicator LED consumes a lot of current, if you want to use batteries, such as two AA batteries. You will need to remove the chip resistor next to it.

My clone has a 1kohm resistor in series with the LED. Once removed, the consumption went down from 15.5mA to 8.8mA (@3V). The resistor I removed was the 3rd one from the right side of the white reset button:


This is 50% saving, which is good and easy to do. You just hold your iron on the chip resistor until it is hot enough to remove it. Having a pair of tweezers helps a lot. Your project may now last weeks on two AA batteries.

After this, you will have to remove the 1117-5.0 regulator to get more power saving. This step is difficult. I used a hot air rework station to carefully heat up the regulator and removed it with only a little damage on the male headers’ plastics. If you don’t plan on putting the regulator back, get a small flush wire cutter and just cut out the regulator at the pins. This should do:

https://www.sparkfun.com/products/11952

The wide tab might be difficult to cut but once you cut the three small pins, you can just hold your iron on the wide tab and remove it.

Once removed with the Arduino put to sleep, only 0.74mA of current is consumed at 3V on the battery. This will project into months of operation, if you are logging data at moderate rate.

So if some of you thinks this is not really what a “pico” processor promises, then I am sorry to tell you that the Nano just wasn’t designed for battery operation. The CH340 consumes a certain amount of power even when sleeping. About 0.15mA at 5V so it could translate to 0.25mA at 3V. Its pull-up and pull-down resistors also cost similar amount of current draw. There is no way to remove the CH340 chip and still use the USB port. Using an arduino mini will solve this problem since it has no USB chip.

On a data logger I designed around ATMEGA1284P, I used FT232 but it will not receive power from the battery. The power consumption is about 0.4mA when sleeping. Not as low as it can go but good enough for 6 months.

So the conclusion is to at least remove the LED’s resistor if not 1117-5.0 regulator as well. If you want to use AA batteries etc. get a boost converter.

Then, if you have other devices, such as sd card, rtc, sensors, you need to power them with a transistor so you can but their powers when arduino goes to sleep.

Hope this helps. I’ll write about parasitic power and how you eliminate it another day.

Saving data to sparkfun server

If you’ve been following my posts, I discussed how to use my SDI-12 USB adapter and raspberry pi (or PC) to log data from SDI-12 sensors. Now that you have your data, you can also send them to sparkfun’s data server for storage and later retrieval. In the following video, I explained what you can do with this feature included in my data logger python script. I also constructed a sample webpage to display data saved on the server. Lots of customization can be made based on this sample webpage.

 

Log data with SDI-12 USB adapter

On my last post, I showed a couple of videos of how to connect an SDI-12 sensor to the SDI-12 USB adapter. Here is a video of how to configure an SDI-12 sensor and log data with the adapter:

 

Sorry there was some noise in the background that I couldn’t get rid off with the authoring software. Here is the transcript in case you need it. I will explain how to send data to sparkfun’s data server in my next post.

Let’s plug in the SDI-12 USB adapter. Windows will automatically install its driver and create a COM port, in this case, COM18. If this is your first time installing a COM port on your computer, you will need to be patient.

What we are looking at are two windows. The window on the left is the python shell. It shows you the input and output of python scripts. The window on the right side is the SDI-12 configuration script version 1.1. We run this script to set up the sensor’s SDI-12 address.

On the left window, you can see the messages printed by the configuration script in blue. This script runs on windows, mac, linux, and raspberry pi.

The script has discovered a number of COM ports and listed them all on the left window. The first one, item zero, is our SDI-12 USB adapter, i.e. COM18. Type zero and enter. Ignore the rest. If you are unsure about the COM port number, run this script with the adapter disconnected and then run it again with the adapter connected. This way you can see which port belongs to the adapter.

After a short moment, the script has detected the SDI-12 sensor at address 1. The information printed out indicates sensor address 1, compliant with SDI-12 standard version 1.3, and the manufacturer is Decagon. The sensor is a spectral reflectance sensor and its serial number is also printed out.

What this script does is to detect the one-character SDI-12 sensor address, print out the sensor information, and allows you to change its SDI-12 sensor address. Valid address includes 0-9, A-Z, and a-z.

Let’s set the sensor address to 2. To check that this has taken effect, we run the script again.

Now it has detected the new address. The address is saved on the sensor until it is changed again.

Now let’s look at the data logger script.

What this script does is: it logs data to two places, a file on the computer, and it also sends the same data to a server at sparkfun electronics. Everyone that runs this sample data logging script shares the same storage on the server and can see results from everyone else. You can also create your own storage or stream on sparkfun so you can keep the data to yourself.

Here I have python shell and sdi_12_logger script version 1.1. Let’s run the script with F5. This time there is a longer printout and it does say it runs on Mac OSX.

Next we see a list of serial ports. We select zero for COM18, like before.

Then we are required to enter the total number of data points. Let’s try 5. You can enter any large number of data points and stop the data acquisition anytime with Ctrl-C. Data saved on your computer and sent to the server will not be lost if you stop the script.

Delay between data points is specified in seconds. We’ll use 10 seconds. Then we decide whether to store each data point with local or universal time. Enter 1 to pick local time.

Enter the SDI-12 sensor address, 2.

Here is the first data point. Date and time, then two spectral reflectance values, and then 2.0 for facing upward.

The curl command that appears on the next line will only appear if you have curl installed, which is a tool to send HTTP requests. So besides a local copy of your data, you also have an online copy on sparkfun electronics server.

OK now we finished collecting 5 data points and the python shell prompt has returned.

We will take a look at the data file and the server data next.

Connect SDI-12 USB adapter to a sensor

If you wonder how to connect your SDI-12 sensor to the SDI-12 USB adapter, here are two videos showing you how to. In the video, I was using a Decagon PRI spectral reflectance sensor. It comes with 3 wire leads. Bare wire is ground. Wire wire is power. Red wire is SDI-12 bus. If you purchased a Decagon sensor with the 1/4″ stereo plug, you can purchase an adapter from them that break out the wires into 3 wire leads.

Decagon sensor wiring

The jeweler’s screw driver that I used was 0.1″ wide flat-head. On my next post, I will demonstrate how to run configuration and data logging scripts.

Install python on windows PC

Last time I described how to install python on Debian or Raspberry Pi. If you are a beginner and have a windows PC, here is my short tutorial on how to install python AND pyserial on a windows PC. Since I’m interested in Arduino, I will certainly need a serial module in python to talk to Arduino, such as sending Real time (yyyy/mm/dd hh:mm:ss or epoch) to Arduino to set its Real Time Clock or collect data from Arduino.

Python as of the writing has two main versions, Python 2.x and 3.x. The latest 2.x is 2.7.11. Lots of code have been developed on Python 2.x so many people still hang on to this version. The other day I tinkered with a Raspberry Pi LCD hat (add-on) and the Adafruit Python module that drives it can only be installed on Python 2.7.11. The most current version of Python 3 is 3.5.1. We will focus on this version since it has been out for quite a while. Also, Pyserial works flawlessly on Python 3.5.1.

First, download the installer here:

https://www.python.org/downloads/release/python-351/

Go to the bottom of the page and find your PC’s version (32 or 64 bit).

Install Python. Now, you need to find where the installed Python resides so you can include its path in your system’s PATH environment variable.

For me, it is here:

C:\Users\your_username\AppData\Local\Programs\Python\Python35\

Make sure you replace your_username with your actual user name (usually shortened from your full user name in windows 10).

Right click your “This PC” icon on desktop and select “Properties”. Then go to “Advanced system settings” on the left pane.

Once the dialog pops up, select “Environment Variables” on bottom right:

Advanced system settings

Once opened, you see the top half as user variables.

Environment VariablesClick PATH and Edit. Add the following TWO paths:

C:\Users\your_username\AppData\Local\Programs\Python\Python35\

C:\Users\your_username\AppData\Local\Programs\Python\Python35\Scripts\

Save and restart your computer.

Now you should have Python installed and ready to go. Still, you will need to install various modules to expand your Python, or simply to run “that script that wants this and that module”. Lots of modules are available to Python such as pyserial (serial port support) and idlex (more customization to Python’s IDE).

To install pyserial (or any other module), first open a command prompt. Then type in:

“pip install pyserial”

The package will install automatically. You are done!

 

30-day temperature data

Since I was going away for a month, I decided, before I left, to set my SDI-12 data logger to run a 30-day data collection routine, just to see how robust the software is. Here is the result:

30-day temperature

The temperature sensor was in my office so its variation was small. The steep slope towards the end was because I opened the window after I got back and kept the window open for a whole day. You may notice the lack of variation of temperature between 5/10 and 5/13 but that WAS what happened. History data from wunderground.com showed less-than-average variation of outdoor temperature in my area during the same period:

wundergroundSo my conclusion was that my python data logger code running on raspberry pi was robust enough for at least 30 days so was the SDI-12 USB adapter. If this were done outdoors, then the raspberry pi and the SID-12 USB adapter both need protection from the element. Also a solar panel and battery will be needed.

Parts and components:

Raspberry pi 2 B (or 3 B)

SDI-12 USB adapter (Free python data logger code)

Decagon 5TM soil sensor

 

Soil data logger telemetry

I have finally found time to build a simple website for my soil data logger with telemetry. The system works as the following:

  1. The data logger consists of a raspberry pi and my SDI-12 USB adapter with a Decagon 5TM soil sensor
  2. The data logger runs the open-source datalogger code I wrote in Python to first get parameters from the user (COM port, SDI-12 address, delay etc.), and then collect data, save to a local .CSV file, and then send the same data to sparkfun’s phant server.
  3. I constructed a web interface to plot the data using Google Charts and download .CSV version from sparkfun’s phant server.

Here is a screen shot:

soil logger webpage

I’ve uploaded the webpage to a server with a link below. The sensor is apparently NOT buried in soil so I can easily take the setup and set it up in different places to test its stability.

Link to the website: Link

Update SDI-12 USB adapter firmware

I have just released an update to the SDI-12 USB adapter. This update makes the adapter recover from errors in communication so it won’t hang your data logging process in case a sensor is broken while you are logging.

To make the firmware update easy, I wrote a Python script to use avrdude.exe to load the firmware to the adapter. You will run the script just like the data logging script or config script:

  1. Install Python 3.5 with Pyserial 3.0 (instruction in the manual).
  2. Unzip the content of the firmware update package in a folder.
  3. Check the properties of avrdude.exe to make sure that it is not blocked from running.
  4. Run the script SDI_12_firmware_update.py in Python environment IDLE.
  5. Select the adapter’s serial port from a list.
  6. Pick the firmware file, usually SDI_12_translator_v1_x.hex (must be stored in the same folder as the script), then wait for it to complete, which takes less than 30 seconds.

Here is the output from Python:

 

sdi-12 USB adapter firmware update

I had the adapter on COM4.

Once updated, try connect to it using a terminal program and send zI! (zee-EYE!) and you will see the response has version 1.2 in it.

(Updated) This script now works on Windows, Mac OSX, GNU/Linux (64-bit) and Raspberry Pi.

Back up and clone raspberry pi

In this post, I will explain how to back up and restore or clone a raspberry pi. I am assuming that you have installed Raspbian or Ubuntu mate on your raspberry pi. Since a raspberry pi runs its OS entirely on an sd card, it is easy to clone your system so you can pass along to a friend or create your special distro for others to use. For example, a company that makes compact optical spectrometer, Ocean Optics, has created a spectrometer program that runs on raspberry pi and has its own web interface. You just have to download their raspberry pi image and put it on your card. No need to do multiple build from source and apt-get etc. to get what you need.

If you are already a Linux guru, this post is NOT for you. I am assuming that you don’t have a Linux machine other than raspberry pi or you are still learning Linux.

The “easy” way:

Raspberry pi foundation has raw images of their Raspbian and NOOB on their website. On a windows system, all you need is win32 disk imager. It reads in the image and writes it on an SD card. You can save the image it reads and keep it as a backup or use it to clone a system.

The catch:

Since win32 disk imager only reads and writes in raw format, it actually doesn’t know what it is reading/writing. Say if you want to clone your raspberry pi 2B running Raspbian Jessie on an 8GB Kingston microSD card onto an 8GB SanDisk microSD card, you simply can’t. Although both cards claim to be 8GB, the SanDisk card is about 20MB smaller than the Kingston. Not understanding what it reads and writes, win32 disk imager is unable to shrink even one byte of what it reads (the whole 8GB from Kingston) to try to fit it onto a smaller card. This has pushed people into buying larger cards, such as 16GB. Then when they try to clone a system again, they run into the same issue unless they have bought a few identical SD cards as clone targets. Even that has issues (read the end of the post for details). So they have to step up to 32GB!!!

The solution:

We need a disk/partition reader and writer that understands what it is processing and is able to resize the partitions so they fit. If you are not hosting a lot of large files, you should be OK with an 8GB card or more than OK with a 16GB card. I’ve looked around for quite some time and found my solution: Paragon backup and Recovery 14 Free edition (Home edition for $39,99 has more features not useful for us). It is only for windows so those mac users will either need to become Linux gurus or shell out a small amount of money to get a win 10 netbook.

What the program does is that it is able to back up a whole disk or SD card and restore it on disk or SD card of different size. The reason is it understands most common file systems such as FAT, NTFS, HFS, and EXT. Besides, it also automatically compresses the backup so it doesn’t take more space than it should on your PC.

How to do it:

First you make a backup of your raspberry pi card using back up to VD. It is a virtual hard drive. This backs up both BOOT partition (FAT partition) and your Linux partition (ext4).

backup_to _vd

Next, insert your new SD card (you need to quit the program and restart it). Select restore from VD. It will ask you whether to resize partition and you can choose yes. This way you can even squeeze your clone from a larger card (32GB) to a smaller card (8GB) if your Linux partition has a lot of space.

Another neat trick:

What else you can do is to resize BOOT partition. Although BOOT partition has not much use for most of us, it IS the only partition that windows will recognize. If you are using your pi as a data logger or need to copy files from and to it (without the hassle of SSH and network setting), then the best way to get data is if it is written to the BOOT partition. You turn off raspberry pi and put its SD card on your windows machine. Immediately you have access to BOOT partition. You can copy your data out, you can change some config files of your data logger etc. It is useful to have a large BOOT partition. To do this, you will have to restore one partition at a time. Restore BOOT first. You will be prompted to select the size of the restored partition. Make your selection, say 500MB. Then when the restore is over, use the rest of the space on card to restore your Linux partition (make sure you leave enough space for Linux partition).

Why buying identical cards may not solve clone/restore issue?

Well, say you purchased 10 Kingston 8GB microSD cards, they are all the same size, correct? Wrong! I learned it the hard way. I had a friend that sent me an image he saved from such a card. I have identical cards. But, when I was telling win32 disk imager to write the image on my card, it complained: the target card doesn’t have enough space! Guess what, the target card is one sector less than the image file has?! But how can this happen? It’s simple now that I got my answer: some sectors on the SD card may have become bad either at factory or post purchase. The SD card controller (inside the card there is a controller) has disabled those sectors, making not all “identical” cards identical.

 

Follow

Get every new post delivered to your Inbox.

Join 67 other followers

%d bloggers like this: