Simple Python SDI-12 logging scrip

In case you wish to integrate the SDI-12 USB adapter into your existing Python script, here I provide a simple script to demonstrate how to get data. You can also use this script as a spring board to establish serial port communication in Python for anything else, such as talking to Arduino.

The goal is to not complicate things with the full-feature data logger script. You’ll see that the actual data logging only needs a few lines of code:

Simple sensor detection and reading (2018-12-03)

This script demonstrates how to integrate SDI-12 sensors into your existing Python data logging system by providing the minimal necessary features. It MUST run with a single sensor on the bus. For full-featured logging script, download the Data Logger script.

I’ll keep this script together with all my other scripts and provide updates to it when necessary. I’m also considering writing simple scripts for other programming languages. Do you wish to use the SDI-12 USB adapter with a programming language other than Python? Leave me a message here.

2018 5-month SDI-12 USB adapter run

I conducted a long run (5 months) on an SDI-12 sensor with a raspberry pi zero in the months of June to October to test the long-term stability of my data logging solution. I was using a regular SDI-12 USB adapter, a Decagon/METER Group soil sensor model 5TM with dielectric constant and temperature sensors and I buried it in my back yard. Over 100K data points later, the ground was freezing and I unearthed my sensor and stopped the feed. Here is an overview of the data feed. Please click the plot to see a full-size screen grab.

The top plot is dielectric constant. Each spike represents a major rainfall in my area that made the electrical conductivity of the soil shoot up and then gradually die down. Each small step is when I turned on my sprinkler system to water my lawn (smaller than the spikes but still significant within a range of a few days).

The bottom plot is ground temperature, about 20 cm in the soil. The diurnal temperature variation is greater during the summer than during the fall, as represented by the larger amplitude oscillations of temperature at the beginning of the graph and the gradual reduction of oscillation amplitudes towards the end of October.

Despite of a number of power outages (can you even see them from the graphs?), the system worked flawlessly, restarting when power was restored. In retrospect, I should have invested in a backup battery such as this one:

https://www.amazon.com/gp/product/B01M7Z9Z1N/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1

It is charged by a 12V ac adapter and provides 5V power on its USB port. When the ac power is out, it uses its internal battery to continue to provide 5V power. For a small system such as raspberry pi zero, it could power it for hours.

If you wish to see the complete data set, here is the link to download:

https://www.dropbox.com/s/4sry1pqwp9hdxuo/2018%20summer%20fall%20lawn%20data%20feeds.xlsx?dl=0

For more information on the SDI-12 adapter and data logging scripts, go to their dedicated page here:

https://liudr.wordpress.com/gadget/sdi-12-usb-adapter/

Selling my gadgets at Tindie.com

I have been looking for more market places to sell my gadgets. Recently a fellow maker recommended tindie.com. I decided to give it a try. There are more sellers and items for sale on tindie.com than inmojo market place and tindie is better maintained (by charging 4% seller fees). For now I only have 4 items listed on tindie for sale:

Tindie

It seems to work alright, although the fees have more than doubled from just the 2%-4% paypal fee for inmojo or my blog sales to 5% Tindie selling fee + 2%-4% paypal fee. I decided to not change the selling prices to compensate. Hopefully if you’re able to use inmojo market place or my sales links on my blog page, you will use them to help me save some fees and keep prices reasonable.

The tindie market place is global and has a fair amount of interesting gadgets for sale. I sometimes just browse the site looking for interesting products and inspirations from fellow makers. Check it out!

Here is a clickable badge to take you to my tindie store:
I sell on Tindie

Only 4 orders so far, a long way to build up the popularity that I have on inmojo.

Provide or receive remote help

Sometimes it’s my mom, or someone having trouble setting their their SDI-12 USB adapters, I always find it necessary to provide remote help on computers. Luckily there are tools to do that. Dell has its own version where their tech just take control of your mouse and keyboard (a bit scary at first) and other companies have similar ones. If you’re finding yourself needing to provide or receive remote help on computers, I recommend teamviewer:

https://www.teamviewer.us/

You can use it free of charge to provide help on a non-commercial basis (read their terms). You don’t have to worry about network firewalls regular people have on their home router and having to find the correct IP address on the target computer. Teamviewer solves all those problems for you by giving each computer a phone-number-like 9-digit number so the person seeking help can give to the helper. There is also a one-time password that can be given to the helper for one-use only. The password changes after each use or you can refresh it manually. If it’s your mom, you will likely help her set up a permanent access:

Go to “Extra” then “Options” and find the following dialog:

Select “Security” and set a personal password so you can always log on to your mom’s computer using the 9-digit number and this password as long as she has turned on her computer and has internet connection.

Besides poking around the remote computer, you can send files back and forth, talk/text chat to the person on the other computer, do some screen share etc. There are lots of features that you might find useful. Besides, you can run this software on win/Linux/Mac and raspberry pi (Linux version is eval. and has no file transfer features as of the writing of this post).

Metal stencil – first impression

Over the past summer I have been designing my SDI-12 data logger. So finally I was ready to send my design out to JLCPCB.com for a small batch (10 boards). I thought, why not trying out their $7 stainless steel stencil for a change? I’ve used Kapton stencils for over four years and have grown tired of the floppy stencils and frequent need to clean under the stencil while stenciling the boards. Bigger holes like crystals often leak solder paste under them and make the printing a bit messy. Overall leakage causes printing to degrade after a few board and you might have to wash it clean after a dozen boards if you can’t stand the degraded prints. Plus, alignment with Kapton stencils is not great since the film can be stretched very slightly to lose alignment across a slightly larger board (a few inches long).

Since I reviewed the quality of a batch of printed circuit boards made by JLCPCB.com in comparison with a popular PCB service a while ago, they were pretty happy with the details I went into with my favorable review and promised to print another batch of boards for free. I took the offer and asked for a stainless steel stencil. So the following story is my first impression of stainless steel stencils, which is very positive after some minor struggle. Hope it is as entertaining as it is useful.

It took me approximately one week to get the board and the stencil in a DHL envelop. Here is the package inside the DHL envelop:

It’s actually pretty big due to the large size stencil. You can make a rather large (non-framed) stencil with their $7 service. Let’s see what’s inside the package:

The package has two wood/paper/cellulose boards that sandwich the “gigantic” stencil. The stencil is protected (on both sides) by a polymer film. By the way, I used half of one of the cellulose board (probably a sacrificial base board for PCB routing) and built a home sprinkler controller on it. Details of that project will be forthcoming when I clear my backlog of “things to be posted”.

The above is a close-up. The exact side is 28cm by 38cm (11inch by 15 inch). The thickness is 0.12mm or between 4mil and 5mil, just what I wanted. My other stencils made with Kapton are only 3inch by 3inch or 5inch by 5inch. The actual stencil area is no more than 4inch by 4inch. Here is how 4*4 area looks like on an 11*15 area:

Here are the PCBs (10-qty) and a ball pen with paper stem, very environmental!

Now I have to find a good way to cut out the stencil from the whole sheet of stainless steel into something around 5inch by 5inch. I asked around online and was suggested to use a knife:

I tried several times as hard as I could with a ruler over the right side as a test but only was able to scratch a line without any hope to cut through. What was I thinking, cutting steel with steel blade? Let’s try scissors now that I have a line :

Wait a minute, let’s bend along the line to see if it breaks. LOL it won’t break! Back to scissors.

I made the cut but the edge was not flat. I can’t use this on printed circuit boards! I won’t even use these as decent shims.

I started looking for tools around my lab and office area. How about this paper cutter?

It seems to cut just fine although I need to push hard to get it started. The cut edge is flat and smooth:

A few more cuts later, just the right size (with PCB on left):

 

One things worth showing is that the surface of the stencil is not as smooth as the rest of the surface. There are fish-scale shaped marks (see the photo above for overall):

Here is the surface around a 0805 resistor under a macro-zoom lens I use for board inspection:

The reason for the fish-scale marks is that the surface has been brushed by a metal brush to remove the burr from laser cut edges. This was expected. Let’s get started stenciling. The photo also shows how nice the square corners are. The width of the pattern is a little over 0.05 inches or 1.27mm. I secured a board (black) with some older phi-panel PCBs (green):

I used a a “credit-card” squeegee with a new syringe of SAC-305 solder paste:

Here is how nice the print looks (please click and zoom in):

I’ve never see clean prints like this before! The ADS1115 (0.5mm pitch) on the top left side is super clean! And you know the end result. I posted this photo in the announcement of my data logger prototype:

I didn’t have a single solder bridge or spot that needs rework. A list of components I put on the board: ATMEGA328P-AU, ADS1115, ESP32, MicroSD card slot, DS3231, battery, crystal, LM1117-5.0, various 0805 resistors and capacitors, polyfuse, transistors (FET and BJT), and a number of thru-hole components.

So the end result was pretty darn good, for $7 extra only! My Kapton stencils of smaller sizes cost more (including shipping). I would highly recommend this stencil service if you’re already considering getting PCB from them, JLCPCB.com.

By the way, the PCB printing quality was good as well. The black one on top is from JLCPCB.com while the purple one on the bottom is from a reputable service provider in the US (guess who uses purple exclusively?). Tye quality is comparable. The pitch of the pins is 0.5mm:

More about the data logger

Enclosed logger

Now that the logger prototype is enclosed, it is a lot easier to use. I have a METER group 5TM soil sensor connected to it from the left side and 12V power to it from the bottom. Without a heavy enclosure, these cables will make it hard to set the logger flat and connect an adapter to it. Plus, I’ve started using Telnet to connect to it wirelessly so I don’t need an adapter to connect to it most of the time.

Here is the adapter I modified to use on the logger:

2018-07-26 22.33.25

I bought it from moderndevices.com (Not shown on this photo) Later I had to short a resistor and a capacitor in order to actually make it work, due to the fact that the adapter was designed for Arduino boards.

Besides Telnet, I can also upload Python sketches via FTP. I later installed a couple of jumpers to easily reset the logger since I am still developing the logging script and need to reset the logger when I make changes. I’ll install a push button instead of the jumpers when I have some time.

Everything is coming along nicely. Soon I’ll start testing 4G LTE-M connectivity of the logger. 4G LTE-M has a lower bandwidth than the regular 4G LTE on smartphones so the modem would consume less power and data plans would cost less (in Megabytes instead of in Gigabytes). The Digi XBee3 LTE-M modem is rather expensive, at $100, or $150 for a dev kit. Anyway, if you are deploying the logger in a research field close to your lab/office or really don’t mind regularly visiting remote areas to collect data logs, you may skip the modem.

Data logger is here!

After months of planning and designing, I’m excited to announce that the all-in-one SDI-12 data logger is here! I got the board printed and assembled one for testing last week. Been testing its features for several days now. I’m putting it in a nice IP66-rated enclosure for more tests and development.

This device is meant to be a one-stop-shop for data logging needs with minimal setup and learning time while my SDI-12 USB adapters are meant to add SDI-12 and other sensors to your existing datalogging PC or Raspberry Pi (assuming you’re willing to learn some Linux and the Pi).

For now, here are a couple of photos of just the board:

Board in my hand

Board with a 4-channel hi-res analog input extension board

Here is a list of features:

  • Open-source full-feature Python data logging script
  • SDI-12 translator with 4 SDI-12 sensor connectors¬†(more can be added)
  • 4 analog auto-scaling single-ended analog inputs up to 20uV precision
  • 2 analog auto-scaling differential analog inputs up to 20uV precision (alt. to 4-chn single-end)
  • Socket for optional 2G/3G/4G LTE/ZigBee/Digimesh RF modules
  • Resistance sensing with optional precision 1K or 10K pull-up resistors (0.1% 20ppm/degC)
  • Extension port allows up to 12 more precision analog and resistance sensors
  • Micro-SD card slot (up to 32GB of non-volatile program and data storage)
  • Data file download via FTP (Wi-Fi)
  • Hi-precision temperature-compensated Real Time Clock for time keeping (2-4 ppm)
  • Power cycle SDI-12 sensors to reset problematic SDI-12 sensors
  • Wi-Fi connection to upload data to servers (requires a Wi-Fi hotspot/router)
  • 4 analog single-ended analog inputs up to 5mV precision where less precision is acceptable
  • 4 digital on/off inputs (alt. to 4 5mV analog chn)
  • 4 digital pulse counters for rain gauges, anemometers etc. (alt. to 4 5vV analog chn)
  • Device temperature monitoring
  • Starter MicroSD card with logging script included
  • Xbee sleep to conserve power
  • Interactive sensor and logger setup interface
  • Optional water-proof enclosure
  • Optional wired adapter for more software development

I will add more details later.

SDI-12 USB adapter upgraded

2018-07-02 15.25.47

After some more firmware development and testing, I am happy to announce that the upgraded SDI-12 USB adapter is now available. The above is the first batch of these adapters and one hi-res analog input add-on board.

The upgraded board features the following:

  1. 4 analog inputs. 12-pole terminal block that features 4 analog voltage inputs. These are 10-bit or 5mV resolution inputs without differential reading. There are there to provide basic voltage inputs for projects that don’t require hi-res analog voltage inputs.
  2. Pulse counters. Alternatively, these 4 inputs can be used as pulse counters. Say you have a rain gauge or flow meter that outputs pulses, these pins can count the pulses. You may need additional filtering (one capacitor and two resistors) if the pulses are noisy. Each time you read the pulse counts, you get the counts since you last read and the adapter will start counting from zero again. This way, if you collect data every minute, the counts will be counts/min. Because each data point is accompanied with date-time information, you can always calculate the count rate with your data set.
  3. Extension port. There is now an 8-pin extension port for add-on boards. The first extension board I have designed and tested is a hi-res analog voltage input board. This board features the same four 16-bit auto-scaling inputs and differential inputs as the SDI-12 + Analog USB adapter, with an added benefit of address jumper. You can add as many as 4 such extension boards to the new SDI-12 USB adapter, with each extension board taking a different address. That is up to 16 hi-res voltage inputs.
  4. Serial port. There is also a serial port connector with RX, TX, etc. This port helps you connect the adapter to an arduino or a micropython board that don’t have USB connections but have serial ports.

With the added features, comes added costs of parts, quality checking, and development times. So I am currently offering two-tier pricing:

  1. The board with everything included and tested at $55
  2. The board without the 12-pole terminal block or these pins tested at $45
  3. If you need the serial port, I can solder the header and configure it free of charge.
  4. The hi-res analog add-on board is $35 each. If you want the 4 additional SDI-12 port soldered on it, I can do it for $5 extra.

The full-featured board has the additional terminal block and needs to be tested with all the analog pins to make sure they are all properly connected (reflowed). In the photo, the bread board and 4 blue potentiometer is the test rig I use to test hi-res analog inputs for the SDI-12 + Analog USB adapter. You CAN solder on your own 12-pole terminal block and test the pins yourself too. You will have to do a lot of screw/unscrew of a potentiometer or resistors though. The firmware is the same so once you solder the connector on, you can use the features.

2018-07-02 14.05.33

Although the analog add-on + new adapter costs the same as the original SDI-12 Analog USB adapter, the stack of two boards does add to its height so the board needs more space. My intention is to add flexibility to the adapter so I can later add more features to the ecosystem without scraping the existing devices. I did a custom board for someone that wants magnetometer, accelerometer, and gyroscope with SDI-12 sensors:

2018-06-17 12.15.47.jpg

I glued the sensor board on top of the adapter board and wired them together via the extension header. This way I didn’t have to spend time designing a new board, which will likely cost more time and money. Also, designing a new extension board is easier than designing a whole new adapter with the sensors on it. I can do more custom sensor boards even if there is only market for a few.

The purchase links to the full-featured adapter and the hi-res add-on board have not been established yet. I’ll get them up and running on my blog and at inmojo soon.

New data logging script

multiple sensor and adapter deploymentThe SDI-12 USB adapter data logging script has been updated to version 1.6.0. Here are the new features:

  • New configuration file format. I went from a basic format that is essentially the key strokes of the interactive inputs to a more human-readable, expandable, and future-proof dictionary-type of format.
  • The serial port is remembered in the configuration file by a unique ID instead of device name such as COM3. This makes it impossible to open the wrong serial port. You can also copy your configuration files to a different OS type and expect it to still work.
  • The data file name now includes the configuration file’s name, such as Liudr.cfg_20180709.csv. This makes it possible to distinguish different data that are collected using different configurations.
  • You can specify a different configuration file name in command line instead of the default. This makes it possible to run multiple adapters on the same system, each running its own configuration file. You can also set up different data logging scenarios and store them in different configuration files for easy recall.
  • There are also stability updates that prevent the script from quitting if, say, the sensor fails to return any valid response. Instead, the logger waits until the next data point and try again.
  • The logging script can now use multiple sensing commands per sensor, such as M!, M1!, M2! etc. besides reading multiple sensors. The interactive input has changed a bit to accommodate the upgrade.
  • Now if you run the script for the first time, it takes you to the interactive inputs to collect configuration parameters to save them into a configuration file. Then when you run it the second time, it automatically loads the configuration file. If you wish to make minimal changes, you can just edit the configuration file. If you want to run the interactive inputs again, delete the configuration file and run the script again.

This version is still considered “experimental” while I hunt for potential problems. You can find the script under the download section along with “changes in V1.6.0” document:

https://liudr.wordpress.com/gadget/sdi-12-usb-adapter/

Let me know what you think!

Open FTDI USB-serial UART port by ID

In this post, I will explain how to open serial port to your Arduino or SDI-12 USB adapter by its unique ID so you always open the correct port even when there are multiple such devices on your computer or raspberry pi.

For Arduino and SDI-12 USB adapter users, I have a nice trick to help you manage multiple Arduinos or SDI-12 USB adapters on the same computer or raspberry pi. On raspberry pi, as on a typical linux system, your device shows up as a serial port, such as /dev/ttyUSB0. This serial port designation is usually bound by the order that the device is discovered at boot time, which may not be the same even if you keep your adapter plugged into the same USB port. This means if you have more than one device on your raspberry pi, you may open the wrong port at times, which should be a big issue. To prevent your program from opening the wrong port, you need a unique ID for each device. Luckily FTDI chips already come with unique IDs. We just have to find those IDs and possibly change them into more meaningful things for us to remember. Assume for the moment you are making a data logger for your test fields. There is one field that can be called “NORTH”. The following steps will help you change the ID of the FTDI chip on your device so you can later open its port by that ID, instead of a port name. Here is a list of which devices are using FTDI’s chips that have the reprogrammable ID feature:

  • Liudr SDI-12 USB adapter (all types)
  • Sparkfun Redboard
  • Certain Arduino clone boards
  • Lots of other devices such as GPS etc.

Here is the FT_PROG tool FTDI provides. It’s windows only but I’m sure you can find a windows machine to run it. I’ve not tested it in a virtual machine whose host is linux or macos. I’ll do that when I have more time. If you are unsure whether your device has an FTDI chip, a quick scan using the program will tell you.

http://www.ftdichip.com/Support/Utilities.htm#FT_PROG

First, press the scan icon (magnifying glass). If you have a device with FTDI chip, it will show up. See the screen grab below:

So I have an FT232R chip with a chip serial number “A106DHE5”. I can open port with this serial number but I’d rather change it to “NORTH”. Click on the “SerialNumber” from the left side.

Uncheck the “auto generate serial no” so you can edit the serial number to “NORTH”. You have up to 16 characters to name the adapter. Once done, press flash icon (thunder bolt).

Now that you have programmed your chip, you can read the information back using “scan” again to verify that the ID has changed:

Now that you have this nice ID, let’s open port by this ID.

Here is a small complication. On linux, the chip ID is returned, such as “NORTH”. On window, the port ID is returned, such as “NORTHA“. The addition of the “A” indicates the port “A” on chip “NORTH”. This is because some FTDI chips have two serial ports. The port IDs will be “NORTHA” and “NORTHB“. Even for FTDI chips that have only one port, such as for our case, the “A” is still there. So I recommend comparing chip ID instead of port ID. If you only work on linux/rpi systems, this doesn’t seem to concern you. But if you wish to make your code platform independent, i.e. running on windows without an incident, you will only extract whichever ID you receive with the stored chip ID, up to the length of the stored chip ID. Note: in the platform-independent code, you can’t slice a port’s serial_number with your stored ID because some internal ports don’t have serial numbers thus returns empty that will throw an error when you try to slice an empty array.

The following is a snippet that works ONLY on linux/rpi systems:


import serial.tools.list_ports  # For listing available serial ports
import serial  # For serial communication

my_ID='A817EQLG'
port_device=''
# List ports for user to select
a = serial.tools.list_ports.comports()
print('\nDetected the following serial ports:')
for w in a:
    print('Port:%s\tID#:=%s' % (w.device, w.serial_number))
    if (w.serial_number==my_ID): # Match ID with the correct port
        port_device=w.device # Store the device name to later open port with.
if len(port_device)!=0:
    print('\r\n%s is the correct port.' %(port_device))
else:
    print("Port with ID: %s is not found!" %(my_ID))

The following is a snippet that works on ALL OS:

import serial.tools.list_ports  # For listing available serial ports
import serial  # For serial communication

my_ID='A817EQLG'
port_device=''
# List ports for user to select
a = serial.tools.list_ports.comports()
print('\nDetected the following serial ports:')
for w in a:
    print('Port:%s\tID#:=%s' % (w.device, w.serial_number))
    if (w.serial_number.__str__()[:len(my_ID)]==my_ID): # Match ID with the correct port
        port_device=w.device # Store the device name to later open port with.
if len(port_device)!=0:
    print('\r\n%s is the correct port.' %(port_device))
else:
    print("Port with ID: %s is not found!" %(my_ID))

Your choice, simplicity of code or cross-platform compatibility.

Here is the cross-platform code’s result on my windows machine:

Detected the following serial ports:
Port:COM23	ID#:=A817EQLGA
Port:COM3	ID#:=None


COM23 is the correct port.

As you can see, there is an added “A” at the end of the ID reported by windows, which the python code ignored to produce a match.

I’ve also attempted to do this using Processing 3.0. Unfortunately, the Serial.getProperties() function that should return similar information returns blank (possibly not implemented on windows and yet to be tested on linux). If you have tested Processing method with success, please reply below with your results. I’ll add your comment to the post.

Here is the code I used in Processing 3.0:

import processing.serial.*;
import java.util.Map;
 
void setup() {
  String[] ports = Serial.list();
 
  for (int i=0; i < ports.length; i++) {
    Map<String, String> props = Serial.getProperties(ports[i]);
    print(ports[i]+": ");
    println(props);
  }
}

Results:

COM3: {}
COM23: {}
:(

Closing note: even if you work on Windows that assigns unique COM port number to your arduino or adapters, the assignment relies entirely on the currently available port numbers. If you develop your project on one windows PC and deploy on another windows PC, you WILL get different COM port numbers. On a Mac, the ID is embedded on the port name such as /dev/ttl.usbserial-A103RU9T so you are better off. But, will you be willing to shell out the money to get a mac and have it sit somewhere to collect data just because of this feature? If you are a linux wiz, you can bind names with serial numbers using some scripts. That’s beyond the scope of our general discussion, which assumes minimal experience with linux administration.

%d bloggers like this: