2019 summer run update

The 2019 summer SDI-12 USB adapter data logging run has been relatively smoothly but there were a few issues that are worth mentioning.

1. Battery

The battery that I recommended as a backup battery for raspberry pi (see photo below) needs to be used with some consideration:

I have not tested how long the charge lasts when it is powering a raspberry pi 3B (I used a raspberry pi zero last year but had to use some USB OTG adapter and a USB hub with Ethernet dongle etc. so I decided to throw my 3B in since I have a 3B+ now). What I thought should happen is that the battery will eventually drain completely after hours of power outage but once power is back on, it should turn on the raspberry pi right away. That turned out to be not the case. The battery seems to need about half an hour of charge time after it has been drained before it can power the raspberry pi 3B with enough stability that it would run normally. Within the first half hours of initial charging after a power outage, the raspberry pi was not stable and I couldn’t log in to it. The logging script didn’t run either. I ended up unplugging the pi and let the battery charge for 30 minutes and plugged the pi back in.

In case you want to prevent this issue, which may or may not affect the integrity of the pi’s operating system, you need to buy a backup battery with proper capacity. Mine is only 3000mAh, since I have the logger in my garage, only expecting at most hours of power outage:

https://www.amazon.com/TalentCell-Rechargeable-Amplifier-Multi-led-Indicator/dp/B00MHNQIR2/

There are 6000mAh and 12000mAh versions that will definitely address longer periods of power outage. If I was only expecting up to a few hours of power outage, then how come my logger drained the battery, twice?

The issue was actually with the power outlet it was plugged in, which is protected by an upstream Ground Fault Circuit Interrupter (GFCI) outlet. When I bought the house, there was no GFCI outlet in the garage! I wonder how those previous owners never thought about installing one. So I replaced the outlet in the garage with a GFCI outlet, which protects all downstream outlets. The outlet my logger is plugged in is downstream of the GFCI outlet thus is protected against ground fault, which means shorting the hot with the ground. It was the right move but unfortunately not enough. Outside my garage there is an outdoor outlet with some “outdoorish” cover. We had some VERY heavy downpours in the past month.

As you can see on the plot below, the two high spikes on soil dielectric constant correspond to two heavy rain falls:

The vertical blue lines simply indicate how quickly the rain fall caused soil’s dielectric constant to first rise then later drop (exponentially). The two almost horizontal blue lines immediately following the spikes were periods of power loss. I didn’t check my online plots and was surprised to find that the plot stopped. I guess I could use the spikes to estimate my battery’s run time to be between 4 hours and 13 hours, probably 4 hours. The following soil temperature plot during the same period shows the power outages more clearly:

So what caused the GFCI outlet to trip? It turned out to be the outdoor outlet. I took it apart. Here is what it looks like:

The gasket on the left wasn’t doing enough to protect the outlet. You can see the outlet’s rusted top side (the outlet was mounted horizontally). This must have been caused by rain water seeping into the outlet, causing ground fault.

I do recall a couple of times after bad weather our garage lost power (circuit breaker). That must have been before I installed the GFCI outlet. So in order to address the real issue that the outdoor outlet is causing, I purchased a better cover and installed a separate GFCI outlet in the power box. To be honest, the brick veneer didn’t help. It was harder to seal against due to its rough surface but the new cover with a new gasket hopefully will work better. I’ll later apply some ready-to-use cement on the top of the cover to seal any possible leaks. With this new GFCI outlet, hopefully even when it is shorted, it would act before the upstream one acts and prevents the logger from losing power for extended time again. Here is how it looks:

Next time I will write about my experience running the logging script automatically with the latest raspbian distribution.

SDI-12 USB adapter with larger terminals

I have recently received a few comments regarding the SDI-12 USB adapter’s terminals being too small. As a matter of fact, they are not big. They are 2.54mm (0.1″) pitch terminals. On the other hand, they can comfortable accept wires as thick as 18 AWG. I’ve rarely seen sensor cables having wires thicker than 20 AWG. Larger gauges are thinner so 18>20>22>24, AWG-wise. Also, if you have wire leads that are not tinned, you should twist the wire strands and tin the leads before inserting them into terminal blocks.

Still, having wider pitch makes it easier to insert the wire leads, including power and ground for the external power if you need that. So I made an update with 3.5mm (0.1385″) pitch terminals. I really like the size of the board and the mounting hole positions have not been changed since I made these square boards. I would like to keep them unchanged for past customers who may rely on the size to make more loggers. So here is an updated version board view (top) vs. current version (bottom):

The new 3.5mm pitch terminals will hang over the edge of the board a bit but fit the same board, after I moved the external power jumper a bit. To save space, I used a 2-pole terminal for the external power connector instead of 3-pole. The 3.5mm terminals can accept up t o16 AWG wires. I will print this board out during my next board order, which is probably a month from now. If you care to give me your opinion, please use the poll below, leave me a message, or write me a private email to zliudr@gmail.com. I will print this board on paper and make a mock-up to compare side-by-side with the current version. There is no reason both versions can’t coexist.

I also considered 5.07mm (0.2″) pitch terminals but they are just way too big to secure sensors with thinner wires. Let me know if you are interested in having this 3.5mm terminal version and why. If there is enough interest, I’ll make a batch or two.

I have a photo of 2.54mm vs. 3.5mm vs 5.07mm terminals:

2.54mm terminals are narrower than 3.5mm terminals. 5.07mm terminals are both wider and taller than the smaller versions. These are the most common terminal sizes on circuit boards.

 

 

Conversion table

2019 demo logger started on 6/3/2019

Last year’s demo logger data stream was very successful. It demonstrated the stability of my SDI-12 adapters for long-term data logging. It ran between June and October (Minnesota is cold) for 124 days. Except for occasional power outage at my house, the logger was running without a problem using the 1.5.0 logging script. This year I updated the logging script to 1.6.0 and have got a CCTV power bank to use as a backup power supply in case of power outage.

Here is a battery similar to mine that is sold on amazon.com:

The nice thing about this battery is that it gets charged via the 12V power barrel and discharged via the USB connector while it is charged. Most power banks can only be charged or discharged but not both. This one does them simultaneously. It is always charged to full when there is power in the AC. The USB port always has 5V power either from the AC or from its internal battery. Essentially this is a cheap Uninterruptible Power Supply. There is no surge protection except if your power strip has it. It is also very compact. I’ll embed it in an enclosure for a more complete enclosed logger later this month.

I expect to see next to zero down time due to this battery. This is also great for the raspberry pi since every time it loses power it could corrupt the SD card a little.

Here is the first day of data. We had some rain in the afternoon.

You can visit the live stream by clicking here.

Update to SDI-12 USB adapters

I recently got a request to add bidirectional transceivers to my SDI-12 USB adapters to handle very long SDI-12 bus wires (result of long wires for each sensor and a large number of sensors). Currently a couple of these adapters are being tested by one customer who requested this feature but I am pretty confident with its functions and will conduct my own testing with long wires. If this is what you have in mind, I have a handful of them I’ve built as prototypes. You can go ahead and purchase a regular SDI-12 USB adapter and request one with a transceiver. I don’t have a lot of them so I can only send you one or two. If you really need more of these, I’ll need to order boards and components.

The added transceiver will not affect any program code such as my Python data logging script. It is operated transparently. When the adapter receives a complete SDI-12 command, it will turn on the transceiver and transmit the command to the SDI-12 bus. Once done with transmission, it turns the transceiver off and returns to listening mode. The transceiver in the following photo is located just to the left of the top-right 3-pole terminal block (small black rectangle with 6 pins).

Additionally, I have received several requests to use my USB adapter as a TTL/serial adapter, such as connecting to arduino or MicroPython boards, either at 5V or 3.3V. I’ve updated my board design to make those requests easier to fulfill. This option is now added to inmojo marketplace as well as to Tindie marketplace (options used to cost $2.5 and now is free).

First, if you purchase a TTL/serial only adapter, you will not get USB connection anymore (notice the missing long black chip to the right of the empty USB connector pattern). You can’t really have both active simultaneously since there is only one TTL/serial port on the processor. It’s either connected to the breakout pins for TTL/serial use, or connected to the USB chip to communicate to PC/raspberry pi. The use cases of USB vs. TTL/serial also don’t overlap. One is for those who want to use PC or raspberry pi to log data, and another who want to use MicroPython boards or Arduino boards to log data. What you will get is a 6-pin connector on the bottom of the board, at 90 degrees so it’s not pointing straight down, rather sideways. See how the wires are under the board, running along the board and the next photo for the underside. This makes it possible to stack expansion boards or have optional analog/digital input headers (12-pole block on top edge). You still need a 5V supply even if you want a 3.3V TTL/serial interface. The following photo shows a 3.3V version. Note the solder blob on the top right to the immediate left of the text “TX3”. Then the TX3 on the serial port (marked JP9 on left and Serial Port on right) is outputting 3.3V logic. Remember that the adapter’s TX or TX3 should be connected to your other board’s RX pin since the adapter’s transmit (T) goes to your other board’s receive (R).

Raspberry pi data logging sd card image updated

To help people getting started using raspberry pi to log data with my SDI-12 USB adapters, I’ve been offering ready-to-go MicroSD cards with raspberry pi system image. You can pop it into your raspberry pi and get started with all the necessary tools installed, including the python logging script. Every few months, I update the image so you get the most recent operating system and the python logging script. This time I updated the image in late December. I just recently tested it on a raspberry pi 3A+ model and it works. I guess it’s no surprise. My image works on 3B and 3B+, Zero and ZeroW. The 3A+ has the same processor as 3B+ so it worked right away. I remember when Zero came out, the image I had didn’t work on it since it has a different processor than 3B (latest model then). I had to put the image in a 3B and run updates. After that it worked. It’s been a while since the raspberry pi folks designed the 3B model. I wonder if they are ready to release a model 4B later this year. I am hoping to see better performance but also hoping to see programmatic ways to dial back the performance for battery operation so the pi can enter a low-power mode while collecting data and spring back to full-throttle if it needs to crunch data (program controls power mode) or handle a user remote login (user select power mode). Anyway, if you’re stuck with an sd card having a working older image, all you have to do is to update:

sudo apt-get update
sudo apt-get upgrade

In case you need more details regarding updating your raspberry pi, here is the official post:

https://www.raspberrypi.org/documentation/raspbian/updating.md

 

More about the data logger

Enclosed logger

Now that the logger prototype is enclosed, it is a lot easier to use. I have a METER group 5TM soil sensor connected to it from the left side and 12V power to it from the bottom. Without a heavy enclosure, these cables will make it hard to set the logger flat and connect an adapter to it. Plus, I’ve started using Telnet to connect to it wirelessly so I don’t need an adapter to connect to it most of the time.

Here is the adapter I modified to use on the logger:

2018-07-26 22.33.25

I bought it from moderndevices.com (Not shown on this photo) Later I had to short a resistor and a capacitor in order to actually make it work, due to the fact that the adapter was designed for Arduino boards.

Besides Telnet, I can also upload Python sketches via FTP. I later installed a couple of jumpers to easily reset the logger since I am still developing the logging script and need to reset the logger when I make changes. I’ll install a push button instead of the jumpers when I have some time.

Everything is coming along nicely. Soon I’ll start testing 4G LTE-M connectivity of the logger. 4G LTE-M has a lower bandwidth than the regular 4G LTE on smartphones so the modem would consume less power and data plans would cost less (in Megabytes instead of in Gigabytes). The Digi XBee3 LTE-M modem is rather expensive, at $100, or $150 for a dev kit. Anyway, if you are deploying the logger in a research field close to your lab/office or really don’t mind regularly visiting remote areas to collect data logs, you may skip the modem.

Data logger is here!

After months of planning and designing, I’m excited to announce that the all-in-one SDI-12 data logger is here! I got the board printed and assembled one for testing last week. Been testing its features for several days now. I’m putting it in a nice IP66-rated enclosure for more tests and development.

This device is meant to be a one-stop-shop for data logging needs with minimal setup and learning time while my SDI-12 USB adapters are meant to add SDI-12 and other sensors to your existing datalogging PC or Raspberry Pi (assuming you’re willing to learn some Linux and the Pi).

For now, here are a couple of photos of just the board:

Board in my hand

Board with a 4-channel hi-res analog input extension board

Here is a list of features:

  • Open-source full-feature Python data logging script
  • SDI-12 translator with 4 SDI-12 sensor connectors¬†(more can be added)
  • 4 analog auto-scaling single-ended analog inputs up to 20uV precision
  • 2 analog auto-scaling differential analog inputs up to 20uV precision (alt. to 4-chn single-end)
  • Socket for optional 2G/3G/4G LTE/ZigBee/Digimesh RF modules
  • Resistance sensing with optional precision 1K or 10K pull-up resistors (0.1% 20ppm/degC)
  • Extension port allows up to 12 more precision analog and resistance sensors
  • Micro-SD card slot (up to 32GB of non-volatile program and data storage)
  • Data file download via FTP (Wi-Fi)
  • Hi-precision temperature-compensated Real Time Clock for time keeping (2-4 ppm)
  • Power cycle SDI-12 sensors to reset problematic SDI-12 sensors
  • Wi-Fi connection to upload data to servers (requires a Wi-Fi hotspot/router)
  • 4 analog single-ended analog inputs up to 5mV precision where less precision is acceptable
  • 4 digital on/off inputs (alt. to 4 5mV analog chn)
  • 4 digital pulse counters for rain gauges, anemometers etc. (alt. to 4 5vV analog chn)
  • Device temperature monitoring
  • Starter MicroSD card with logging script included
  • Xbee sleep to conserve power
  • Interactive sensor and logger setup interface
  • Optional water-proof enclosure
  • Optional wired adapter for more software development

I will add more details later.

SDI-12 USB adapter upgraded

2018-07-02 15.25.47

After some more firmware development and testing, I am happy to announce that the upgraded SDI-12 USB adapter is now available. The above is the first batch of these adapters and one hi-res analog input add-on board.

The upgraded board features the following:

  1. 4 analog inputs. 12-pole terminal block that features 4 analog voltage inputs. These are 10-bit or 5mV resolution inputs without differential reading. There are there to provide basic voltage inputs for projects that don’t require hi-res analog voltage inputs.
  2. Pulse counters. Alternatively, these 4 inputs can be used as pulse counters. Say you have a rain gauge or flow meter that outputs pulses, these pins can count the pulses. You may need additional filtering (one capacitor and two resistors) if the pulses are noisy. Each time you read the pulse counts, you get the counts since you last read and the adapter will start counting from zero again. This way, if you collect data every minute, the counts will be counts/min. Because each data point is accompanied with date-time information, you can always calculate the count rate with your data set.
  3. Extension port. There is now an 8-pin extension port for add-on boards. The first extension board I have designed and tested is a hi-res analog voltage input board. This board features the same four 16-bit auto-scaling inputs and differential inputs as the SDI-12 + Analog USB adapter, with an added benefit of address jumper. You can add as many as 4 such extension boards to the new SDI-12 USB adapter, with each extension board taking a different address. That is up to 16 hi-res voltage inputs.
  4. Serial port. There is also a serial port connector with RX, TX, etc. This port helps you connect the adapter to an arduino or a micropython board that don’t have USB connections but have serial ports.

With the added features, comes added costs of parts, quality checking, and development times. So I am currently offering two-tier pricing:

  1. The board with everything included and tested at $55
  2. The board without the 12-pole terminal block or these pins tested at $45
  3. If you need the serial port, I can solder the header and configure it free of charge.
  4. The hi-res analog add-on board is $35 each. If you want the 4 additional SDI-12 port soldered on it, I can do it for $5 extra.

The full-featured board has the additional terminal block and needs to be tested with all the analog pins to make sure they are all properly connected (reflowed). In the photo, the bread board and 4 blue potentiometer is the test rig I use to test hi-res analog inputs for the SDI-12 + Analog USB adapter. You CAN solder on your own 12-pole terminal block and test the pins yourself too. You will have to do a lot of screw/unscrew of a potentiometer or resistors though. The firmware is the same so once you solder the connector on, you can use the features.

2018-07-02 14.05.33

Although the analog add-on + new adapter costs the same as the original SDI-12 Analog USB adapter, the stack of two boards does add to its height so the board needs more space. My intention is to add flexibility to the adapter so I can later add more features to the ecosystem without scraping the existing devices. I did a custom board for someone that wants magnetometer, accelerometer, and gyroscope with SDI-12 sensors:

2018-06-17 12.15.47.jpg

I glued the sensor board on top of the adapter board and wired them together via the extension header. This way I didn’t have to spend time designing a new board, which will likely cost more time and money. Also, designing a new extension board is easier than designing a whole new adapter with the sensors on it. I can do more custom sensor boards even if there is only market for a few.

The purchase links to the full-featured adapter and the hi-res add-on board have not been established yet. I’ll get them up and running on my blog and at inmojo soon.

New data logging script

multiple sensor and adapter deploymentThe SDI-12 USB adapter data logging script has been updated to version 1.6.0. Here are the new features:

  • New configuration file format. I went from a basic format that is essentially the key strokes of the interactive inputs to a more human-readable, expandable, and future-proof dictionary-type of format.
  • The serial port is remembered in the configuration file by a unique ID instead of device name such as COM3. This makes it impossible to open the wrong serial port. You can also copy your configuration files to a different OS type and expect it to still work.
  • The data file name now includes the configuration file’s name, such as Liudr.cfg_20180709.csv. This makes it possible to distinguish different data that are collected using different configurations.
  • You can specify a different configuration file name in command line instead of the default. This makes it possible to run multiple adapters on the same system, each running its own configuration file. You can also set up different data logging scenarios and store them in different configuration files for easy recall.
  • There are also stability updates that prevent the script from quitting if, say, the sensor fails to return any valid response. Instead, the logger waits until the next data point and try again.
  • The logging script can now use multiple sensing commands per sensor, such as M!, M1!, M2! etc. besides reading multiple sensors. The interactive input has changed a bit to accommodate the upgrade.
  • Now if you run the script for the first time, it takes you to the interactive inputs to collect configuration parameters to save them into a configuration file. Then when you run it the second time, it automatically loads the configuration file. If you wish to make minimal changes, you can just edit the configuration file. If you want to run the interactive inputs again, delete the configuration file and run the script again.

This version is still considered “experimental” while I hunt for potential problems. You can find the script under the download section along with “changes in V1.6.0” document:

https://liudr.wordpress.com/gadget/sdi-12-usb-adapter/

Let me know what you think!

Open FTDI USB-serial UART port by ID

In this post, I will explain how to open serial port to your Arduino or SDI-12 USB adapter by its unique ID so you always open the correct port even when there are multiple such devices on your computer or raspberry pi.

For Arduino and SDI-12 USB adapter users, I have a nice trick to help you manage multiple Arduinos or SDI-12 USB adapters on the same computer or raspberry pi. On raspberry pi, as on a typical linux system, your device shows up as a serial port, such as /dev/ttyUSB0. This serial port designation is usually bound by the order that the device is discovered at boot time, which may not be the same even if you keep your adapter plugged into the same USB port. This means if you have more than one device on your raspberry pi, you may open the wrong port at times, which should be a big issue. To prevent your program from opening the wrong port, you need a unique ID for each device. Luckily FTDI chips already come with unique IDs. We just have to find those IDs and possibly change them into more meaningful things for us to remember. Assume for the moment you are making a data logger for your test fields. There is one field that can be called “NORTH”. The following steps will help you change the ID of the FTDI chip on your device so you can later open its port by that ID, instead of a port name. Here is a list of which devices are using FTDI’s chips that have the reprogrammable ID feature:

  • Liudr SDI-12 USB adapter (all types)
  • Sparkfun Redboard
  • Certain Arduino clone boards
  • Lots of other devices such as GPS etc.

Here is the FT_PROG tool FTDI provides. It’s windows only but I’m sure you can find a windows machine to run it. I’ve not tested it in a virtual machine whose host is linux or macos. I’ll do that when I have more time. If you are unsure whether your device has an FTDI chip, a quick scan using the program will tell you.

http://www.ftdichip.com/Support/Utilities.htm#FT_PROG

First, press the scan icon (magnifying glass). If you have a device with FTDI chip, it will show up. See the screen grab below:

So I have an FT232R chip with a chip serial number “A106DHE5”. I can open port with this serial number but I’d rather change it to “NORTH”. Click on the “SerialNumber” from the left side.

Uncheck the “auto generate serial no” so you can edit the serial number to “NORTH”. You have up to 16 characters to name the adapter. Once done, press flash icon (thunder bolt).

Now that you have programmed your chip, you can read the information back using “scan” again to verify that the ID has changed:

Now that you have this nice ID, let’s open port by this ID.

Here is a small complication. On linux, the chip ID is returned, such as “NORTH”. On window, the port ID is returned, such as “NORTHA“. The addition of the “A” indicates the port “A” on chip “NORTH”. This is because some FTDI chips have two serial ports. The port IDs will be “NORTHA” and “NORTHB“. Even for FTDI chips that have only one port, such as for our case, the “A” is still there. So I recommend comparing chip ID instead of port ID. If you only work on linux/rpi systems, this doesn’t seem to concern you. But if you wish to make your code platform independent, i.e. running on windows without an incident, you will only extract whichever ID you receive with the stored chip ID, up to the length of the stored chip ID. Note: in the platform-independent code, you can’t slice a port’s serial_number with your stored ID because some internal ports don’t have serial numbers thus returns empty that will throw an error when you try to slice an empty array.

The following is a snippet that works ONLY on linux/rpi systems:


import serial.tools.list_ports  # For listing available serial ports
import serial  # For serial communication

my_ID='A817EQLG'
port_device=''
# List ports for user to select
a = serial.tools.list_ports.comports()
print('\nDetected the following serial ports:')
for w in a:
    print('Port:%s\tID#:=%s' % (w.device, w.serial_number))
    if (w.serial_number==my_ID): # Match ID with the correct port
        port_device=w.device # Store the device name to later open port with.
if len(port_device)!=0:
    print('\r\n%s is the correct port.' %(port_device))
else:
    print("Port with ID: %s is not found!" %(my_ID))

The following is a snippet that works on ALL OS:

import serial.tools.list_ports  # For listing available serial ports
import serial  # For serial communication

my_ID='A817EQLG'
port_device=''
# List ports for user to select
a = serial.tools.list_ports.comports()
print('\nDetected the following serial ports:')
for w in a:
    print('Port:%s\tID#:=%s' % (w.device, w.serial_number))
    if (w.serial_number.__str__()[:len(my_ID)]==my_ID): # Match ID with the correct port
        port_device=w.device # Store the device name to later open port with.
if len(port_device)!=0:
    print('\r\n%s is the correct port.' %(port_device))
else:
    print("Port with ID: %s is not found!" %(my_ID))

Your choice, simplicity of code or cross-platform compatibility.

Here is the cross-platform code’s result on my windows machine:

Detected the following serial ports:
Port:COM23	ID#:=A817EQLGA
Port:COM3	ID#:=None


COM23 is the correct port.

As you can see, there is an added “A” at the end of the ID reported by windows, which the python code ignored to produce a match.

I’ve also attempted to do this using Processing 3.0. Unfortunately, the Serial.getProperties() function that should return similar information returns blank (possibly not implemented on windows and yet to be tested on linux). If you have tested Processing method with success, please reply below with your results. I’ll add your comment to the post.

Here is the code I used in Processing 3.0:

import processing.serial.*;
import java.util.Map;
 
void setup() {
  String[] ports = Serial.list();
 
  for (int i=0; i < ports.length; i++) {
    Map<String, String> props = Serial.getProperties(ports[i]);
    print(ports[i]+": ");
    println(props);
  }
}

Results:

COM3: {}
COM23: {}
:(

Closing note: even if you work on Windows that assigns unique COM port number to your arduino or adapters, the assignment relies entirely on the currently available port numbers. If you develop your project on one windows PC and deploy on another windows PC, you WILL get different COM port numbers. On a Mac, the ID is embedded on the port name such as /dev/ttl.usbserial-A103RU9T so you are better off. But, will you be willing to shell out the money to get a mac and have it sit somewhere to collect data just because of this feature? If you are a linux wiz, you can bind names with serial numbers using some scripts. That’s beyond the scope of our general discussion, which assumes minimal experience with linux administration.

%d bloggers like this: