SDI-12 USB adapter manual and logging script updated

Due to the discontinuation of data.sparkfun.com, I moved data logging to ThingSpeak.com

I have other updates that I rolled up in the manual, such as more details on telemetry. New manual is posted on SDI-12 USB adapter page as well as the updated data logging code. Here is a snapshot of data I logged to ThingSpeak.com:

The full data stream is here:

https://thingspeak.com/channels/359964

Upload data to ThingSpeak

I have been using sparkfun’s Phant server for data upload and retrieval for a few years since they started the service. The service was easy to use and was free. Several months back they discontinued the service, unfortunately. I started looking for suitable alternatives.

Sparkfun recommended three, ThingSpeak, Cayenne, and Blynk. I went ahead and did some research on these services. My goal is to be able to log data online and later retrieve them and possibly visualize them using google charts, like before. I am not at the moment interested in automating my home with actuators or smart phone apps to turn on and off my hall lights. Here are my findings and why I decided that ThingSpeak was the best fit. If you are logging data for later processing or visualization, read on.

ThingSpeak.com

This service is provided by the company behind Matlab. I am not a fan of expensive commercial data manipulation tools such as Matlab but they do have a fair amount of business between universities and industry so their service might be a safer way to go against sudden discontinuation of service such as sparkfun. Basically you create a data stream and send data to the stream. It’s very similar to sparkfun. You can also retrieve your data, possibly good for running your data through google chart. They also provide some basic graphs and matlab analysis tools that I have yet tried.

There are two types of application programming interfaces (APIs) you can use: a REST API, and an MQTT API.

The REST API is based on HTTP so it’s very similar to existing services elsewhere. You use HTTP GET or POST command to send data using an API key, like a private key with sparkfun. Your data are limited to up eight values per data point. If you need more, then you need to create more streams. They also have a bulk update feature that you can use to send multiple sets of data instead of one set of data. This method allows a device to collect data and sleep in between data points. Then when it collects a fair amount of data, it connects to the Internet and sends all data in one shot. It saves power and network bandwidth. You can also create and modify the properties of streams with this API.

With the MQTT API, the underlying protocol is TCP/IP. There is no acknowledgement of data received and it is intended for low-powered devices to just wake up, take data, send it out, and go back to sleep. From my tests, data sent via this method were lost over 50% of the time. Unless future holds differently, I am not recommending this API.

Getting start is easy with ThingSpeak. Just set up an account and follow their tutorial to create a new data stream. Then the following bash code should get you started posting code:


curl "https://api.thingspeak.com/update/?api_key=your_appkey&field1=1.23&field2=2.34"

You can post any number of field values between 1 and 8. You will receive a zero as a positive response. Then you will see your results like the plot on the top of this page.

I have updated my Python data logging code to use ThingSpeak.com instead of the now discontinued data.sparkfun.com. The plot in this post is from Thingspeak.com. Here is a link to the data stream:

https://thingspeak.com/channels/359964

SDI-12 + GPS USB adapter

After a final revision, I am happy to release the SDI-12 GPS USB adapter! This adapter is the latest one to add to the line of SDI-12 USB adapters. In August 2015, I released my first SDI-12 USB adapter with this post. It was an idea that I thought about while traveling. I was working on data logger designs that use SDI-12 sensors and felt that interacting with SDI-12 sensors is not easy for agricultural or water resource researchers. Having an adapter that connects a computer to an SDI-12 sensor and reads measurements directly from the sensor would be very useful. So I made the adapter to simplify lab tests and data logger deployments. Since then, I’ve written free Python scripts for basic data logging (read the SDI-12 USB adapter main page). The demand for the adapter since then has been high enough to support my continued update on the data logging script, expanding from PC/Mac/Linux to single-board computers such as Raspberry Pi and Beagle Bone Bone. I have also expanded the adapter with an SDI-12 + Analog USB adapter that includes four high-precision analog inputs.

Later I found some need to add GPS modules to the existing SDI-12 USB adapter so that mobile data loggers such as those mounted on tractors will be able to produce with Geo-tagged data that can be made into maps. After some initial struggle using the new ATMEGA328PB processor that sports two hardware serial ports (one to talk to PC and the other with GPS), I realized that the GPS module actually interfered with the processor and caused program freeze-up. Then I made some hardware revisions and was able to prevent interference. It turned out that the new ATMEGA328PB processor that I used in my initial prototype was especially susceptible to interference when I used its second hardware serial port that have the same pins as the SPI pins that program the processor. So I switched to the ATMEGA1284P processor that I have been using on my open source physics laboratory design.

After extensive tests, I am happy to add this adapter to the product line. You can purchase (small quantity at the moment) at inmojo.com or on my blog (in the middle of the page). The adapter requires a separate purchase of the GPS module that Adafruit makes and sells, the Ultimate GPS module part number 746. You only need to solder four pins on the GPS module, the TX, RX, GND, and VIN, and the same pins on the adapter. Since the GPS module is relatively expensive, I can’t stock them up. But if you really need it assembled, you may have a GPS unit sent to me and a few extra dollars for assembly and testing. Just contact me once you make a purchase if you want assembly.

Free assistance on data logger projects

Summer is finally coming to my backyard and my spring semester is coming to an end. Thinking ahead (skipping over all the final papers to grade), with the whole summer ahead of me, starting 5/15/17, I can provide some free assistance to those that are working on your data logger projects using my devices, such as the SDI-12 data logging shield and SDI-12 USB adapters.

My goal is to get you started so you can quickly work on your own after my help. I’ve used Teamviewer to remotely help people install software, test their adapters with their own sensors, and modified my Python data logging code in the past. As long as I have some time to spare, I am willing to keep providing help. I appreciate it if you could help me spread the word. I might ask you to provide a blurb such as what sensors you use and what type of project you are working on etc. as a form of exchange for my free help.

SDi-12 + GPS USB adapter test

I was able to perform some tests on the new SDI-12 + GPS USB adapter. I don’t have the GPS module but do have an arduino shield that features the same GolbalTol GPS module so I used some jumper wires to connect the GPS to the adapter. I did tests last night and overnight. Things are looking good. Here are some results:

Commands:

To get longitude and latitude, you will issue “zM!”. The return values are z(long)(lat)\r\n. The longitude and latitude are both in standard NMEA format of 100*(degree.minute). For instance, a longitude of -9412.3411 means -(94 degrees 12.3411 minutes).

To get day, month, and year, you will issue “zM1!”. The return value is again in standard NMEA format of +DDMMYY. For example, a date of +190317 means the 19th of March, 2017.

To get hour, minute, and second, you will issue “zM2!”. The return value is also in standard NMEA format of +hhmmss. For example, a time of +123507 means 12:35:07 in 24hr style so it is 12:35:07 PM for those that use 12hr style.

Sample commands (in red) and returns (in green):

Single-sensor measurement:

zM!
z0012
z
zD0!
z-09456.1234+4578.9012
zM1!
z0011
z
zD0!
z+190317
zM2!
z0011
z
zD0!
z+065402

Concurrent measurement:

zC!
z00102
zD0!
z-09456.1234+4578.9012
zC1!
z00101
zD0!
z+190317
zC2!
z00101
zD0!
z+065713

SDI-12 + GPS module

After some development, I am glad to show a prototype of an SDI-12 + GPS USB module. This module incorporates the following features:

  1. USB connection
  2. SDI-12 translator with 4 SDI-12 connections (on a single SDI-12 bus)
  3. Header for a GPS module
  4. External power connection for sensors that need more than 5V from USB
  5. External power/5V USB selection jumper
  6. You can also use other serial devices or sensors such as Maxbotix serial sonic ranger, with some modification to the firmware
  7. Both SDI-12 senors and GPS are addressed like SDI-12 sensors, for easy integration of GPS signal into your existing SDI-12 logging scripts

Here is a picture:

I ran out of GPS modules. New ones are on the way. Once I get them, I’ll solder one on an adapter and do a demo video.

Open source data logger videos

Open source data logger videos:

Quick demo:

Features introduction 1,2,3

Assembling the logger

 

Open source data logger

I have been designing data logger for a number of years. This is my answer to lots of data logging needs. An Arduino Nano-based open source data logger:

ospl-th-on

The logger provides the following features (in green) including features of Arduino Nano (in black):

Microcontroller Atmel ATMEGA328P
Power 5 V via USB or 2X AA battery (internally)
Digital I/O 10 (4 PWM output, other Arduino pins used internally)
Analog Input 4 10-bit ADC (8 on ATMEGA328P, only 4 brought out)
DC Current per I/O Pin 40 mA max
Flash Memory 32 KB of which 2 KB used by bootloader
SRAM 2 KB
EEPROM 1 KB on ATMEGA328P, 32 KB on real-time clock breakout board
Clock Speed 16 MHz
MicroSD card 32 GB maximum
Real-time clock Temperature compensated (DS3231)
ADS1115 4-chn 16-bit differential ADC with up to 16X programmable gain
LCD 16 column by 2 row character LCD with back light on/off control
Input Rotary encoder with switch (when shaft is pressed)

Table. Specification of Arduino Nano and the rest of the modules.

Another photo:

red-version-assembled-lcd-removed

As you can see, the logger incorporates a number of breakout boards instead of including these ICs on a single circuit board. More to come…

Sensing temperature

Sensing temperature seems to be an easy enough task using Arduino etc. You take a thermistor that changes resistance with temperature, such as a 10K ohm thermistor, which has a nominal resistance of 10K ohm at 25DegC. You take a fixed-value resistor, say 10K ohm, and make a voltage divider with the thermistor. You use analog input to sense the voltage of the divider and calculate the resistance from the voltage, then temperature from resistance. You can call this fixed resistor a serial resistor since it is in series with the thermistor. The following diagram is a voltage divider. The R_known is the fixed-value resistor. The R_unknown is the thermistor. A 5V voltage is applied to the resistors and a “sense” pin is sensing the divider voltage.

voltage divider

Then why am I wasting time to reiterate this simple task? Consider, the fixed-value resistor is made of a carbon film. What does carbon do at different temperatures? It changes resistance! So the “fixed-value” resistor you have is no longer fixed in value, especially if you live in places with temperature extrema, like where I live. I can get below 30DegC and above 30DegC outdoors in winters and summers. When you consider temperature, you can easily span almost 60DegC temperature range from a 25DegC nominal room temperature, where the fixed resistance values are specified, to a -35DegC where your data logger is logging data. If your fixed-value resistor doesn’t come with a temperature coefficient, well, it should! You can assume maybe 200ppm/DegC. This means 200 parts per million change of resistance per degree C. With -35DegC at 60 DegC below nominal temperature, you are looking at:

200*60/1,000,000=0.012=1.2% resistance change

This change of resistance could result in several percent of temperature reading error. The exact relation requires some calculus maybe I can discuss if there is interest. So you can expect to have several DegC or more error.

Here is what I noticed by driving two cars:

They both exhibit this behavior: After I park the car outside for several hours in cold temperature, I start the car, the “outside temperature display” says it is something like 10 degF. After several minutes, the temperature starts to drop to lower values.

Car 1: there is a few DegF drop. Say it will say 10DegF at startup, then it will say 7DegF after a few minutes

Car 2: there is as much as 15 DegF drop. If is says 10DegF at startup, it may say -5DegF after a few minutes.

Car 2 is newer than Car 1 but both cars are newer cars.

So my thought, although not tested (that might require disassembling the car computer), is that, the newer car is using a lower-grade serial resistor with the temperature sensor (thermistor) than the older car. This serial resistor must be inside the cabin, on the car computer’s board. When it is cold, its value increases considerably to maybe a couple percent higher than the nominal value. This trend of increasing resistance with decreasing temperature is shared between carbon and semiconductor (thermistor). So the effect of thermistor increasing resistance with decreasing temperature is countered by the effect of carbon increasing resistance with decreasing temperature. Then once the engine warms up the cabin enough, the fixed resistor warms up and the temperature display changes.

So how do you counter this effect i.e. temperature coefficient of fixed resistor? You can buy better resistors with less temperature coefficient, such as 20ppm instead of 200ppm. You will drop that effect to 1/10, which will be able to provide you the right accuracy. But these resistors aren’t exactly cheap. For instance, the 10K resistor I used as serial resistor in one of my designs has 10ppm/DegC:

279-RN73CA-10K

This resistor is about a dollar each. It is not only low in temperature coefficient, but also high in precision, 0.1% (i.e. it is at most 0.1% off from 10K when measured at nominal temperature)

 

When I don’t need this precision, just need a ~10K pull-up resistor for reset pin, I use this one:

652-CRT0805FZ1002ELF

This resistor is only 20 cents but it’s not too bad. 1% precision and 50ppm/DegC. I suspect the car 2 has something like this or even a bit worse! A 5% precision 200ppm resistor is only 10 cents. What would I use if I needed to make lots of cars?!

Python code for multiple SDI-12 sensors

As you probably know, the SDI-12 sensor logger code in Python can only log one sensor at a time. It is not a hardware limitation. I wrote the logger code as an example of how to do logging with the SDI-12 adapters and Python. To make sure people don’t have the wrong ideas that you can ONLY get one sensor logged, I have been working on the logger code for the past couple of days and have increased the number of sensors from one to any number you need. The improvement is backward compatible with the configuration file for Raspberry Pi logging, in case you wonder. All that is changed to the user interface is the prompt:

Original prompt:

‘SDI-12 sensor address: (0-9, A-Z, a-z)’

New prompt:

‘Enter all SDI-12 sensor addresses, such as 1234:’

 

So if you have 4 sensors you want to log together, then just enter all their addresses in a string, such as 1234 and hit enter. All sensor inputs will be saved to log file and sent to sparkfun’s data server. The only limitation on the code now is the sparkfun data server stream. The server stream is set up to only take 6 values so the logger code will send the first 6 values from all sensors to the server. If you wish to lift this limitation, you should create your own stream and set up as many values per data point as you need, and modify the logger code (see the magic number 6?).

Below are some sample data logs:

2/3/2017  12:15:25 AM 1 1.11 26 z 5.09419 5.09381 0.24388 5.09419
2/3/2017  12:15:56 AM 1 1.11 26 z 5.09325 5.0925 0.24388 5.09306
2/3/2017  12:16:28 AM 1 1.11 26 z 5.09363 5.094 0.24375 5.09438
2/3/2017  12:17:02 AM 1 1.11 26 z 5.09194 5.09269 0.24375 5.09306

As you can see, the data are separated by sensor address. The address z is the analog-to-digital converter’s address for SDI-12 + Analog adapter. As you can see, my computer outputs 5.09V instead of the nominal 5V on its USB port.

Here is a link to the new logger code. Give it a try and let me know how you like it.

sdi_12_logger_v1_4_1.py

%d bloggers like this: