Open source data logger

I have been designing data logger for a number of years. This is my answer to lots of data logging needs. An Arduino Nano-based open source data logger:

ospl-th-on

The logger provides the following features (in green) including features of Arduino Nano (in black):

Microcontroller Atmel ATMEGA328P
Power 5 V via USB or 2X AA battery (internally)
Digital I/O 10 (4 PWM output, other Arduino pins used internally)
Analog Input 4 10-bit ADC (8 on ATMEGA328P, only 4 brought out)
DC Current per I/O Pin 40 mA max
Flash Memory 32 KB of which 2 KB used by bootloader
SRAM 2 KB
EEPROM 1 KB on ATMEGA328P, 32 KB on real-time clock breakout board
Clock Speed 16 MHz
MicroSD card 32 GB maximum
Real-time clock Temperature compensated (DS3231)
ADS1115 4-chn 16-bit differential ADC with up to 16X programmable gain
LCD 16 column by 2 row character LCD with back light on/off control
Input Rotary encoder with switch (when shaft is pressed)

Table. Specification of Arduino Nano and the rest of the modules.

Another photo:

red-version-assembled-lcd-removed

As you can see, the logger incorporates a number of breakout boards instead of including these ICs on a single circuit board. More to come…

Phi-shield revised and released

phi-3-shield-on-in-hand

It has been a while since I gave the phi-shield a major revision. I’ve been working on this for a while and now I am releasing the Phi-3 shield. This shield continues to support user interaction with LCDs and buttons. Here is a list of the features:

The following hardware are provided by the shield:

  • 20X4 LCD with back light on/off control
  • Six buttons (up/down/left/right/B/A)
  • Two LED indicators
  • Speaker
  • MicroSD card slot
  • Real-time clock (DS3231)
  • EEPROM (32KB 24LC256)
  • Connector for Adafruit Ultimate GPS module or Bluetooth module
  • Stacking headers for easy access to all pins.
  • Recessed board right edge for easy access to MEGA’s 18X2 pin headers on the right side.
  • Reset button

phi-3-shield-lcd-side-by-side

The following software functions are provided by various supporting libraries:

  • User-selectable menu (LCD + buttons)
  • Number and text entry (LCD + buttons)
  • Scrollable long text (LCD + buttons)
  • Date and time (DS3231 or GPS)
  • Location (GPS)
  • Data and configuration storage (MicroSD card and EEPROM)
  • Playing simple tones (speaker)
  • Indicators (LEDs)
  • Wireless connection (Bluetooth module)

phi-3-shield-lcd-removed-annotated

There are three tiers of Phi-3 shield kits: kit0, kit1, and kit2, none of which includes a GPS module. The kits are immediately available. Buttons with color caps as pictured will be included while supplies last.

Here is the Phi-3 shield’s own page. There are links on the page to make purchases. Or you can visit the BUY page to see what stores carry this shield.

Phi-3 shield

Video demonstrations will be available next week. Meanwhile, the support of Phi-2 shield will remain. If you need Phi-2 shields, I have them available.

phi-3-shield-bottom-rtc-lcd-wire-removed

Sensing temperature

Sensing temperature seems to be an easy enough task using Arduino etc. You take a thermistor that changes resistance with temperature, such as a 10K ohm thermistor, which has a nominal resistance of 10K ohm at 25DegC. You take a fixed-value resistor, say 10K ohm, and make a voltage divider with the thermistor. You use analog input to sense the voltage of the divider and calculate the resistance from the voltage, then temperature from resistance. You can call this fixed resistor a serial resistor since it is in series with the thermistor. The following diagram is a voltage divider. The R_known is the fixed-value resistor. The R_unknown is the thermistor. A 5V voltage is applied to the resistors and a “sense” pin is sensing the divider voltage.

voltage divider

Then why am I wasting time to reiterate this simple task? Consider, the fixed-value resistor is made of a carbon film. What does carbon do at different temperatures? It changes resistance! So the “fixed-value” resistor you have is no longer fixed in value, especially if you live in places with temperature extrema, like where I live. I can get below 30DegC and above 30DegC outdoors in winters and summers. When you consider temperature, you can easily span almost 60DegC temperature range from a 25DegC nominal room temperature, where the fixed resistance values are specified, to a -35DegC where your data logger is logging data. If your fixed-value resistor doesn’t come with a temperature coefficient, well, it should! You can assume maybe 200ppm/DegC. This means 200 parts per million change of resistance per degree C. With -35DegC at 60 DegC below nominal temperature, you are looking at:

200*60/1,000,000=0.012=1.2% resistance change

This change of resistance could result in several percent of temperature reading error. The exact relation requires some calculus maybe I can discuss if there is interest. So you can expect to have several DegC or more error.

Here is what I noticed by driving two cars:

They both exhibit this behavior: After I park the car outside for several hours in cold temperature, I start the car, the “outside temperature display” says it is something like 10 degF. After several minutes, the temperature starts to drop to lower values.

Car 1: there is a few DegF drop. Say it will say 10DegF at startup, then it will say 7DegF after a few minutes

Car 2: there is as much as 15 DegF drop. If is says 10DegF at startup, it may say -5DegF after a few minutes.

Car 2 is newer than Car 1 but both cars are newer cars.

So my thought, although not tested (that might require disassembling the car computer), is that, the newer car is using a lower-grade serial resistor with the temperature sensor (thermistor) than the older car. This serial resistor must be inside the cabin, on the car computer’s board. When it is cold, its value increases considerably to maybe a couple percent higher than the nominal value. This trend of increasing resistance with decreasing temperature is shared between carbon and semiconductor (thermistor). So the effect of thermistor increasing resistance with decreasing temperature is countered by the effect of carbon increasing resistance with decreasing temperature. Then once the engine warms up the cabin enough, the fixed resistor warms up and the temperature display changes.

So how do you counter this effect i.e. temperature coefficient of fixed resistor? You can buy better resistors with less temperature coefficient, such as 20ppm instead of 200ppm. You will drop that effect to 1/10, which will be able to provide you the right accuracy. But these resistors aren’t exactly cheap. For instance, the 10K resistor I used as serial resistor in one of my designs has 10ppm/DegC:

279-RN73CA-10K

This resistor is about a dollar each. It is not only low in temperature coefficient, but also high in precision, 0.1% (i.e. it is at most 0.1% off from 10K when measured at nominal temperature)

 

When I don’t need this precision, just need a ~10K pull-up resistor for reset pin, I use this one:

652-CRT0805FZ1002ELF

This resistor is only 20 cents but it’s not too bad. 1% precision and 50ppm/DegC. I suspect the car 2 has something like this or even a bit worse! A 5% precision 200ppm resistor is only 10 cents. What would I use if I needed to make lots of cars?!

Python code for multiple SDI-12 sensors

As you probably know, the SDI-12 sensor logger code in Python can only log one sensor at a time. It is not a hardware limitation. I wrote the logger code as an example of how to do logging with the SDI-12 adapters and Python. To make sure people don’t have the wrong ideas that you can ONLY get one sensor logged, I have been working on the logger code for the past couple of days and have increased the number of sensors from one to any number you need. The improvement is backward compatible with the configuration file for Raspberry Pi logging, in case you wonder. All that is changed to the user interface is the prompt:

Original prompt:

‘SDI-12 sensor address: (0-9, A-Z, a-z)’

New prompt:

‘Enter all SDI-12 sensor addresses, such as 1234:’

 

So if you have 4 sensors you want to log together, then just enter all their addresses in a string, such as 1234 and hit enter. All sensor inputs will be saved to log file and sent to sparkfun’s data server. The only limitation on the code now is the sparkfun data server stream. The server stream is set up to only take 6 values so the logger code will send the first 6 values from all sensors to the server. If you wish to lift this limitation, you should create your own stream and set up as many values per data point as you need, and modify the logger code (see the magic number 6?).

Below are some sample data logs:

2/3/2017  12:15:25 AM 1 1.11 26 z 5.09419 5.09381 0.24388 5.09419
2/3/2017  12:15:56 AM 1 1.11 26 z 5.09325 5.0925 0.24388 5.09306
2/3/2017  12:16:28 AM 1 1.11 26 z 5.09363 5.094 0.24375 5.09438
2/3/2017  12:17:02 AM 1 1.11 26 z 5.09194 5.09269 0.24375 5.09306

As you can see, the data are separated by sensor address. The address z is the analog-to-digital converter’s address for SDI-12 + Analog adapter. As you can see, my computer outputs 5.09V instead of the nominal 5V on its USB port.

Here is a link to the new logger code. Give it a try and let me know how you like it.

sdi_12_logger_v1_4_1.py

Phi-2 20X4 shield kits

This kit is nearly sold out. Due to Chinese New Year, all re-ordered boards and components are expecting long delays. If Inmojo.com and my paypal sales buttons indicate the kits are sold out, then the kits are indeed sold out. Please be patient and check back in a week or so. Sorry for the inconvenience.

SDI-12 concurrent measurement

If you have a few SDI-12 sensors and want to quickly read their values, you may need to perform concurrent measurement. In regular measurement, using an M command, your logger issues the M command and expects the sensor to respond with the number of seconds to wait until data are ready and the total number of values. Here is an example. Assume sensor address is 1. Green is logger and blue is sensor:

1M!

10054\r\n

1\r\n

As you see, the sensor 1 needs 005 seconds to perform the measurement, which returns 4 values. Only the sensor needs to send carriage return (\r) and new line (\n). If the sensor finishes measurement before 5 second expires, it issues a service request, i.e.  its address 1 and carriage return and new line. The logger will respond by asking for data once it receives the service request.

Once the service request is received, or the expected 5 seconds elapses, the logger will ask for data using a D0 command, i.e. D and zero. The sensor will respond with data:

1D0!

1+12.34+56.78-90.12+34.56\r\n

If you have multiple sensors, you need to repeat the process multiple times, waiting about 5 seconds for each sensor. That is a lot of time. If your sensor supports concurrent measurement, you can shorten your wait time to only 5 seconds instead of 5*(number of sensor) seconds. Here is the process using concurrent command C, assuming 5 sensors:

1C!

100504\r\n

2C!

200504\r\n

3C!

300504\r\n

4C!

400504\r\n

5C!

500504\r\n

There won’t be any service request issued and the logger can go ahead to talk with the next sensor as soon as the current sensor responds with number of seconds to wait and number of data points 04 it will return. Notice that the last TWO digits now indicate the number of data points it will return, not the single last digit.

Now the logger proceeds to wait for 5 seconds and then start asking for data the same way it does when it used M command:

1D0!

1+12.34+56.78-90.12+34.56\r\n

2D0!

2+12.34+56.78-90.12+34.56\r\n

3D0!

3+12.34+56.78-90.12+34.56\r\n

4D0!

4+12.34+56.78-90.12+34.56\r\n

5D0!

5+12.34+56.78-90.12+34.56\r\n

Since all sensors are taking measurement almost simultaneously, you don’t have to wait for them individually, saving a lot of time.

Don’t stop here. Sometimes concurrent measurement is SLOWER than reading one sensor at a time! This only applies to faster sensors, such as Decagon 5TM soil temperature and volumetric water content sensor. It reports that it needs 1 second but it requires significantly less than 1 second. So if you use concurrent measurement, you will be forced to wait for a whole second. SDI-12 sensors are intended to be low-power but not for speed. The 1200 baud rate translates into about 120 characters per second, shared between the logger and sensor. The time to send the commands and data back and forth between a single sensor and the logger could take half a second to do!

Procedure to produce small batches of devices

Over the past years, I’ve designed and built devices in small batches such as 10-25 units at a time. I’ve gained some experience doing such small builds. The following is a procedure for building a small batch SDI-12 + Analog USB devices.

The device has a USB serial converter IC FT232RL, an MCU ATMEGA328P (same as Arduino Uno’s MCU), an ADS1115 16-bit analog-to-digital converter, precision resistors, fuse, screw terminal blocks, jumpers etc.

I build the device first by reflow soldering the surface components, and then hand solder the thru-hole components once I’m done confirming the reflow soldered components are working. You’ll see why these devices can’t be sold at $4 a piece, not just because the components cost more from legitimate vendors such as digikey, Mouser, Newark, etc. Most of the steps can’t be completed by someone without proper training of what the device does.

Here it goes. Any recommendation? Post a response!

Manufacturing and inspection procedure

Date: 2016-11-23

1.    Preparation and printing solder paste:

  1. Remove solder paste from refrigerator and set it in room temperature for one hour.
  2. Clean board with alcohol and lint-free wipe.
  3. Visually inspect board for defects and reject boards with visual defects.
  4. Align board with stencil.
  5. Print solder paste.
  6. Check registration especially for ADC, MCU and FTDI chips. Reprint in case registration is too far off.
  7. Wash and dry stencil.
  8. Return solder paste to refrigerator.

 

2.    Components placement:

  1. Place MCU. Check alignment with 3X loupe.
  2. Place FTDI chip. Check alignment with 3X or 10X loupe.
  3. Place ADC. Check alignment with 10X loupe.
  4. Place (5) 100nF caps.
  5. Place (1) 10uF cap.
  6. Place (2) 22pF caps.
  7. Place (1) fuse.
  8. Place (1) EMI bead.
  9. Place (3) 10K resistors.
  10. Place (1) 100K resistor.
  11. Place (4) 1K precision resistors.
  12. Place (4) 10K precision resistors.
  13. Place (1) crystal.
  14. Place (1) mini-USB connector.
  15. Check resistor values with loupe. Replace wrong resistors.

 

3.    Reflow soldering:

  1. Place boards on reflow over racks.
  2. Attach thermal couple to a plated hole on one board.
  3. Turn on temperature gauge.
  4. Connector reflow oven to power.
  5. Follow reflow specifics, make sure to time each step with stopwatches. STOP if expected temperature is not reached within expected time.
  6. Disconnect reflow over from power.
  7. Remove boards and cool them off.

 

4.    Visual inspection:

  1. Visually check for solder issues on chip resistors, caps and other components.
  2. Check for solder bridges on ICs with a 10X loupe.
  3. Check USB connector for bridges.

Found any issues?

  • Y: fix with flux paste, solder braid, and iron
  • N: proceed

 

5.    Flash firmware:

  1. Connector programmer to PC.
  2. Place board on pogo pins.

Programmer detected board?

  • Y: flash firmware

MCU detected after flash?

  • Y: proceed
  • N: check soldering near XTAL pins
  • N: inspect MCU for solder issues and other issues (IC in wrong orientation etc.). Add flux paste and reflow pins.

 

6.    USB connection and initial ADC check:

  1. Connect board to PC.

USB port detected?

  • Y: proceed

Device responding to ID command?

  • Y: proceed
  • N: check MCU and FTDI chip for solder issues

Device returning analog measurements?

  • Y: proceed
  • N: check ADC and MCU I2C pins for solder issues
  • N: inspect connector and FTDI chip for solder issues

 

7.    Check ADC and resistors:

  1. Solder headers and connectors.
  2. Connect different resisters to analog pins.
  3. Set HIGH jumpers on all channels.
  4. Read single-ended inputs.

Results are as expected?

Command ‘zI!’ should return ‘z13Liudr   SDITRD130\r\n’. Command ‘zM!’ should return ‘z0014\r\nz\r\n’. Then command ‘zD0!’ should return ‘z+0.xxxxx+0.xxxxx+0.xxxxx+0.xxxxx ‘ where x is arbitrary but the first x in each number is unlikely going to change if you poll again. Command ‘zM1!’ should return ‘z0012\r\nz\r\n’. Then command ‘zD0!’ should return ‘z+0.000xx+0.000xx ‘ where x is arbitrary and positive signs may be negative signs at times but both values should be very small with at least 3 zeros after decimal.

  • Y: proceed
  • N: check resistors and ADC for solder issue

 

  1. Set LOW jumpers on all channels.
  2. Read single-ended inputs.

Results are as expected?

  • Y: proceed
  • N: check resistors and ADC for solder issue

 

8.    Check SDI-12 bus:

  1. Connect SDI-12 sensor to one SDI-12 terminal.
  2. Poll SDI-12 sensor.

Does sensor return values?

Command ‘1M!’ should return ‘10012\r\n1\r\n’ Then command ‘1D0!’ should return ‘1+1.xx+2x.x’ where x is arbitrary but unlikely to change if you poll again. The second number is temperature so indoor reading should be above 20.0 C.

  • Y: Device passed QC test J
  • N: check MCU SDI-12 bus pin for solder issues

Teaser photo

What can this new board do?

2016-11-22-11-08-48

Guesses? Comments? Answer revealed after US Thanksgiving holiday (2016-11-24)!

(There is nothing on the back side)

Read analog sensors on SDI-12 USB + Analog adapter

Reading analog sensors are easy. The adapter has SDI-12 address of ‘z’, lower case. So reading the analog sensor just involves querying the SDI-12 address ‘z’. There are two sensing modes: single-ended, and differential. If you have mixed single-ended and  differential channels, read single-ended, then differential. Discard channels you don’t need. It won’t hurt the sensors or the adapter if you wire them in differential mode but read in single-ended mode. The reverse is also true.

The sensing commands are ‘zM!’ for single-ended readings, and ‘zM1!’ for differential readings.

In both modes, you use ‘zD01!’ zee-Dee-zero-!, to get data. Essentially, the adapter itself is an SDI-12 sensor that reports 2 or 4 values, depending on sensing mode. This makes it very easy to integrate analog sensors into your existing data logger that is based on the original SDI-12 USB adapter. It is still advantageous to keep the original SDI-12 USB adapter so it can split SDI-12 sensors with the SDI-12 + Analog adapter. In case one SDI-12 sensor gets broken and interferes with the rest of the sensors on that adapter, the SDI-12 sensors on the other adapter will be unaffected.

To make this complete, the SDI-12 USB + Analog adapter also responds to the following commands:

Command:’z!’

Response: ‘z\r\n’ This means that the adapter is responding to queries.

Command: ‘z!’

Response: ‘z13Liudr   SDITRD130\r\n’ This indicates that the firmware is in version 1.3.0.

Command: ‘zM!’

Response: ‘z0014\r\nz\r\n’ This means that the adapter needs 1 second to acquire 4 single-ended auto-scale analog values. The second ‘z’ indicates it completed the acquisition.

Command: ‘zM1!’

Response: ‘z0012\r\nz\r\n’ This means that the adapter needs 1 second to acquire 2 differential auto-scale analog values. The second ‘z’ indicates it completed the acquisition.

C0mmand: ‘zD0!’

Response: ‘z+1.23456+2.34567+3.45678+4.56789\r\n’ or ‘z+1.23456+2.34567\r\n’ These are single-ended or differential channel readings, depending on whether M or M1 was issued before D0.

 

Update on the SDI-12 + Analog USB adapter

sdi-12-usb-analog

Here is an update:

In case you wonder what all those green screw terminal blocks are doing, here is a graphical explanation:

Both the SDI-12 USB and SDI-12 + Analog USB are explained in this illustration.

To maintain the same compact size, I printed all the pin information on the bottom of the board again. So if you don’t know what a certain pin on a block does, just flip it around and you’ll see it. The jumper information is all on top side.

The SDI-12 + Analog USB adapter comes with a jumper to select either internal 5V or external voltage at the Ext. Power screw terminal block. You may connect a small 9V battery to the Ext. Power screw terminal block. You can also connect your  12V battery that powers your logger to this pin. The external power is only sent to the SDI-12 sensors. It’s not powering the adapter or sent to the analog inputs’ “+” connections. Those “++ connections are always from the 5V USB power. There are 3 pins on the terminal block and the center pin is not connected to anything. It makes it easier to separate the + and – of the external power and I don’t have to source 2-pole blocks besides 3-pole blocks.

All four SDI-12 blocks have “+ S -“. The “+” is either USB 5V or external power depending on the power jumper. “S” is SDI-12 signal. “-” is ground. All grounds should be connected together. These four blocks are all connected. They are not four separate buses. There is no way to transparently bridge one USB serial port to more than one SDI-12 bus. If you wish separate SDI-12 bus for each sensor, which is unnecessary, get a separate adapter for a separate SDI-12 bus. This need for separate SDI-12 bus may come from some suspicion that if a single SDI-12 sensor breaks, it may take the whole bus down with it. I have not been so unfortunate and broken SDI-12 sensors I have had didn’t affect good ones. In any case, a broken sensor needs replacement. Unless you deploy redundant sensors one set on each SDI-12 bus, you are OK with a single adapter that bridges a single SDI-12 bus for all sensors.

The four analog channels are as accurate as 0.02mV when the signal is small, below 0.256V. The adapter automatically uses the best scale to determine the signal. The highest signal allowed is 6.144V on any channel. There are 6 ranges (gain levels), with maximal ranges of 6.144V, 4.096V, 2.048V, 1.024V, 0.512V and 0.256V. Within each range of voltage, the analog input is turned into a numerical value between 0 and 32767. So if you have a signal that is 0.1V, using the largest range of 6.144V will give an smallest change of 0.1875mV. This sounds very accurate, because this change is 0.1% of the signal. But the real resolution of the ADC is not the smallest change. It is usually many times that. Plus there is fluctuation in supply voltage and noise in the signal. The result is likely in the neighborhood of 2mV. This becomes 2% of the signal magnitude. But if you use the 0.256V range, its smallest change is 0.0078125mV. The accuracy is about 0.02mV to be conservative. Since SDI-12 standard has no way to change scale, the adapter does it automatically.

The auto scale is done with a 10%~90% range. The adapter starts with the largest scale to protect the converter and reads the signal. It then calculates the smallest scale that will fit the signal within 10%-90% of the scale. It reads at this scale and returns the value. Each channel is auto scaled independently from the other channels so you may have some larger signals automatically read at a larger scale and smaller signals automatically read at a smaller scale.

The meaning of single-ended channel is that each one of the four channels is read against the common ground. This is less accurate for small signals over long wires. If you have a pyronometer or some other small voltage signal sensor, you may want to use two channels in differential mode. In this mode, the “+” wire is connected to say channel 0, and the “-” wire is connected to channel 1. The difference between these two are read and the difference may either be positive or negative. Range of the difference between these wires can be +-6.144, … +-0.256Vetc.

(to be continued)

 

%d bloggers like this: