Open FTDI USB-serial UART port by ID

In this post, I will explain how to open serial port to your Arduino or SDI-12 USB adapter by its unique ID so you always open the correct port even when there are multiple such devices on your computer or raspberry pi.

For Arduino and SDI-12 USB adapter users, I have a nice trick to help you manage multiple Arduinos or SDI-12 USB adapters on the same computer or raspberry pi. On raspberry pi, as on a typical linux system, your device shows up as a serial port, such as /dev/ttyUSB0. This serial port designation is usually bound by the order that the device is discovered at boot time, which may not be the same even if you keep your adapter plugged into the same USB port. This means if you have more than one device on your raspberry pi, you may open the wrong port at times, which should be a big issue. To prevent your program from opening the wrong port, you need a unique ID for each device. Luckily FTDI chips already come with unique IDs. We just have to find those IDs and possibly change them into more meaningful things for us to remember. Assume for the moment you are making a data logger for your test fields. There is one field that can be called “NORTH”. The following steps will help you change the ID of the FTDI chip on your device so you can later open its port by that ID, instead of a port name. Here is a list of which devices are using FTDI’s chips that have the reprogrammable ID feature:

  • Liudr SDI-12 USB adapter (all types)
  • Sparkfun Redboard
  • Certain Arduino clone boards
  • Lots of other devices such as GPS etc.

Here is the FT_PROG tool FTDI provides. It’s windows only but I’m sure you can find a windows machine to run it. I’ve not tested it in a virtual machine whose host is linux or macos. I’ll do that when I have more time. If you are unsure whether your device has an FTDI chip, a quick scan using the program will tell you.

http://www.ftdichip.com/Support/Utilities.htm#FT_PROG

First, press the scan icon (magnifying glass). If you have a device with FTDI chip, it will show up. See the screen grab below:

So I have an FT232R chip with a chip serial number “A106DHE5”. I can open port with this serial number but I’d rather change it to “NORTH”. Click on the “SerialNumber” from the left side.

Uncheck the “auto generate serial no” so you can edit the serial number to “NORTH”. You have up to 16 characters to name the adapter. Once done, press flash icon (thunder bolt).

Now that you have programmed your chip, you can read the information back using “scan” again to verify that the ID has changed:

Now that you have this nice ID, let’s open port by this ID.

Here is a small complication. On linux, the chip ID is returned, such as “NORTH”. On window, the port ID is returned, such as “NORTHA“. The addition of the “A” indicates the port “A” on chip “NORTH”. This is because some FTDI chips have two serial ports. The port IDs will be “NORTHA” and “NORTHB“. Even for FTDI chips that have only one port, such as for our case, the “A” is still there. So I recommend comparing chip ID instead of port ID. If you only work on linux/rpi systems, this doesn’t seem to concern you. But if you wish to make your code platform independent, i.e. running on windows without an incident, you will only extract whichever ID you receive with the stored chip ID, up to the length of the stored chip ID. Note: in the platform-independent code, you can’t slice a port’s serial_number with your stored ID because some internal ports don’t have serial numbers thus returns empty that will throw an error when you try to slice an empty array.

The following is a snippet that works ONLY on linux/rpi systems:


import serial.tools.list_ports  # For listing available serial ports
import serial  # For serial communication

my_ID='A817EQLG'
port_device=''
# List ports for user to select
a = serial.tools.list_ports.comports()
print('\nDetected the following serial ports:')
for w in a:
    print('Port:%s\tID#:=%s' % (w.device, w.serial_number))
    if (w.serial_number==my_ID): # Match ID with the correct port
        port_device=w.device # Store the device name to later open port with.
if len(port_device)!=0:
    print('\r\n%s is the correct port.' %(port_device))
else:
    print("Port with ID: %s is not found!" %(my_ID))

The following is a snippet that works on ALL OS:

import serial.tools.list_ports  # For listing available serial ports
import serial  # For serial communication

my_ID='A817EQLG'
port_device=''
# List ports for user to select
a = serial.tools.list_ports.comports()
print('\nDetected the following serial ports:')
for w in a:
    print('Port:%s\tID#:=%s' % (w.device, w.serial_number))
    if (w.serial_number.__str__()[:len(my_ID)]==my_ID): # Match ID with the correct port
        port_device=w.device # Store the device name to later open port with.
if len(port_device)!=0:
    print('\r\n%s is the correct port.' %(port_device))
else:
    print("Port with ID: %s is not found!" %(my_ID))

Your choice, simplicity of code or cross-platform compatibility.

Here is the cross-platform code’s result on my windows machine:

Detected the following serial ports:
Port:COM23	ID#:=A817EQLGA
Port:COM3	ID#:=None


COM23 is the correct port.

As you can see, there is an added “A” at the end of the ID reported by windows, which the python code ignored to produce a match.

I’ve also attempted to do this using Processing 3.0. Unfortunately, the Serial.getProperties() function that should return similar information returns blank (possibly not implemented on windows and yet to be tested on linux). If you have tested Processing method with success, please reply below with your results. I’ll add your comment to the post.

Here is the code I used in Processing 3.0:

import processing.serial.*;
import java.util.Map;
 
void setup() {
  String[] ports = Serial.list();
 
  for (int i=0; i < ports.length; i++) {
    Map<String, String> props = Serial.getProperties(ports[i]);
    print(ports[i]+": ");
    println(props);
  }
}

Results:

COM3: {}
COM23: {}
:(

Closing note: even if you work on Windows that assigns unique COM port number to your arduino or adapters, the assignment relies entirely on the currently available port numbers. If you develop your project on one windows PC and deploy on another windows PC, you WILL get different COM port numbers. On a Mac, the ID is embedded on the port name such as /dev/ttl.usbserial-A103RU9T so you are better off. But, will you be willing to shell out the money to get a mac and have it sit somewhere to collect data just because of this feature? If you are a linux wiz, you can bind names with serial numbers using some scripts. That’s beyond the scope of our general discussion, which assumes minimal experience with linux administration.

Can’t upgrade pyserial in latest raspbian distribution?

This is just for your information if you are a Raspberry Pi user and playing with Python code from my blog. If you are trying to use the latest distro of raspbian with pyserial for some serial port project, you may have come across this issue that regardless how you upgrade pyserial using pip3, your python3 will always call up the old pyserial 2.6 that came with the distribution. I am a bit disappointed that the foundation has included such an old version of pyserial, couldn’t they just try a pyserial 3.0 instead? My solution was to remove the python3-serial module using apt-get and then install pyserial 3.3 using pip3.

sudo apt-get remove python3-serial
sudo pip3 install pyserial

Hope this helps.

Saving data to sparkfun server

If you’ve been following my posts, I discussed how to use my SDI-12 USB adapter and raspberry pi (or PC) to log data from SDI-12 sensors. Now that you have your data, you can also send them to sparkfun’s data server for storage and later retrieval. In the following video, I explained what you can do with this feature included in my data logger python script. I also constructed a sample webpage to display data saved on the server. Lots of customization can be made based on this sample webpage.

 

30-day temperature data

Since I was going away for a month, I decided, before I left, to set my SDI-12 data logger to run a 30-day data collection routine, just to see how robust the software is. Here is the result:

30-day temperature

The temperature sensor was in my office so its variation was small. The steep slope towards the end was because I opened the window after I got back and kept the window open for a whole day. You may notice the lack of variation of temperature between 5/10 and 5/13 but that WAS what happened. History data from wunderground.com showed less-than-average variation of outdoor temperature in my area during the same period:

wundergroundSo my conclusion was that my python data logger code running on raspberry pi was robust enough for at least 30 days so was the SDI-12 USB adapter. If this were done outdoors, then the raspberry pi and the SID-12 USB adapter both need protection from the element. Also a solar panel and battery will be needed.

Parts and components:

Raspberry pi 2 B (or 3 B)

SDI-12 USB adapter (Free python data logger code)

Decagon 5TM soil sensor

 

Soil data logger telemetry

I have finally found time to build a simple website for my soil data logger with telemetry. The system works as the following:

  1. The data logger consists of a raspberry pi and my SDI-12 USB adapter with a Decagon 5TM soil sensor
  2. The data logger runs the open-source datalogger code I wrote in Python to first get parameters from the user (COM port, SDI-12 address, delay etc.), and then collect data, save to a local .CSV file, and then send the same data to sparkfun’s phant server.
  3. I constructed a web interface to plot the data using Google Charts and download .CSV version from sparkfun’s phant server.

Here is a screen shot:

soil logger webpage

I’ve uploaded the webpage to a server with a link below. The sensor is apparently NOT buried in soil so I can easily take the setup and set it up in different places to test its stability.

Link to the website: Link

Back up and clone raspberry pi

In this post, I will explain how to back up and restore or clone a raspberry pi. I am assuming that you have installed Raspbian or Ubuntu mate on your raspberry pi. Since a raspberry pi runs its OS entirely on an sd card, it is easy to clone your system so you can pass along to a friend or create your special distro for others to use. For example, a company that makes compact optical spectrometer, Ocean Optics, has created a spectrometer program that runs on raspberry pi and has its own web interface. You just have to download their raspberry pi image and put it on your card. No need to do multiple build from source and apt-get etc. to get what you need.

If you are already a Linux guru, this post is NOT for you. I am assuming that you don’t have a Linux machine other than raspberry pi or you are still learning Linux.

The “easy” way:

Raspberry pi foundation has raw images of their Raspbian and NOOB on their website. On a windows system, all you need is win32 disk imager. It reads in the image and writes it on an SD card. You can save the image it reads and keep it as a backup or use it to clone a system.

The catch:

Since win32 disk imager only reads and writes in raw format, it actually doesn’t know what it is reading/writing. Say if you want to clone your raspberry pi 2B running Raspbian Jessie on an 8GB Kingston microSD card onto an 8GB SanDisk microSD card, you simply can’t. Although both cards claim to be 8GB, the SanDisk card is about 20MB smaller than the Kingston. Not understanding what it reads and writes, win32 disk imager is unable to shrink even one byte of what it reads (the whole 8GB from Kingston) to try to fit it onto a smaller card. This has pushed people into buying larger cards, such as 16GB. Then when they try to clone a system again, they run into the same issue unless they have bought a few identical SD cards as clone targets. Even that has issues (read the end of the post for details). So they have to step up to 32GB!!!

The solution:

We need a disk/partition reader and writer that understands what it is processing and is able to resize the partitions so they fit. If you are not hosting a lot of large files, you should be OK with an 8GB card or more than OK with a 16GB card. I’ve looked around for quite some time and found my solution: Paragon backup and Recovery 14 Free edition (Home edition for $39,99 has more features not useful for us). It is only for windows so those mac users will either need to become Linux gurus or shell out a small amount of money to get a win 10 netbook.

What the program does is that it is able to back up a whole disk or SD card and restore it on disk or SD card of different size. The reason is it understands most common file systems such as FAT, NTFS, HFS, and EXT. Besides, it also automatically compresses the backup so it doesn’t take more space than it should on your PC.

How to do it:

First you make a backup of your raspberry pi card using back up to VD. It is a virtual hard drive. This backs up both BOOT partition (FAT partition) and your Linux partition (ext4).

backup_to _vd

Next, insert your new SD card (you need to quit the program and restart it). Select restore from VD. It will ask you whether to resize partition and you can choose yes. This way you can even squeeze your clone from a larger card (32GB) to a smaller card (8GB) if your Linux partition has a lot of space.

Another neat trick:

What else you can do is to resize BOOT partition. Although BOOT partition has not much use for most of us, it IS the only partition that windows will recognize. If you are using your pi as a data logger or need to copy files from and to it (without the hassle of SSH and network setting), then the best way to get data is if it is written to the BOOT partition. You turn off raspberry pi and put its SD card on your windows machine. Immediately you have access to BOOT partition. You can copy your data out, you can change some config files of your data logger etc. It is useful to have a large BOOT partition. To do this, you will have to restore one partition at a time. Restore BOOT first. You will be prompted to select the size of the restored partition. Make your selection, say 500MB. Then when the restore is over, use the rest of the space on card to restore your Linux partition (make sure you leave enough space for Linux partition).

Why buying identical cards may not solve clone/restore issue?

Well, say you purchased 10 Kingston 8GB microSD cards, they are all the same size, correct? Wrong! I learned it the hard way. I had a friend that sent me an image he saved from such a card. I have identical cards. But, when I was telling win32 disk imager to write the image on my card, it complained: the target card doesn’t have enough space! Guess what, the target card is one sector less than the image file has?! But how can this happen? It’s simple now that I got my answer: some sectors on the SD card may have become bad either at factory or post purchase. The SD card controller (inside the card there is a controller) has disabled those sectors, making not all “identical” cards identical.

 

Dropbox Python API

Assume that you are building a data logger and need to send your data from your logger to you, one option that will not cost you money or much programming time is to send your data file to your Dropbox. Dropbox is a cloud storage service. It not only provides you free cloud storage, but also provides application programming interface so you can upload files via programs. This requires some minimal setup. Once set up, you can proceed to add the file upload feature to your code. I am using Python to do the job. It is quick and easy. Plus, you can port your code to any operating system, such as PC running windows, GNU/Linux, Mac OSX, or Raspberry Pi. I am assuming that you have Python 3.X.

There are two versions of Dropbox Python API, V1 and V2. V2 came out around the end of 2015 and only has minimal tutorial on Dropbox.com. Nevertheless, I will use V2. Installing the API is a snap:

On Windows:

Start a command prompt and enter:

pip install dropbox

On Raspberry Pi (Raspbian Jessie) or Debian PC

sudo pip3 install dropbox

If you still have Raspbian Wheezy, pip3 may tell you that there is a newer version of pip. Don’t attempt to upgrade your pip or pip3. It will break.

Here is a short Python script to upload and download files:

import dropbox
file_name='test_image.jpg'
dropbox_path='/'
dbx=dropbox.Dropbox('Your access token')
with open(file_name, 'rb') as f:
    dbx.files_upload(f.read(),dropbox_path+file_name,mute=True) # The change from f to f.read() was made to comply with the late-2016 change in the API.

dbx.files_download_to_file('Copy of '+file_name,dropbox_path+file_name)

This was easy. There are only three functions that I used, first an authentication, then upload, followed by download. Your access token is generated by Dropbox. This post has the details.
The function dropbox.Dropbox() returns an object. You can use this object to upload or download files and more.
The files_upload() function does the upload. You need to first open the file on your local computer with open(). I am assuming the file is stored in the same folder as your Python script. This returns a file handler f. Then pass f to the files_upload(). This is the first argument. The next argument is the path you want the file to be uploaded to dropbox. It has to start with ‘/’, then the file name (including additional path). You don’t have to preserve file name. The third argument is useful. When you sent mute to True, you won’t get notification for the file upload. If you are making a camera trap, you don’t want your PC flooded with Dropbox notifications just because a bunny decides to visit your backyard.
The files_download_to_file requires first a local computer name, then the Dropbox file name. Again you don’t have to preserve file name. It’s useful to first check if the file exists on your local computer and decide what to do (overwrite, or add prefix/postfix to new file).
This is it! How you use it is up to you.

If you are interested, you can explore the rest of the API, such as creating folders, moving files, listing folders, etc. by reading the documentation (alert: document is very dry)

https://www.dropbox.com/developers/documentation/python

In my next post, I will make a simple Raspberry Pi camera logger to activate only during several intervals of time of the day.

Connect to Dropbox

If you are making a data logger, or camera trap, this is definitely good news. With the right approach, you can sit in your home and data/pictures just flow to your desktop. All you need is internet connection and some Dropbox programming. With a raspberry pi or a pc, you can have your data logger or camera logger automatically upload data to your Dropbox. On a windows PC, this is nothing more than saving your data file to a folder inside your Dropbox folder. Still, constantly writing to a Drpobox file is not the best way to make use of it. You won’t be able to have a meaningful history of the file since the your data logger updates the file too many times. Also, raspberry pi doesn’t have dropbox client. So it makes sense to programmatically upload your file to Dropbox using its API or application programming interface.

I am starting a series or posts to detail how to upload files to Dropbox from within your code. This post only explains how to set things up.

Using Python, it is extremely easy to upload files to dropbox. There is only one catch, you have to set up as an app developer. Here is their logic thus how to get set up:

You have some good ideas for an app. You want to store your app’s data files on dropbox so your user can access them on any device they are using. Good idea, but how to set it up?

First you need a Dropbox account yourself. Once you’re set, log on with your web browser. You’ll see three circles on the bottom left of your window:

dropbox dev-1

Clicking the dots brings up a menu. Select Developers:

dropbox dev-2

 

Click “Create app” to create a new app. I’ve already created a couple of similar apps. It looks like I’ve been trying this camera logger idea for one too many times 😀

dropbox dev-3

Make sure that you select the right options. Select “App folder” access so all your actions are contained in a folder inside the App folder. It used to be called sandbox access. You may not want this app to access your entire Dropbox folder and accidentally delete files it is not intended to touch.

dropbox dev-4

Once done, you will be able to edit your project. Scroll down to find “Generated access token”. This will give you a very long alpha-numeric sequence that grants access to your Dropbox app folder, or the entire Dropbox if you chose “Full Dropbox”. Safeguard this string. If you need to ask questions about your code, remove/black out this string. Anyone with this string can do anything they want with your Dropbox app folder, no password needed. alternatively, you may try the App key and App secret path. That is a longer path than we need to take, just to show how things work.

dropbox dev-5

Now you are all set to develop this app, which is probably a program that uploads data/pictures to your dropbox. Read my next post about how to access Dropbox using Python. Python is one of the least preferred language I use but there are benefits of using it. It’s quick to see results. I am a big fan of Arduino. Unfortunately, it is almost impossible to upload to Dropbox using just an Arduino (except Arduino Yun).

SDI-12 USB adapter on Raspberry pi

I have successfully developed Python code to run the SDI-12 USB adapter data logger on Raspberry PI. It was quite a learning experience for me but I was able to boil down the steps into a tutorial on how to install the latest Python on Debian/Raspbian here.

Here are some screen shots of the data logger:

2016-02-03-171037_1680x1050_scrot

2016-02-05-100346_1680x1050_scrotI was using a Decagon 5TM soil temperature and humidity sensor (hanging free in air and sometimes touching my table).

I have translated the Tera Term scripts into Python so it would run on all operating systems, including Win 10, Linux (Debian), and Raspberry PI (Raspbian). I have not got my Mac fixed but don’t expect any issues.

Now you have two Python scripts:

The configuration script detects the SDI-12 sensor’s address, prints the name of the sensor, and you can change it to a different address.

The data logger script lists all available serial ports to let you choose the correct one, then asks for sensor address, total data points, delays between data points, and also what time (GMT or local) to use. You can expand the script to take data from multiple probes and upload data to a server. I’ll add more info on the server upload to sparkfun’s phant data server.

While developing the code, I also discovered that the serial port console was very useful. You can set up auto login and run the data logger script upon auto login. This way the data logger starts logging data once the RPI boots up and auto logs on. No need to log in or use VNC (not good if your logger is using 4G hotspot for internet). Just a command prompt is all that is needed. Next step is to add a config file so the data logger will no longer ask for user inputs when this file exists and takes parameters from the file. This config file will reside in the FAT partition so that the user can easily update the parameters without having to boot into raspberry pi. Just remove the sd card and change parameters on a PC.

Install python on raspberry pi or debian

I have just started learning the python programming language. It is very different than C/C++ that I’m used to. So to Arduino and Java folks it will take some getting used to. But the reason that I am learning is the promise of cross-platform portability and powerful libraries/modules. If you could send an HTTP request in a few lines, or imagine that you write and test a program on PC and run it on a raspberry pi with no/minimal modification, would you stop complaining about weird syntax or loose data types and just learn the darn language? Well, I did.

So in order to use Python, you need to install it. Since I am interested in using raspberry pi with an arduino, I need the best serial port support that I can find. My conclusion is to use Python 3.5.1 and pyserial 3.0.1 (the latest of both as of the blog post).

On a windows PC, this would be the easiest. Download the installers and install the program.

https://www.python.org/downloads/release/python-351/

https://pypi.python.org/pypi/pyserial

On a linux PC or raspberry pi, python is included but is a lower version. The tutorial helps you install python on linux machines and raspberry pi’s.

This tutorial is based on this tutorial and pieces of other tutorials and information online:

http://www.extellisys.com/articles/python-on-debian-wheezy

To install the latest version of python, you need to build it from source code.

  1. Download the source code and unzip it in a folder.

https://www.python.org/downloads/source/

  1. Update apt-get first

$ sudo apt-get update

  1. Install required tools to build Python from source. Debian/Raspbian Jessie will install libdb5.3-dev while Wheezy will install libdb5.1-dev. You need to install tk-dev to make Python’s IDE IDLE3.5 work

$ sudo apt-get install build-essential

$ sudo apt-get install tk-dev

$ sudo apt-get install libncurses5-dev libncursesw5-dev libreadline6-dev

$ sudo apt-get install libdb5.1-dev libgdbm-dev libsqlite3-dev libssl-dev

$ sudo apt-get install libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev

  1. Prepare for Pip (I didn’t do it but the tutorial I followed mentioned this, which seems a bit old)

$ mkdir -p ~/.pip/cache

$ echo ‘[global]’ > ~/.pip/pip.conf

$ echo ‘download_cache = ~/.pip/cache’ >> ~/.pip/pip.conf

  1. Make python in the unzipped source code folder (Python-3.5.1), install it in a folder that will not overwrite the current python version, such as in /usr/local/opt, delete upzipped source folder after done

$ cd Python-3.5.1

$ ./configure –prefix=/usr/local/opt/python-3.5.1

$ make

$ sudo make install

$ sudo mkdir /usr/local/opt/python-3.5.1

  1. Download pyserial source code and unzip in a folder.
  2. In the unzipped folder, run python 3.5 to install the module. If you used an alias to call python, then the install is in the default python version (3.4 on RPI) so don’t use an alias.

$ sudo /usr/local/opt/python-3.5.1/bin./python3.5 setup.py install

  1. Make aliases to run python 3.5 and idle more easily. You have to log out and back in after making them.

$ echo ‘alias python35=”/usr/local/opt/python3.5.1/bin/python3.5″‘ >> .bashrc

$ echo ‘alias idle35=”/usr/local/opt/python3.5.1/bin/idle3.5″‘ >> .bashrc

Now you are done! I forgot to mention that if you are installing Python on raspberry pi 1 or zero, the build process will take quite some time to complete. Log out and back in. In a terminal, type idle35 and you will be able to run your new Python.

%d bloggers like this: