Can’t upgrade pyserial in latest raspbian distribution?

This is just for your information if you are a Raspberry Pi user and playing with Python code from my blog. If you are trying to use the latest distro of raspbian with pyserial for some serial port project, you may have come across this issue that regardless how you upgrade pyserial using pip3, your python3 will always call up the old pyserial 2.6 that came with the distribution. I am a bit disappointed that the foundation has included such an old version of pyserial, couldn’t they just try a pyserial 3.0 instead? My solution was to remove the python3-serial module using apt-get and then install pyserial 3.3 using pip3.

sudo apt-get remove python3-serial
sudo pip3 install pyserial

Hope this helps.

Saving data to sparkfun server

If you’ve been following my posts, I discussed how to use my SDI-12 USB adapter and raspberry pi (or PC) to log data from SDI-12 sensors. Now that you have your data, you can also send them to sparkfun’s data server for storage and later retrieval. In the following video, I explained what you can do with this feature included in my data logger python script. I also constructed a sample webpage to display data saved on the server. Lots of customization can be made based on this sample webpage.

 

30-day temperature data

Since I was going away for a month, I decided, before I left, to set my SDI-12 data logger to run a 30-day data collection routine, just to see how robust the software is. Here is the result:

30-day temperature

The temperature sensor was in my office so its variation was small. The steep slope towards the end was because I opened the window after I got back and kept the window open for a whole day. You may notice the lack of variation of temperature between 5/10 and 5/13 but that WAS what happened. History data from wunderground.com showed less-than-average variation of outdoor temperature in my area during the same period:

wundergroundSo my conclusion was that my python data logger code running on raspberry pi was robust enough for at least 30 days so was the SDI-12 USB adapter. If this were done outdoors, then the raspberry pi and the SID-12 USB adapter both need protection from the element. Also a solar panel and battery will be needed.

Parts and components:

Raspberry pi 2 B (or 3 B)

SDI-12 USB adapter (Free python data logger code)

Decagon 5TM soil sensor

 

Soil data logger telemetry

I have finally found time to build a simple website for my soil data logger with telemetry. The system works as the following:

  1. The data logger consists of a raspberry pi and my SDI-12 USB adapter with a Decagon 5TM soil sensor
  2. The data logger runs the open-source datalogger code I wrote in Python to first get parameters from the user (COM port, SDI-12 address, delay etc.), and then collect data, save to a local .CSV file, and then send the same data to sparkfun’s phant server.
  3. I constructed a web interface to plot the data using Google Charts and download .CSV version from sparkfun’s phant server.

Here is a screen shot:

soil logger webpage

I’ve uploaded the webpage to a server with a link below. The sensor is apparently NOT buried in soil so I can easily take the setup and set it up in different places to test its stability.

Link to the website: Link

Back up and clone raspberry pi

In this post, I will explain how to back up and restore or clone a raspberry pi. I am assuming that you have installed Raspbian or Ubuntu mate on your raspberry pi. Since a raspberry pi runs its OS entirely on an sd card, it is easy to clone your system so you can pass along to a friend or create your special distro for others to use. For example, a company that makes compact optical spectrometer, Ocean Optics, has created a spectrometer program that runs on raspberry pi and has its own web interface. You just have to download their raspberry pi image and put it on your card. No need to do multiple build from source and apt-get etc. to get what you need.

If you are already a Linux guru, this post is NOT for you. I am assuming that you don’t have a Linux machine other than raspberry pi or you are still learning Linux.

The “easy” way:

Raspberry pi foundation has raw images of their Raspbian and NOOB on their website. On a windows system, all you need is win32 disk imager. It reads in the image and writes it on an SD card. You can save the image it reads and keep it as a backup or use it to clone a system.

The catch:

Since win32 disk imager only reads and writes in raw format, it actually doesn’t know what it is reading/writing. Say if you want to clone your raspberry pi 2B running Raspbian Jessie on an 8GB Kingston microSD card onto an 8GB SanDisk microSD card, you simply can’t. Although both cards claim to be 8GB, the SanDisk card is about 20MB smaller than the Kingston. Not understanding what it reads and writes, win32 disk imager is unable to shrink even one byte of what it reads (the whole 8GB from Kingston) to try to fit it onto a smaller card. This has pushed people into buying larger cards, such as 16GB. Then when they try to clone a system again, they run into the same issue unless they have bought a few identical SD cards as clone targets. Even that has issues (read the end of the post for details). So they have to step up to 32GB!!!

The solution:

We need a disk/partition reader and writer that understands what it is processing and is able to resize the partitions so they fit. If you are not hosting a lot of large files, you should be OK with an 8GB card or more than OK with a 16GB card. I’ve looked around for quite some time and found my solution: Paragon backup and Recovery 14 Free edition (Home edition for $39,99 has more features not useful for us). It is only for windows so those mac users will either need to become Linux gurus or shell out a small amount of money to get a win 10 netbook.

What the program does is that it is able to back up a whole disk or SD card and restore it on disk or SD card of different size. The reason is it understands most common file systems such as FAT, NTFS, HFS, and EXT. Besides, it also automatically compresses the backup so it doesn’t take more space than it should on your PC.

How to do it:

First you make a backup of your raspberry pi card using back up to VD. It is a virtual hard drive. This backs up both BOOT partition (FAT partition) and your Linux partition (ext4).

backup_to _vd

Next, insert your new SD card (you need to quit the program and restart it). Select restore from VD. It will ask you whether to resize partition and you can choose yes. This way you can even squeeze your clone from a larger card (32GB) to a smaller card (8GB) if your Linux partition has a lot of space.

Another neat trick:

What else you can do is to resize BOOT partition. Although BOOT partition has not much use for most of us, it IS the only partition that windows will recognize. If you are using your pi as a data logger or need to copy files from and to it (without the hassle of SSH and network setting), then the best way to get data is if it is written to the BOOT partition. You turn off raspberry pi and put its SD card on your windows machine. Immediately you have access to BOOT partition. You can copy your data out, you can change some config files of your data logger etc. It is useful to have a large BOOT partition. To do this, you will have to restore one partition at a time. Restore BOOT first. You will be prompted to select the size of the restored partition. Make your selection, say 500MB. Then when the restore is over, use the rest of the space on card to restore your Linux partition (make sure you leave enough space for Linux partition).

Why buying identical cards may not solve clone/restore issue?

Well, say you purchased 10 Kingston 8GB microSD cards, they are all the same size, correct? Wrong! I learned it the hard way. I had a friend that sent me an image he saved from such a card. I have identical cards. But, when I was telling win32 disk imager to write the image on my card, it complained: the target card doesn’t have enough space! Guess what, the target card is one sector less than the image file has?! But how can this happen? It’s simple now that I got my answer: some sectors on the SD card may have become bad either at factory or post purchase. The SD card controller (inside the card there is a controller) has disabled those sectors, making not all “identical” cards identical.

 

Dropbox Python API

Assume that you are building a data logger and need to send your data from your logger to you, one option that will not cost you money or much programming time is to send your data file to your Dropbox. This requires some minimal setup. Once set up, you can proceed to add the file upload feature to your code. I am using Python to do the job. It is quick and easy. Plus, you can port your code to any operating system, such as PC running windows, GNU/Linux, Mac OSX, or Raspberry Pi. I am assuming that you have Python 3.X.

There are two versions of Dropbox Python API, V1 and V2. V2 came out around the end of 2015 and only has minimal tutorial on Dropbox.com. Nevertheless, I will use V2. Installing the API is a snap:

On Windows:

Start a command prompt and enter:

pip install dropbox

On Raspberry Pi (Raspbian Jessie) or Debian PC

sudo pip3 install dropbox

If you still have Raspbian Wheezy, pip3 may tell you that there is a newer version of pip. Don’t attempt to upgrade your pip or pip3. It will break.

Here is a short Python script to upload and download files:

import dropbox
file_name='test_image.jpg'
dropbox_path='/'
dbx=dropbox.Dropbox('Your access token')
with open(file_name, 'rb') as f:
    dbx.files_upload(f,dropbox_path+file_name,mute=True)

dbx.files_download_to_file('Copy of '+file_name,dropbox_path+file_name)

This was easy. There are only three functions that I used, first an authentication, then upload, followed by download. Your access token is generated by Dropbox. This post has the details.
The function dropbox.Dropbox() returns an object. You can use this object to upload or download files and more.
The files_upload() function does the upload. You need to first open the file on your local computer with open(). I am assuming the file is stored in the same folder as your Python script. This returns a file handler f. Then pass f to the files_upload(). This is the first argument. The next argument is the path you want the file to be uploaded to dropbox. It has to start with ‘/’, then the file name (including additional path). You don’t have to preserve file name. The third argument is useful. When you sent mute to True, you won’t get notification for the file upload. If you are making a camera trap, you don’t want your PC flooded with Dropbox notifications just because a bunny decides to visit your backyard.
The files_download_to_file requires first a local computer name, then the Dropbox file name. Again you don’t have to preserve file name. It’s useful to first check if the file exists on your local computer and decide what to do (overwrite, or add prefix/postfix to new file).
This is it! How you use it is up to you.

If you are interested, you can explore the rest of the API, such as creating folders, moving files, listing folders, etc. by reading the documentation (alert: document is very dry)

https://www.dropbox.com/developers/documentation/python

In my next post, I will make a simple Raspberry Pi camera logger to activate only during several intervals of time of the day.

Connect to Dropbox

If you are making a data logger, or camera trap, this is definitely good news. With the right approach, you can sit in your home and data/pictures just flow to your desktop. All you need is internet connection and some Dropbox programming. With a raspberry pi or a pc, you can have your data logger or camera logger automatically upload data to your Dropbox. On a windows PC, this is nothing more than saving your data file to a folder inside your Dropbox folder. Still, constantly writing to a Drpobox file is not the best way to make use of it. You won’t be able to have a meaningful history of the file since the your data logger updates the file too many times. Also, raspberry pi doesn’t have dropbox client. So it makes sense to programmatically upload your file to Dropbox using its API or application programming interface.

I am starting a series or posts to detail how to upload files to Dropbox from within your code. This post only explains how to set things up.

Using Python, it is extremely easy to upload files to dropbox. There is only one catch, you have to set up as an app developer. Here is their logic thus how to get set up:

You have some good ideas for an app. You want to store your app’s data files on dropbox so your user can access them on any device they are using. Good idea, but how to set it up?

First you need a Dropbox account yourself. Once you’re set, log on with your web browser. You’ll see three circles on the bottom left of your window:

dropbox dev-1

Clicking the dots brings up a menu. Select Developers:

dropbox dev-2

 

Click “Create app” to create a new app. I’ve already created a couple of similar apps. It looks like I’ve been trying this camera logger idea for one too many times 😀

dropbox dev-3

Make sure that you select the right options. Select “App folder” access so all your actions are contained in a folder inside the App folder. It used to be called sandbox access. You may not want this app to access your entire Dropbox folder and accidentally delete files it is not intended to touch.

dropbox dev-4

Once done, you will be able to edit your project. Scroll down to find “Generated access token”. This will give you a very long alpha-numeric sequence that grants access to your Dropbox app folder, or the entire Dropbox if you chose “Full Dropbox”. Safeguard this string. If you need to ask questions about your code, remove/black out this string. Anyone with this string can do anything they want with your Dropbox app folder, no password needed. alternatively, you may try the App key and App secret path. That is a longer path than we need to take, just to show how things work.

dropbox dev-5

Now you are all set to develop this app, which is probably a program that uploads data/pictures to your dropbox. Read my next post about how to access Dropbox using Python. Python is one of the least preferred language I use but there are benefits of using it. It’s quick to see results. I am a big fan of Arduino. Unfortunately, it is almost impossible to upload to Dropbox using just an Arduino (except Arduino Yun).

SDI-12 USB adapter on Raspberry pi

I have successfully developed Python code to run the SDI-12 USB adapter data logger on Raspberry PI. It was quite a learning experience for me but I was able to boil down the steps into a tutorial on how to install the latest Python on Debian/Raspbian here.

Here are some screen shots of the data logger:

2016-02-03-171037_1680x1050_scrot

2016-02-05-100346_1680x1050_scrotI was using a Decagon 5TM soil temperature and humidity sensor (hanging free in air and sometimes touching my table).

I have translated the Tera Term scripts into Python so it would run on all operating systems, including Win 10, Linux (Debian), and Raspberry PI (Raspbian). I have not got my Mac fixed but don’t expect any issues.

Now you have two Python scripts:

The configuration script detects the SDI-12 sensor’s address, prints the name of the sensor, and you can change it to a different address.

The data logger script lists all available serial ports to let you choose the correct one, then asks for sensor address, total data points, delays between data points, and also what time (GMT or local) to use. You can expand the script to take data from multiple probes and upload data to a server. I’ll add more info on the server upload to sparkfun’s phant data server.

While developing the code, I also discovered that the serial port console was very useful. You can set up auto login and run the data logger script upon auto login. This way the data logger starts logging data once the RPI boots up and auto logs on. No need to log in or use VNC (not good if your logger is using 4G hotspot for internet). Just a command prompt is all that is needed. Next step is to add a config file so the data logger will no longer ask for user inputs when this file exists and takes parameters from the file. This config file will reside in the FAT partition so that the user can easily update the parameters without having to boot into raspberry pi. Just remove the sd card and change parameters on a PC.

Install python on raspberry pi or debian

I have just started learning the python programming language. It is very different than C/C++ that I’m used to. So to Arduino and Java folks it will take some getting used to. But the reason that I am learning is the promise of cross-platform portability and powerful libraries/modules. If you could send an HTTP request in a few lines, or imagine that you write and test a program on PC and run it on a raspberry pi with no/minimal modification, would you stop complaining about weird syntax or loose data types and just learn the darn language? Well, I did.

So in order to use Python, you need to install it. Since I am interested in using raspberry pi with an arduino, I need the best serial port support that I can find. My conclusion is to use Python 3.5.1 and pyserial 3.0.1 (the latest of both as of the blog post).

On a windows PC, this would be the easiest. Download the installers and install the program.

https://www.python.org/downloads/release/python-351/

https://pypi.python.org/pypi/pyserial

On a linux PC or raspberry pi, python is included but is a lower version. The tutorial helps you install python on linux machines and raspberry pi’s.

This tutorial is based on this tutorial and pieces of other tutorials and information online:

http://www.extellisys.com/articles/python-on-debian-wheezy

To install the latest version of python, you need to build it from source code.

  1. Download the source code and unzip it in a folder.

https://www.python.org/downloads/source/

  1. Update apt-get first

$ sudo apt-get update

  1. Install required tools to build Python from source. Debian/Raspbian Jessie will install libdb5.3-dev while Wheezy will install libdb5.1-dev. You need to install tk-dev to make Python’s IDE IDLE3.5 work

$ sudo apt-get install build-essential

$ sudo apt-get install tk-dev

$ sudo apt-get install libncurses5-dev libncursesw5-dev libreadline6-dev

$ sudo apt-get install libdb5.1-dev libgdbm-dev libsqlite3-dev libssl-dev

$ sudo apt-get install libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev

  1. Prepare for Pip (I didn’t do it but the tutorial I followed mentioned this, which seems a bit old)

$ mkdir -p ~/.pip/cache

$ echo ‘[global]’ > ~/.pip/pip.conf

$ echo ‘download_cache = ~/.pip/cache’ >> ~/.pip/pip.conf

  1. Make python in the unzipped source code folder (Python-3.5.1), install it in a folder that will not overwrite the current python version, such as in /usr/local/opt, delete upzipped source folder after done

$ cd Python-3.5.1

$ ./configure –prefix=/usr/local/opt/python-3.5.1

$ make

$ sudo make install

$ sudo mkdir /usr/local/opt/python-3.5.1

  1. Download pyserial source code and unzip in a folder.
  2. In the unzipped folder, run python 3.5 to install the module. If you used an alias to call python, then the install is in the default python version (3.4 on RPI) so don’t use an alias.

$ sudo /usr/local/opt/python-3.5.1/bin./python3.5 setup.py install

  1. Make aliases to run python 3.5 and idle more easily. You have to log out and back in after making them.

$ echo ‘alias python35=”/usr/local/opt/python3.5.1/bin/python3.5″‘ >> .bashrc

$ echo ‘alias idle35=”/usr/local/opt/python3.5.1/bin/idle3.5″‘ >> .bashrc

Now you are done! I forgot to mention that if you are installing Python on raspberry pi 1 or zero, the build process will take quite some time to complete. Log out and back in. In a terminal, type idle35 and you will be able to run your new Python.

%d bloggers like this: