• Autonomous data acquisition and recording (currently up to 8 channels @ 500Hz)
  • Based on the (Raspberry PI microcomputer) (low power 3W, running high-availability Linux)
  • Online data in real-time
  • ssh command line access even from browser
  • www lightweight web server with diagnostic tools running on the Raspberry
  • Advanced remote administration tools like remote-syslog over a websocket.
  • ftp access for data transfer, remote synchronization etc.
  • Standard components (Raspberry PI + Analog to Digital converter e.g. ADS1115 + some enclosure)
  • Creative Commons License open source software written in C (community-driven, and still under maintenance and development) and efficient system configuration.
  • Powerful visualization: FlotCharts
  • Highly efficient binary output (NetCDF > compressed with HDF5, JSON, postgreSQL etc.)

It is an adaptable system, even from one configuration file you can define the number of channels used, the voltage range (if not dynamic tracking chosen) or the output format. There are countless opportunities to adapt or extend any part of the software/hardware. You are invited to use, give feedback or even contribute!

Examples

data logger with differential inputs front face of a data logger with display 4 differential channels with isolated BNC connectors power over ethernet (and the alternative power supply input 5V) sensor supply, converting 5V to regulated dual voltage with LM2587 a raspberry equipped with a GPS module to record accurate time power supply 5V 3A data loggers in robust metalic casing

The Operating System (OS) (real time Linux)

The Raspberry PI hosts an RT-PREEMPT kernel Linux which is a real-time OS. All the graphical and disk consuming packages are dropped, and those responsible for e.g network-security (fail2ban) and stable operation (e.g. watchdog) are installed and configured. Only the absolute minimum of the kernel modules are loaded. There is no sound support. This is necessary to have low-latency sampling intervals (in order of microseconds and less). The current logger application runs in the background (starting automatically, writing to log files, checking elapsed time since the last file write, checking disk usage, error handling...), but currently with a bit of overhead (threaded interrupts softirq. However the logger running with 4 channels each 500Hz sampling, utilizes 30% CPU on a Raspberry Pi B (800MHz) only. (The new version based on this article

preemptive OS (www.embeddedlinux.org)

preemptive OS (www.embeddedlinux.org)

Analog-to-Digital converter (ADC)

The universal ADC ADS1115 is a very versatile and cheap IC. The key features

  • One single IC has 2 differential channels (or 4 single ended) where 1 differential (or 2 single ended) are converted at a time. (They are switchable with a multiplexer (MUX) controlled by software)
  • sampling rates from 1Hz to 860Hz possible
  • although the resolution is 16 bit only there is a programmable gain amplifier (PGA) which can be set by software in 5 steps. By tracking the signal's amplitude and setting the PGA dynamically it converges to a 20 bit ADC. (auto-pga feature)
  • at maximum 4 IC-s can be connected simultaneously to the I2C bus of the Raspberry PI (4 differential channels sampled at a time, altogether 8 in two steps).
  • excellent linearity, simple commands, low consumption, high input impedance, internal voltage reference, no thermal drift...
  • fast data transfer using the i2c-dev kernel module, I2C-bus clock speed (SCK) of 3MHz verified!
ADS1115 analog to digital converter

ADS115 analog to digital converter (www.adafruit.com)

File formats

binary files (fast sampling rates)

After trying out many ways for large data storage (4 channels á 500 times a second sampled á 4 bytes (float)), I ended up at the high density (HDF5) compressed binary file format the Network Common Data Form or net-cdf (nc) which is fully implemented. It is as efficient that the 2 channel recording of 24 hours (sampling rate 500Hz each) which sums up to 860MB of ASCII data (as txt or csv files) will be compressed to only 55MB. There is excellent support on the command line (e.g. ncdump) to convert between ascii and binary but also python/matlab/octave deals with it very easily.

databases (lower sampling rates)

There are several C interfaces to put recordings into databases. After an excessive research (I am not a specialist on databases) postgreSQL turned out to be lightweight enough to work smoothly on the raspberry while carrying out performant INSERT INTO-tasks. However, IMHO for sampling rates above 10Hz (assuming 24/7 logging) it slows down just after 1 month with no special precautions (10*60*1440*31 > 26 mil. rows). For temperature logging or whatever slow dynamics measurement it's tool number one for me.

cloud storage of sensor data

Examples: nimbits, thingspeak, orchestra, sensorcloud, relayr, xively or tempoiq. I have not excessively used these, but there are surely no limitations on the side of the logger software, if this is one's preference to store data. The source code of the logger is kept as modular as possible.

There is a chance to implement JSON and some other timeseries data formats soon. However, contributors are welcome!

Further reading

References

Hungarian National Report on IUGG 2011-2014

universal raspberry pi based data logger developed for the NCK geophysical observatory - IAGA division 5. observatory, instruments, surveys and analyses



Comments

comments powered by Disqus

Further articles



Directory tree inside home

The charts below give an overview what is where inside /home/pi. My intention was to put everything related to the sampling program into "opendatalogger"

[/home/pi/]
 ├── analysis/
 ├── conf/
 ├── opendatalogger/
 ├── wsserver/
 └── www/

The source code of ...

more ...

Contributors wanted

Feedback

Please feel free to join our discussion-site on github!. This is probably the best place to ask! (However I try to continue answering e-mails addressed to me personally, but please consider the above discussion site before mailing.)

Start a discussion, ask or just leave a feedback! You are welcome ...

more ...

Disk image files

In the article How to copy an image to an sd-card you can read how to copy the disk image file (e.g. what you have downloaded, e.g. raspbian-RT-opendatalogger-201511.img) to an SD-card in a card reader attached to your computer.

Now, I would like to show, how the ...

more ...

BCM2708 hardware watchdog

The BCM2708 has a very useful hardware watchdog. It will be used to reset the Raspberry if something hangs.

what is a watchdog?

The Linux kernel can reset the system if serious problems are detected. This can be implemented via special watchdog hardware... There needs to be a daemon that ...

more ...

File locks (run single instance)

It is a common issue, that the developer wants to limit the application (process, task or daemon all equivalently used) to run only one single instance at once. Some devices (file descriptors) can be accessed by one process at a time. Usually, a file lock is used to indicate that ...

more ...

The Linux embedded data-logger

  • Autonomous data acquisition and recording (currently up to 8 channels @ 500Hz)
  • Based on the (Raspberry PI microcomputer) (low power 3W, running high-availability Linux)
  • Online data in real-time
  • ssh command line access even from browser
  • www lightweight web server with diagnostic tools running on the Raspberry
  • Advanced remote administration tools like ...
more ...

Firmware

The Analog-To-Digital converter (ADC) transfers data over the I2C bus. The kernel module i2c_dev is used to interact with the physical interface. The currently loaded modules can be checked with the lsmod command. As it can be seen in i2c.c the C commands open(), close(), read(), write() and ioctl ...

more ...

Hardware

The Analog-to-Digital converters ADS1115 are connected to the I2C bus of the Raspberry PI. The ADC is fed from 3.3V, the clock is driven by the I2C-Master (Raspberry) with 2MHz. It is as simple as 4 wires only

  • +3.3V
  • GND (ground, 0V)
  • SCL (serial clock, driven by the ...
more ...

Patch the data logger to the OS of your favour

The instructions below provide a guide to port the datalogger to your existing OS.

It is possible to configure everything on your running OS to use it, however it took me several month of experimenting to find all the settings. There are several system-wide config files and permissions to set ...

more ...