3

We are in the planning phase for a telemetry iot device with only occasional access to the internet.

I found a lot of information online on how to store iot data in the cloud, what databases to use, how to calculate space requirements etc. what I'm missing is:

How do I store the data locally on the client before sending it to the cloud?

  • Newest data is the most interessting for us, but for a subset of metrics we want to keep all data points on device until they get transfered
  • Minimizing device storage is not a top priority
  • Battery life is not crucial our device will be connected to power as long as it is collecting data
seedie
  • 41
  • 3

2 Answers2

1

Perhaps you could use the Round Robin Database Tool? It certainly operate along the lines that you want.

RRDtool (round-robin database tool) aims to handle time series data such as network bandwidth, temperatures or CPU load. The data is stored in a circular buffer based database, thus the system storage footprint remains constant over time.

It also includes tools to extract round-robin data in a graphical format, for which it was originally intended. Bindings exist for several programming languages, e.g. Perl, Python, Ruby, Tcl, PHP and Lua. There is an independent full Java implementation called rrd4j

If you can’t find a port or build it yourself, then the same principles are demonstrated with MySql at Round Robin Database with MySQL.

It’s what we do on every project I have worked on as an embedded develop for the last few decades (although we roll our own):

  • Some data are important/vital, and we do whatever we can to retain them all (not always possible, if there is along communications break)
  • Some we need the last X entries for (and can dimension that to a fixed size at design time)
  • Some we need the last X minutes’ worth of data (“round robin”, also size fixed at design time)
  • Some are “nice to have”, but we can ditch them if the space is needed (not so common, though)

Plus, of course, there is the question of which to store in NVM. This is all really a question for your project’s software architect - unless that’s you, in which case it’s a question for us :-)

There are other things that can help, such as compressing the data, or Run Length Encoding (e.g don't save the values for 1,000 consecutive readings with the same value; just save the value once, with a repeat counter), etc

Mawg
  • 3,147
  • 1
  • 13
  • 35
1

Since as per your spec this would be a linux based system, you could store the data locally on Sqlite database.

Further check out SQLITE-SYNC. With this framework supposedly your application can work completely offline, then perform an automated Bidirectional Synchronization when an internet connection becomes available.

So your app does not need to maintain routines to sync forward and/ back.

Ref: https://ampliapps.com/sqlite-sync/

SudhirR
  • 111
  • 2