Data Management on the edge

With industrial edge computing, data is processed directly at the source. This improves response times and saves bandwidth, as the data no longer has to be sent to the data center and back. The edge device can also work without a connection to the data center or the Internet - but this poses new challenges for data storage.

Unlike in the cloud or with IT systems, storage space on typical industrial edge devices is usually very limited. It is therefore important to plan what data is generated and how it can be managed most efficiently.

Data types

Depending on the application, different types of data may be generated, each of which requires different handling.

Measurement data

A common use case is the collection of measurement data. Database management systems that can store and retrieve this data efficiently are suitable here. In practice, time series databases (TSDBs) such as InfluxDB or VictoriaMetrics are often used. While InfluxDB is popular in the cloud, it lacks support for ARM32-based devices, which are widely used in the industry. VictoriaMetrics is a good alternative here as it highly compresses the data and better protects data integrity in the event of power failures - which can often occur with edge devices.  

Log data

Edge applications generate various log outputs to track the program flow. This data is stored for later error analysis. Typically, these cause a large number of write cycles, which wears out the flash memory. Some systems therefore write the log outputs exclusively to a RAM disk, which means that all log data is lost when the system is restarted at the latest. To save memory space, older log outputs are automatically replaced by new ones.

Application data

This data includes program settings, cache and smaller data that is generated and saved at runtime. File-based approaches are often chosen here. Databases such as SQLite or PostgreSQL are used for more complex data structures.

Secrets

This particularly sensitive data, such as passwords, tokens or keys, must be protected against unauthorized access. Ideally, a hardware security module (HSM) on the edge device takes over the cryptographic tasks so that private keys can be securely stored and used.

SD card for memory expansion

Edge devices often only have limited internal flash memory, which is mainly sufficient for the operating system and some data. However, many devices offer an SD card slot for memory expansion. SD cards with up to 512 GB are available in 2024. It is important to use “industrial” SD cards, as these are more temperature-resistant and can withstand more write cycles.
By using an SD card, the storage capacity can be increased, but there is a risk that the card will be removed and read out. One solution is to encrypt the content and save the key in the device's internal memory. This can be software-based, e.g. with the SQLite Encryption Extension, or hardware-based.

Conclusion

To summarize, despite its challenges, industrial edge computing offers an efficient solution for data processing if the data types and storage options are carefully considered.