Elasticsearch and IoT

A case for Elastic stack as an IoT analytics platform

When I started deep diving into Elastic Stack I found myself coming back to a single thought; “Damn, I really wish I knew about this on my last IoT project.”

Elastic Stack has quickly become a leading platform for application and infrastructure monitoring. 

For the last 10 years, the open-source community has collectively built a ‘best of breed’ solution for asset management. Features have been developed to allow quick, robust data ingestion, real-time alerts and self-service data visualisation. 

Given the strength of this platform for real-time asset management, my question is “why aren’t more people using Elastic for IoT analytics?”

While Elastic’s primary focus generally hasn’t been on IoT, the technical requirements in handling streams of logs from farms of application servers, and streams of sensor data from fleets of IoT devices are uncannily similar. 

In developing a best of breed tool for I.T infrastructure management Elastic have stumbled on a solid IoT analytics platform.

IoT analytics platform requirements

While specific requirements of a platform varies between organisations, developing a platform which serves real-time operational use cases must address these seven key criteria.

• Raw Data Processing
• Real-Time Aggregation
• Auto-Scaling Datastore
• Data Lifecycle Management
• Real-Time Alerting
• Self-Service Visualisation
• Application Monitoring

The Elastic Stack comprises of a suite of open source products that enable users to take data from anywhere and search, analyse and visualise it in real-time.

Between the three major components of the “ELK” stack; Elasticsearch, Logstash and Kibana, Elastic have addressed these seven core features beautifully in my (humble) opinion.

1. Raw Data Processing

Logstash supports over 50 inbound interfaces for data acquisition. 

IoT data can come in any structure (or non-structure) and be conformed to a standard set of fields.

Logstash supports data lookups, so additional asset information can be added to records for downstream use.

2. Real-Time Aggregation

Logstash aggregations can do both time based, and hierarchical based aggregations in real-time. Multiple readings from a set of devices can be aggregated into a single event. 

Aggregated records can be used to trigger real-time models by fanning out the pipeline output to multiple systems. 

3. Auto-Scaling Datastore

Elasticsearch is a distributed document store, spreading data over an array of nodes.

Managed services can spin up more nodes to accommodate growth in data volumes as fleets of IoT devices grow in size.

4. Data Lifecycle Management

Elasticsearch index lifecycle policies make it easy to move time-series data through management cycles. 

Indexes can be automatically created to split IoT time series by date. 

With minimal configuration, a set of data can be set up to automatically run through a lifecycle.

For example, an index may be optimised for data writes for 24 hours, reads for the next 7 days, moved into cold storage for the following 30 days, before being archived and deleted.

5. Real-Time Alerting

Elastic and Kibana both support native alerting features. 

IoT events that indicate critical asset status can send alerts via email, slack or any other http interface. 

Kibana’s ML features can automatically detect anomalies in time series data, soliciting analysts to identify the source of asset faults.

6. Self-Service Visualisation

Kibana supports many serve data visualization tools for time series, geospatial or anomaly analysis.

7. Application Monitoring

Kibana provides monitoring over the entire Elastic Stack out of the box. 

Each component, from back end data ingestion services to head-end data visualisation services can be monitored. Alerts can be configured to ensure platform issues are addressed before the system goes down.

So Why Elastic?

What distinguishes Elastic Stack from other options is that these seven features come with a single stack. It means I get a base IoT analytics platform within the first sprint of a project. 

If you’re looking to kick-start an IoT project within your organisation, consider what platforms are going to best put your project in line for short-term, and long-term success.

Aside from meeting user requirements, what would it take to establish platform stability and hand over ownership to your internal support teams? 

Please get in touch to see how we can help implement the right IoT stack for you.