Evolution of Stateless Computing

Microservices have become dominant over the last few years, so much so that it is hard to imagine encountering a modern application built with a SOAP API. The wide spread usage of stateless microservices has allowed for modern applications to be easily and quickly deployed horizontally and directly on the edge.  The lightweight nature of REST APIs due to their statelessness, have allowed for applications with less overhead, quicker integration times, and a much more enjoyable programming experience.  Microservice adoption at both the application and middleware layer have driven much of the advances in edge computing.

Databases living in a Past State

Databases on the other hand are lagging behind.  Many databases today still rely on statefulness which prevents them from being utilized directly as part of a microservice architecture.  Often it makes more sense for developers to buffer them with a middleware layer or in some cases even a caching database to protect both the database and the application.   

This has major disadvantages from a capital and performance perspective.  Companies have invested heavily in IoT devices; yet need to double that spending on cloud infrastructure. Furthermore, we have heard from developers where latency between the edge and the cloud is causing usability issues as well as crashes in some cases. 

Most SQL databases today, like VoltDB and SAP HANA, are heavily focused on in-memory indexes, which is great for a read only scenario, but suffers massively under a read/write or HTAP scenario. The statefulness of these in-memory databases create problems like out of memory exceptions which lead to crashes, and make them an unreliable platform to directly call as a microservice.  Furthermore, because of their statefulness, when recovering from crashes they take a long time to restore state, and as a result can lead to a long outage.  

NoSQL databases do not suffer from these same problems generally. However, they are difficult to use as HTAP databases as they are slow when attempting to perform advanced analytics and querying due to their structure.  Trying to use them for advanced analytics requires a lot of CPU and memory when performing multi-table queries or complex filters.  This makes it risky to use them as a microservice back end or IoT Database as this could lead to crashes.  The memory and CPU usage is due to their need to transform the data from a NoSQL model to a SQL model; hence the term multi-model. 

To solve these problems in edge computing and industrial IoT most developers today are relying on some form of caching mechanism on the edge, and then migrating their data from their IoT devices back to the cloud to live in their data warehouse.  This is expensive, complex, and slow. 

Middleware and ORM Slowing You Down

Another reason most folks do not use databases directly on the edge as a microservice platform is because they need ORMs and middleware to communicate to them.  While many databases today might support REST APIs it is not necessarily their recommended protocol.  As a result developers need some way to transform result sets returned by databases into usable objects. Often they will rely on ORM tools like Hibernate (http://hibernate.org/) or Mongoose (http://mongoosejs.com/).  These tools are terrible on performance and require a lot of overhead.  It would be ill advised to attempt to run an ORM directly on the edge.  In addition to their performance impact they require a lot of configuration and maintenance, and in an Industrial Internet of Things use case where you could have millions of connected devices it would be a nightmare to maintain an ORM on each of them. 

Microservice, Microdevices, & the IoT Database

HarperDB on the other hand is a fully indexed database with extremely low memory, cpu, or storage overhead.  At REST, HarperDB uses no CPU.  Because HarperDB uses very little resources, it is both unlikely to crash and will recover in seconds when it does. 

It has a 65MB footprint, at the time of this writing, allowing it to be installed directly on IoT devices.  No ORM is required as we natively return JSON object arrays for all operations and can consume JSON arrays or SQL. This allows you to perform both full noSQL CRUD and schema-less SQL directly from your app. 

HarperDB Enterprise also has full clustering capability at a schema and table level allowing for and orchestrating the data sharing across IoT devices.  Searches can also be performed from any node in the cluster and HarperDB will locate the data on the correct devices, amalgamate the results and return them to the executing node.  This makes HarperDB an ideal choice as an IoT database for IoT projects.