How can you possibly have a dynamic schema? Isn’t the purpose of a schema to provide structure to your data? Well, yes, schemas exist to contain objects and define structure. But what happens if I know my data will continue to evolve, as it tends to do? Should you have to constantly rely on developers and DBAs to develop the schema updates, test them, then deploy? To put it simply, no, that’s a lot of overhead. The HarperDB Dynamic Schema adapts to your ever-changing data, all while providing the ability to access that data however you need.

The good old-fashioned relational schema (SQL)

The old industry standard is the relational database, with a full schema structure, where DBA’s and/or developers have to manually adjust and configure any schema changes after the need is identified. This is a slow process that does not allow an organization to rapidly adapt to change as it happens. It handcuffs development and requires bulky and complicated change management processes even just to add a simple field to a table. That said, relational databases are incredibly powerful for storing structured unchanging data. Data is easy to access with SQL, which might as well be a universal language to developers. Where relational databases fail is in data volume and in their ability to adapt to ever-changing data. How did the world respond?

The Newfangled Schemaless Design (NoSQL)

In response to the obvious need to adapt to changing data the new schemaless design was created. What does schemaless mean? Absolutely no structure. It’s the wild west of data. Common data points can be scattered in fields with similar, but different, names throughout the database simply because they were ingested from a different source. Need to find a phone number for a person? If our schemaless database didn’t have incredibly meticulous developers from day one, we might need to look across tens of fields: phone, phoneNumber, phone_number, cell, cellNumber, cell_number, mobile, mobileNumber, mobile_number, to name a few. Data is typically accessed with NoSQL, ironically named because it’s not SQL… Developers switched to schemaless databases because they didn’t have a choice, a relational database couldn’t handle the volume of data. But what about a hybrid of the two?

A high-level comparison between relational databases and NoSQL databases can be found here.

The Happy Medium

HarperDB was founded because the existing solutions are just too complicated. We are merging worlds, providing full SQL and NoSQL interfaces. We’re big data with a bit of structure, we’re relational data without the overbearing restraints. This is the essence of the HarperDB’s Dynamic Schema. The initial data load defines the table structure, but the database has the flexibility to adapt to constantly changing data ingests, adding fields as necessary. With the HarperDB middle ground structure, developers are able to retain the best of both worlds. They can ingest unstructured data and retain SQL functionality, like describing their schema and executing multi-table joins.

The Agile advantage

I come from an Agile world where applications are developed sprint by sprint. This was great for development, we iterated every two weeks, so if the business needed changes, they would be addressed relatively quickly. Did the requirements change and we need to add a couple of fields? Well, now we have to go through the whole database change management process I described above. I can’t tell you how many times I’ve seen teams run into communication issues with DBAs in both development and deployment. Imagine if the development team could just add the field in code and the database would adapt accordingly. Now sprint deliverables could be completed and deployed even faster.

There are other dynamic schemas out there, what makes HarperDB so special?

HarperDB is unique because it unifies the key database features that are currently exploded across the market and adds a cutting-edge data storage solution, ironically, the exploded data model. Our dynamic schema lives in an HTAP database with single-model architecture that’s fully indexed and can be accessed with both SQL and NoSQL. The first time they told me that, I didn’t believe it either. I downloaded the community edition and tested it out, I suggest you do the same. Go try it out for yourself!