Mostafa Ibrahim, software engineer in Fintech and ex ARM engineer, writes about the importance of implementing technologies that enable multi cloud communication. 

Introduction

As a newer addition to the database world, HarperDB is the up-and-coming data platform contender to some of the most well-known database providers out there. Created to make the building process of complex systems such as hybrid clouds not only easier, cheaper, and faster, but most important of all, attainable. In this article we will highlight some of the benefits of utilizing technologies that enable a multi-cloud strategy. We will then jump into a tutorial to demonstrate HarperDB's cross-cloud capabilities to illustrate that as long as there's network connectivity, HarperDB can always talk to HarperDB.

HarperDB Overview

HarperDB is a modern data platform that encapsulates both SQL and NoSQL features into its own unique database model. HarperDB also provides its own Management Studio, custom API built-in functions, clustering and replication, and much more. You can learn more about HarperDB in this article on Distributed Databases.

Cross-Cloud Communication

With the global cloud market expected to reach $623 billion dollars by the year 2023, cloud computing is becoming more and more of a norm for both new and old businesses to rely on. As the name implies, cross-cloud communication is the process of unifying multiple different cloud providers under the same umbrella. These cloud providers will collaborate and work together under the same application and share the same or similar functionalities, infrastructure, and security.

In cases where two or more clouds are utilized, then a data synchronization process often must be performed. The reason behind this is to have both clouds containing exactly the same data about all users. In cases where unmatching data is stored, catastrophic errors may occur. For example, if a user withdraws $1,000 and only one of the two cloud logs update, then an error where the user may be capable of withdrawing additional money may occur. In other cases, organizations may choose to store some data in one cloud and other data in another cloud- it really depends on their specific data fabric. 

So what are some advantages of cross-cloud communication and why would companies spend time perfecting such a (seemingly) complex data structure? By utilizing multiple different cloud providers, developers can choose and pick the specific features that they desire their system to have. A multi-cloud strategy eliminates vendor lock-in while allowing companies to benefit from different services provided by the different major cloud providers. Multiple cloud systems can increase the organization’s footprint so that they can utilize the best cloud provider for their specific region, ultimately reducing the risk of outages.

With all that said about how useful cross-cloud communication is, it must be said that it is not always the case that multi-cloud communication is better than a single or mono-cloud system. In some cases, using multiple cloud providers will cause extra friction between developers working on different sides, slow the system, cause more failures, and could cost more in the long run. However, if used correctly, cross-cloud communication will definitely make your life easier.

Tutorial

In the practical part of this article, we will create two HarperDB instances each on a different cloud provider. The first instance will be hosted using the HarperDB cloud (AWS), while the second instance will be hosted on Azure. After creating and hosting both instances, we will send a request from Azure to AWS. Other than showing the user how to create both instances, the purpose of this tutorial is to clearly show that there is no dip in performance when using two HarperDB instances hosted on two different clouds.

Creating and hosting a HarperDB instance on AWS

1- Open the HarperDB page and click on the start free button on the top right of the screen. Sign in or create your HarperDB account for free.

2- After completing the signing-in process, a new page showing the available instances will open. 

Click on the “Create New HarperDB Cloud Instance” white box on the top left side of the screen.

3- Choose the hosting provider of your instance. For our first instance, we will choose the left option “ Create AWS or Verizon Wavelength Instance” which will allow us to host our instance on the HarperDB cloud (AWS). 

Lastly, choose the HarperDB Cloud in AWS option and click on the instance info button.

4- Fill in the necessary cloud data by entering the cloud name, username, and password. Click on the Instance Details button to move on to the next step. Choose the HarperDB package and click on the Add instance button.

5- Check your new cloud instance. After taking a couple of seconds to finish, a new cloud with the chosen cloud name will be shown on your main page.

Creating and hosting a HarperDB instance on Azure

We will start off by creating a virtual machine on Azure and then installing HarperDB on it. The operating system of the virtual machine can be Linux or Windows. For this tutorial, we will go for a windows virtual machine. To set up a windows virtual machine on Azure we can either do it through the comfort of our command line interface using az CLI or we can do it through the Azure portal.

You will first need an Azure account with a free subscription. After setting that up, you can easily create a virtual machine. First, we need to install azure CLI (this will differ depending on which OS you are on). After that, we need to

1. Login to Azure. Simply run:

az login

This will take you to a portal to login into your Azure account

2. Create a resource group for the virtual machine:

3. Create the virtual machine using that resource group:

az vm create –resource-group myResourceGroup –name myVM –image Win2022AzureEditionCore –public-ip-sku  Standard –admin-username azureuser

4. Open connections to that VM on port 80:

az vm open-port –port 80 –resource-group myResourceGroup –name myVM

The next step is to connect to the Azure virtual machine using SSH. For the sake of brevity, we will not include all of the steps, you can however find them here. You can also find it on the portal here:

After setting up and connecting to the virtual machine, we will start by sending a simple request from the VM to our AWS instance (HarperDB Cloud instance). This step will actually demonstrate real production scenarios where 2 databases hosted on different cloud providers need to communicate effectively and smoothly.

If we look at the HarperDB studio, on the right, we can find “example code”. This tab is quite useful as you can choose from various different programming languages and then you will get example codes that you can use directly to send different requests. For this tutorial, we will go with curl. Note if curl isn’t on your windows Azure virtual machine, you might need to quickly install it.

If we run this through our Azure VM (with SSH):

curl --location --request POST '<HarperDB cloud connection string>' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic bW9zdGFmYW9zYW1hOlJlZGxpZ2h0YmVzdDEyMyU=' \
--data-raw '{    
   "operation": "create_table",    
   "schema": "dev",    
   "table": "dog",    
   "hash_attribute": "id"
}'

Let's quickly break down this curl request. In the first line, we are specifying what type of HTTP request we want to be sending, in this case, it is a POST request. Following the request type is the instance connection string which we can grab directly from either the example code on HarperDB studio or the instance information. After that, we provide 2 headers, the first one specifies that the body of this POST request will be JSON format and the second one uses Basic Authorization (a simple encoded string) to authorize writing to the instance (assuming the auth is correct). After that is the body of the POST request.

Assuming the request was successful, we can quickly notice the creation of the dog table on HarperDB Studio:

Now that we have created a table, let's insert a record and see what happens. We can simply do so by running:

curl --location --request POST '<HarperDB cloud connection string>' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic bW9zdGFmYW9zYW1hOlJlZGxpZ2h0YmVzdDEyMyU=' \
--data-raw '{    
   "operation": "insert",    
   "schema": "dev",    
   "table": "dog",    
   "records": [       
      {           
          "id": 1,            
          "dog_name": "Penny",         
          "owner_name": "Kyle",            
          "breed_id": 154,           
          "age": 7,            
          "weight_lbs": 38        
       }    
   ]
}'

If successful, we can observe this record in HarperDB studio:

For the second part, we will set up a HarperDB instance on Azure:

1. We will start by installing npm
2. Then run:

Note that you might get a message mentioning the need for a different node version. If this is the case simply install nvm which can be used to switch node versions easily.

Simply run:

nvm install 14.19.1
nvm use 14.19.1
npm i -g harperdb

3. Export the appropriate environment variables, add in your username and password and install HarperDB

export TC_AGREEMENT=yes
export HDB_ADMIN_USERNAME=<username>
export HDB_ADMIN_PASSWORD=<password>
export HDB_ROOT=/tmp/hdb/
export SERVER_PORT=9925
harperdb install

4. Start HarperDB:

harperdb run

5. You can also start a custom service on HarperDB using Custom Functions. 

harperdb run –service harperdb,”custom functions”,ipc

And that is it! You now have your own HarperDB instance running on Azure in the cloud.

Conclusion

In this article, we learned about how HarperDB is enabling digital transformation solutions to reduce costs and eliminate vendor lock. We then explained multi-cloud environments and reasons that nowadays organizations utilize them. Moving on to the practical part of this article, we wanted to show that there is no dip in performance when using HarperDB on two different clouds, HarperDB can always talk to HarperDB if there is connectivity. To do this we created two HarperDB instances, one on HarperDB Cloud (AWS) and the other on Azure. We then sent a simple data message between both cloud providers.

I hope after reading this article you have grasped a much better understanding of multi-cloud deployments, what HarperDB is, and why you should start utilizing HarperDB to make your life easier.