In Part I of the series, we demonstrated how to stream AWS DynamoDB updates to HarperDB and downstream consumers using MQTT, WebSockets, and Server Side Events. In this article, we will demonstrate how to stream data from MongoDB Atlas in a similar manner. 

As a reminder, HarperDB provides real-time access to data via various protocols. This enables clients to monitor changes and streams this data in real-time by subscribing to topics. We will utilize the same demo code from Part I of this series to now stream data from MongoDB Atlas.

‍

Part 1: Streaming Data from MongoDB Atlas

MongoDB Atlas Setup

MongoDB Atlas is a multi-cloud database offering from the MongoDB team. Register for an account and follow the setup instructions to deploy your database:

1.Choose a deployment type: free, serverless, dedicated

2. Choose a cloud provider and region 

3. Customize settings: e.g., global coverage, fault tolerance, data compliance 

Make note of the database name as well as the collection name as you’ll need these later. 

Next, we’ll need to create an API key to give programmatic access to the data. Follow these instructions to create an API key and optionally require an IP access list to allow only access from your demo. 

Finally, we’ll need to create a database user to generate our connection string. Using the Atlas Admin role, create a new user and generate a new password. Make note of the connection string, which is in the the following format:
mongodb+srv://<username>:<password>@<clusterName>.mongodb.net

HarperDB Setup

If you are following from Part I, there’s no additional setup. If not, check out the HarperDB setup section from the first article to install HarperDB as well as the demo code. 

Now we need to modify the MongoDB portion of the `dbs/credentials/credentials.js` file.

Update this section accordingly using the information from the MongoDB Atlas setup section:

mongodb: {
   connectionString: 'mongodb+srv://XXXX',
   databaseName: 'XXXX',
   collectionName: 'XXXX',
 },

Sending Simulated Data

Now that we have everything set up, we are ready to send some data. The example code actually has a demonstration UI that lets you publish to MongoDB (it uses fake data underneath) and then pull that data into HarperDB. 

Like in Part I, if you would like to use the UI to send data, you can follow the UI setup portion of the README to install the UI portion. Again, note that if you are running locally, you may run into issues with CORS and will have to allow CORS on your browser. Alternatively, you can run the commands under `dbs/mongodb` manually as well.

The `ingest.js` file sends some random UUID with random lorem ipsum content. The `cdc.js` file handles Change Data Capture by utilizing the MongoDB client to get new data from MongoDB Atlas and publishes the records to HarperDB. 

Either using the UI or manually invoking the functions, try sending some data. You should see the fake data on HarperDB Studio populate like in the demo video.

‍

Part 2: Setting up MQTT, WebSockets, and Server Sent Event Subscriptions

Setting up subscriptions are handled via the `harperdb-config.yaml` file. Let’s go through each one in more detail. 

For MQTT, we will set the following config:

mqtt:
network:
port: 1883
securePort: 8883 # for TLS
webSocket: true # will also enable WS support through the default HTTP interface/port  
requireAuthentication: true

In the UI demo, locate the `MQTTWS.js` file. Here we can see subscriptions to MQTT using the configuration file. It parses the messages and updates React state to show the data. 

WebSockets utilize the REST interfaces and use the `connect(incomingMessages)` method on resources. In the `WS.js` file, you can see a new WebSocket connection. It uses the addEventListener functionality to listen to new messages and parses them like the MQTT connection. 

Finally, for Server Side Events, a new EventSource is added via config. Note that Server Side Events actually use the REST server interface underneath the hood..

‍

Wrapping Up

We were able to stream MongoDB Atlas updates to HarperDB and also use MQTT, WebSockets, and Server Side Events to downstream consumers. If you were following along, you can now see the MongoDB Atlas portion as well as the AWS DynamoDB portion from Part I working from the demo. In our next and final article, we’ll also demonstrate how to enable this for DataStax. 

‍