Backing up (and restoring) *All* Data in MongoDB

The documentation on Mongo’s website tells you to use mongodump with a username, password, destination, and which database you want to back up. Except I wanted to back up and restore everything. Users, multiple databases, I don’t really know what else is in there hence I want everything instead of enumerating the things I want.

Turns out you can just omit the database name and it dumps everything

mongodump --uri="mongodb://<host URL/IP>:<Port>" -u $STRMONGODBUSER -p $STRMONGODBPASS

And restore with

mongorestore --uri="mongodb://<host URL/IP>:<Port>"

Since it’s a blank slate with no authentication or users defined yet.

Redis High Availability — Options

The two primary approaches to high availability with Redis are Redis Sentinel and Redis Cluster. There are also third-party solutions, but the provided budget is zero dollars. That … limiting.

Sentinel is the official high availability solution provided by Redis. It monitors Redis instances, detecting failures, and automatically handling failover to a replica. It also provides monitoring/alerting to advise administrators when a problem has been detected.

Sentinel does not provide much in the way of scalability (it adds additional ‘read only’ copies, but there is a single master) but this architecture better ensures consistency (i.e. the same data is present on all nodes). It does, however, promote a read replica to master in the event the master fails, so high availability is achieved.

More than half of the Sentinels need to consider a master down to invoke failover (quorum) – so we would want at least three nodes. We experienced issues with two-node Microsoft quorum-based clustering when the two nodes were unable to communicate. Each node considered its partner to be ‘down’ and decided to be the server in charge. And having two servers in charge corrupts data. With three nodes, should they all become separated … they cannot reach a quorum of two servers agreeing on states.

Cluster automatically distributes data across multiple Redis nodes (called shards). Doing so allows more data to be processed in parallel. Redis Cluster also supports replication and automatic failover.

Since clustering provides both high availability and scaling, if the write load is a consideration, this may be a preferred option; but distributed data means inconsistent data values may be encountered. If data consistency is paramount, clustering may be undesirable. Additionally, not all Redis clients support communicating with a clustered environment. We would need to have our vendor confirm that the application could use a clustered solution.

The minimum recommended environment for production is larger – six servers. This constitutes three master nodes and three replica nodes.

MongoDB: Basics

We inherited a system that uses MongoDB, and I managed to get the sandbox online without actually learning anything about Mongo. The other environments, though, have data people care about set up in a replicated cluster of database servers. That seems like the sort of thing that’s going to require knowing more than “it’s a NoSQL database of some sort”.

It is a NoSQL database — documents are organized into ‘collections’ within the database. You can have multiple databases hosted on a server, too. A document is a group of key/value pairs with dynamic schema (i.e. you can just make up keys as you go).

There are GUI clients and a command-line shell … of course I’m going with the shell 🙂 There is a db function for basic CRUD operations using db.nameOfCollection then the operation type:

db.collectionName.insert({"key1": "string1", "key2" : false, "key3": 12345})
db.collectionName.find({key3 : {$gt : 10000} })
db.collectionName.update({key1 : "string1"}, {$set: {key3: 100}})
db.collectionName.remove({key1: "string1"});

CRUD operations can also be performed with NodeJS code — create a file with the script you want to run, then run “node myfile.js”

Create a document in a collection

var objMongoClient = require('mongodb').MongoClient;
var strMongoDBURI = "mongodb://mongodb.example.com:27017/";
  
objMongoClient.connect(strMongoDBURI, function(err, db) {
  if (err) throw err;
    var dbo = db.db("dbNameToSelect");
    var objRecord = { key1: "String Value1", key2: false };
    dbo.collection("collectionName").insertOne(objRecord, function(err, res) {
         if (err) throw err;
         console.log("document inserted");
         db.close();
    });
}); 

Read a document in a collection

var objMongoClient = require('mongodb').MongoClient;
var strMongoDBURI = "mongodb://mongodb.example.com:27017/";

objMongoClient.connect(strMongoDBURI, function(err, db) {
  if (err) throw err;
    var dbo = db.db("dbNameToSelect");
    var objQuery = { key1: "String Value 1" };
    dbo.collection("collectionName").find(objQuery).toArray(function(err, result) {
     if (err) throw err;
     console.log(result);
     db.close();
  });
}); 

Update a document in a collection

var objMongoClient = require('mongodb').MongoClient;
var strMongoDBURI = "mongodb://mongodb.example.com:27017/";

objMongoClient.connect(strMongoDBURI, function(err, db) {
if (err) throw err;
  var dbo = db.db("dbNameToSelect");
  var objRecord= { key1: "String Value 1" };
  dbo.collection("collectionName").deleteOne(objRecord, function(err, obj) {
    if (err) throw err;
    console.log("Record deleted");
    db.close();
});
}); 

Delete a document in a collection

var objMongoClient = require('mongodb').MongoClient;
var strMongoDBURI = "mongodb://mongodb.example.com:27017/";

objMongoClient.connect(strMongoDBURI, function(err, db) {
if (err) throw err;
  var dbo = db.db("dbNameToSelect");
  var objQuery = { key1: "String Value 1" };
  var objNewValues = { $set: {key3: 12345, key4: "Another string value" } };
  dbo.collection("collectionName").updateOne(objQuery, objNewValues , function(err, res) {
    if (err) throw err;
    console.log("Record updated");
    db.close();
   });
}); 

Tableau – Data Source Connection Info and Workbooks

I think I finally have a query that links workbooks where data sources are used and the connection information from the data_connections table!

-- Query to find all data sources and where they are used
select system_users.email
, datasources.id, datasources.name, datasources.created_at, datasources.updated_at, datasources.db_class, datasources.db_name, datasources.site_id
, data_connections.server, data_connections.dbclass
, sites.name as SiteName, projects.name as ProjectName, workbooks.name as WorkbookName
from datasources
left outer join data_connections on data_connections.datasource_id = datasources.id
left outer join users on users.id = datasources.owner_id
left outer join system_users on users.system_user_id = system_users.id
left outer join sites on datasources.site_id = sites.id
left outer join projects on datasources.project_id = projects.id
left outer join workbooks on datasources.parent_workbook_id = workbooks.id
order by datasources.name
;

Finding Benefits Anywhere

At a recent school board meeting, we had a lady suggesting a list of things they should teach about how slaves benefited … evidently this is some recent research? I propose we nab her and all of these researchers out of their houses, throw them in a crowded van, and take them to a prison facility for a year or three. While they are there, they will be fed well, entered into a program to become certified in a trade, and given free access to health care. They’ll be given free range of the facility, not locked in cells; but they’ll have to work and complete their training classes. They’ll be given clothing to wear, a bed in which to sleep. If they’re really lucky, their spouse and kids will be nabbed and get to “benefit” from this great service too.

Obviously, they’d be free to leave at any point they wanted — not an option for actual slaves. If they opt to leave early, they no longer get to claim slavery had benefits for anyone other than those exploiting free labor. They will be admitting that no matter how nicely you treat someone — and these folks are going to be treated far better than most slaves were, so they’re experiencing the best case scenario — doing it against their will is not benefiting them.

Blinds

We got two blinds from Tidwe that zip together and form a double-size blind … three sides of the blind are “see through” fabric. If you focus on the fabric (like my camera does), you really see it … but, focusing farther out it looks like you are looking through a screen window. It’s nice, shady, and really cool to sit in the middle of nature and observe.

Creating an Azure DevOps Work Item From a Teams Message

You can use Power Automate to create an ADO work item (bug, user story, etc) when a user posts into specific Teams channel.

Log into Power Automate and create a new workflow. Find a Teams trigger that suits your need – in my case, I wanted to use a key word (you could even use different key words to create work items in different projects or with different content). Note that automation cycles accrue based on execution — so if you elect to link up to a busy Teams channel and filter for keywords to indicate you want an ADO item created, you may be “wasting” workflow cycles. In our case, I have a “user group” Teams space and set up a special channel where users can submit bug and feature requests. This means workflow cycles are accrued when someone specifically wishes to create an ADO item not when messages are posted into the user group’s general chat channels.

You can source messages from channels or group chats in the “Message type” selection. You cannot use hash-tags as key words! The workflow execution reports a gateway error. Select the Team and channel(s) that you want the workflow to monitor.

Add a new step to “create a work item” from Azure DevOps

Configure the project into which you want to create the work item – the organization and project name, the type of work item, and content of the work item.

If you want to set priority, add an assignment, etc – click on “Show advanced options”. I added a few fields to provide a clue as to where the bug report came from.

Save the workflow and post a message in your channel with the key word. Go into the ADO project work items; your Teams-initiated bug should be there.