MongoDB: Increasing Log Level

Avatar
by Lisa

We had a problem with an application accessing our MongoDB cluster, and the log files didn’t provide much useful information. I could see the client connect and disconnect … but nothing in between. I discovered that the default logging level is very low. Good for disk utilization and I/O, but not great for troubleshooting.

db.runCommand({getParameter: 1, logLevel: 1}) # Get the current logging level
db.setLogLevel(3) # Fairly robust logging
db.setLogLevel(5) # don't try this is prod huge stream of text super logging
db.setLogLevel(0) # and set logging back to a low level once you are done troubleshooting

You can also increase the log level for individual components of MongoDB to minimize logging I/O:

db.setLogLevel(2, "accessControl" )

 

Tableau: Workbooks and Views Created or Modified By a Specific Individual

Avatar
by Lisa

I had a manager looking to locate a ‘something in Tableau’ that was created by a specific individual — in this case, it was a terminated employee so “just ask the person” was not a viable solution. I put together a query to list all workbooks owned by or modified by an individual:

SELECT w.id, w.name, w.description, w.owner_id, w.modified_by_user_id, owner_system_users.email AS owner_email, modified_system_users.email AS modifier_email
     FROM  public.workbooks AS w
      LEFT OUTER JOIN public.users AS owner_users on w.owner_id = owner_users.id
      LEFT OUTER JOIN public.users AS modified_users ON w.owner_id = modified_users.id
      LEFT OUTER JOIN public.system_users AS owner_system_users ON owner_system_users.id = owner_users.system_user_id
		LEFT OUTER JOIN public.system_users AS modified_system_users ON modified_system_users.id = modified_users.system_user_id
      WHERE owner_system_users.name = 'UserLogonID';
--      WHERE owner_system_users.email LIKE '%Smith%' OR modified_system_users.email = '%Smith%'
		;

As well as a query to identify all views owned by an individual:

SELECT views.*, owner_system_users.email AS owner_email
     FROM  public.views 
      LEFT OUTER JOIN public.users AS owner_users on views.owner_id = owner_users.id
      LEFT OUTER JOIN public.system_users AS owner_system_users ON owner_system_users.id = owner_users.system_user_id

      WHERE owner_system_users.name = 'UserLogonID';
--      WHERE owner_system_users.email LIKE '%Smith%' OR modified_system_users.email = '%Smith%'
		;

The email address based search is most reasonable — our email addresses are algorithmically based on our names, so we always know what the address would have been. Many contractors, however, don’t have Office 365 licenses or mailboxes … so I have to fall back to finding their logon ID in those cases.

MongoDB: Setting Up a Replica Set

Avatar
by Lisa

On one server create a key file. Copy this key file to all other servers that will participate in the replica set

mkdir -p /opt/mongodb/keys/
openssl rand -base64 756 > /opt/mongodb/keys/$(date '+%Y-%m-%d').key
chmod 400 /opt/mongodb/keys/$(date '+%Y-%m-%d').key
chown -R mongodb:mongodb /opt/mongodb/keys/$(date '+%Y-%m-%d').key

On each server, edit /etc/mongo.conf and add the keyfile to the security section and define a replica set

security:
 authorization: enabled
 keyFile:  /etc/mongodb/keys/mongo-key
#replication:
replication:
  replSetName: "myReplicaSet"

Restart MongoDB on each node.

On one server, use mongosh to enter the MongDB shell.

rs.initiate(
{
_id: "myReplicaSet",
members: [
{ _id: 0, host: "mongohost1.example.net" },
{ _id: 1, host: "mongohost2.example.net" },
{ _id: 2, host: "mongohost3.example.net" }
]
})

Use rs.status() to view the status of the replica set. If it is stuck in STARTING … check connectivity. If the port is open, I ran into a snag with some replacement servers. They’ve got temporary hostnames. But you cannot add a host on itself — it ignores that you typed mongohost1.example.net … and it takes it’s hostname value. And then sends that value to the other servers in the replica set. If you cannot change the hostname to match what you want, there is a process to change the hostname in a replicaset.

MongoDB: Where is my shell?!?

Avatar
by Lisa

We are upgrading servers from really old MongoDB (4.2.15) to much new MongoDB (6.something). I am used to getting into the MongoDB shell using:

mongoserver:~ # mongo -u $STRMONGOUSER -p $STRMONGOPASS
MongoDB shell version v4.2.15
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("5658a72f-fea0-4316-aa97-4b0c0ffab7ff") }
MongoDB server version: 4.2.15

Except the new server says there’s no such file. And there isn’t. A bit of research later, I learn that the shell is now called mongosh … which is a more reasonable name. It works the same way: mongosh -u $STRMONGOUSER -p $STRMONGOPASS gets me there, and all of the commands I know work.

Backing up (and restoring) *All* Data in MongoDB

Avatar
by Lisa

The documentation on Mongo’s website tells you to use mongodump with a username, password, destination, and which database you want to back up. Except I wanted to back up and restore everything. Users, multiple databases, I don’t really know what else is in there hence I want everything instead of enumerating the things I want.

Turns out you can just omit the database name and it dumps everything

mongodump --uri="mongodb://<host URL/IP>:<Port>" -u $STRMONGODBUSER -p $STRMONGODBPASS

And restore with

mongorestore --uri="mongodb://<host URL/IP>:<Port>"

Since it’s a blank slate with no authentication or users defined yet.

Redis High Availability — Options

Avatar
by Lisa

The two primary approaches to high availability with Redis are Redis Sentinel and Redis Cluster. There are also third-party solutions, but the provided budget is zero dollars. That … limiting.

Sentinel is the official high availability solution provided by Redis. It monitors Redis instances, detecting failures, and automatically handling failover to a replica. It also provides monitoring/alerting to advise administrators when a problem has been detected.

Sentinel does not provide much in the way of scalability (it adds additional ‘read only’ copies, but there is a single master) but this architecture better ensures consistency (i.e. the same data is present on all nodes). It does, however, promote a read replica to master in the event the master fails, so high availability is achieved.

More than half of the Sentinels need to consider a master down to invoke failover (quorum) – so we would want at least three nodes. We experienced issues with two-node Microsoft quorum-based clustering when the two nodes were unable to communicate. Each node considered its partner to be ‘down’ and decided to be the server in charge. And having two servers in charge corrupts data. With three nodes, should they all become separated … they cannot reach a quorum of two servers agreeing on states.

Cluster automatically distributes data across multiple Redis nodes (called shards). Doing so allows more data to be processed in parallel. Redis Cluster also supports replication and automatic failover.

Since clustering provides both high availability and scaling, if the write load is a consideration, this may be a preferred option; but distributed data means inconsistent data values may be encountered. If data consistency is paramount, clustering may be undesirable. Additionally, not all Redis clients support communicating with a clustered environment. We would need to have our vendor confirm that the application could use a clustered solution.

The minimum recommended environment for production is larger – six servers. This constitutes three master nodes and three replica nodes.