Category: Technology

Persisting Port Names For OpenHAB

If you only have one device connected to your Linux box, your controller may well always be assigned the same port when the system is rebooted. In the real world, multiple connected devices make this unlikely. We use udev rules to create symlinks used within the OpenHAB configuration. The udev rule essentially uses attributes of the device to identify the real port and creates a statically named symlink to that port on boot. So the first thing you need to identify is something unique about the device. We have several video capture cards and a Z-Wave/ZigBee combo controller attached to our server. If you do not have udevinfo, you can use lsusb to find details about the device. First list the devices, then identify the proper one and use the -d switch with the ????:???? formatted ID number. The -v switch outputs verbose information. Find a unique attribute or set of attributes that create a unique identifier. In this case, the interface name is unique and we can stop there.

[lisa@server ~]#  lsusb
Bus 001 Device 004: ID 0bda:0111 Realtek Semiconductor Corp. RTS5111 Card Reader Controller
Bus 001 Device 003: ID 1058:1230 Western Digital Technologies, Inc. My Book (WDBFJK0030HBK)
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 10c4:8a2a Cygnal Integrated Products, Inc.
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
[lisa@server ~]# lsusb -d 10c4:8a2a -v
Bus 002 Device 002: ID 10c4:8a2a Cygnal Integrated Products, Inc.
Device Descriptor:
 bLength 18
 bDescriptorType 1
 bcdUSB 2.00
 bDeviceClass 0
 bDeviceSubClass 0
 bDeviceProtocol 0
 bMaxPacketSize0 64
 idVendor 0x10c4 Cygnal Integrated Products, Inc.
 idProduct 0x8a2a
 bcdDevice 1.00
 iManufacturer 1 Silicon Labs
 iProduct 2 HubZ Smart Home Controller
 iSerial 5 90F0016B
 bNumConfigurations 1
 Configuration Descriptor:
 bLength 9
 bDescriptorType 2
 wTotalLength 55
 bNumInterfaces 2
 bConfigurationValue 1
 iConfiguration 0
 bmAttributes 0x80
 (Bus Powered)
 MaxPower 100mA
 Interface Descriptor:
 bLength 9
 bDescriptorType 4
 bInterfaceNumber 0
 bAlternateSetting 0
 bNumEndpoints 2
 bInterfaceClass 255 Vendor Specific Class
 bInterfaceSubClass 0
 bInterfaceProtocol 0
 iInterface 3 HubZ Z-Wave Com Port
 Endpoint Descriptor:
 bLength 7
 bDescriptorType 5
 bEndpointAddress 0x81 EP 1 IN
 bmAttributes 2
 Transfer Type Bulk
 Synch Type None
 Usage Type Data
 wMaxPacketSize 0x0040 1x 64 bytes
 bInterval 0
 Endpoint Descriptor:
 bLength 7
 bDescriptorType 5
 bEndpointAddress 0x01 EP 1 OUT
 bmAttributes 2
 Transfer Type Bulk
 Synch Type None
 Usage Type Data
 wMaxPacketSize 0x0040 1x 64 bytes
 bInterval 0
 Interface Descriptor:
 bLength 9
 bDescriptorType 4
 bInterfaceNumber 1
 bAlternateSetting 0
 bNumEndpoints 2
 bInterfaceClass 255 Vendor Specific Class
 bInterfaceSubClass 0
 bInterfaceProtocol 0
 iInterface 4 HubZ ZigBee Com Port
 Endpoint Descriptor:
 bLength 7
 bDescriptorType 5
 bEndpointAddress 0x82 EP 2 IN
 bmAttributes 2
 Transfer Type Bulk
 Synch Type None
 Usage Type Data
 wMaxPacketSize 0x0020 1x 32 bytes
 bInterval 0
 Endpoint Descriptor:
 bLength 7
 bDescriptorType 5
 bEndpointAddress 0x02 EP 2 OUT
 bmAttributes 2
 Transfer Type Bulk
 Synch Type None
 Usage Type Data
 wMaxPacketSize 0x0020 1x 32 bytes
 bInterval 0

We then need to create a file under /etc/udev/rules.d. The file name begins with a number that is used for a load order – I generally number my custom files 99 to avoid interfering with system operations. The bit in “ATTRS” is the attribute name and value to match. The KERNEL section contains the search domain (i.e. look at all of the ttyUSB### devices and find ones where the interface is this). The symlink bit is the name you want to use (more on this later). Set the group and mode to ensure OpenHAB is able to use the symlink.

[lisa@server rules.d]# cat 99-server.rules
KERNEL=="ttyUSB[0-9]*", ATTRS{interface}=="HubZ Z-Wave Com Port", SYMLINK+="ttyUSB-5", GROUP="dialout", MODE="0666"
KERNEL=="ttyUSB[0-9]*", ATTRS{interface}=="HubZ ZigBee Com Port", SYMLINK+="ttyUSB-55", GROUP="dialout", MODE="0666"

The symlink name can be anything – when I created udev rules for our video capture cards, I named them something immediately obvious: video-hauppauge250 is the Hauppauge 250 card. Tried to do the same thing here, naming the ports controller-zigbee; while the symlink appeared and had the expected ownership and permissions … OpenHAB couldn’t use it.

Turns out there’s a nuance to Java RxTx where non-standard port names need to be accommodated in the java options. So I could have added -Dgnu.io.rxtxSerialPorts=/dev/controller-zigbee and -Dgnu.io.rxtxSerialPorts=/dev/controller-zwave to the OpenHAB startup and been OK (in theory), it was far easier to name the symlinks using the Linux standard conventions. Hence I have symlinks named ttyUSBsomethingsomethingsomething. 

SAN Certificates From OpenSSL CA

For some reason, I had to combine three different sets of instructions to get a SAN added to my certificate. Getting the SAN into the request was easy enough … but actually carrying the extension through to the signed certificate was a significant challenge. There may be unnecessary changes in my custom config file, but this process worked. 

cp /etc/pki/tls/openssl.cnf ./myssl.cnf

Edit the copied file (i.e. don’t change your OpenSSL default config)
# Uncomment:
copy_extensions = copy
# Uncomment:
req_extensions = v3_req # The extensions to add to a certificate request

 

# Add:
[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = exchange01.rushworth.us
DNS.2 = exchange01

 

Save the file and we’re ready to create a certificate. Make a key

openssl genrsa -aes256 -out exchange01.rushworth.us.key 2048

Then create the cert request using the copied config file. Include the -reqexts option with value of the section of your custom file that includes subjectAltName (e.g. it is called req_ext in my cnf file, so I used -reqexts req_ext)
openssl req -new -key exchange01.rushworth.us.key -config ./myssl.cnf -reqexts req_ext -out exchange01.rushworth.us.csr

Sign the request against your CA – again using the custom config file and req_ext extensions
openssl x509 -req -in exchange01.rushworth.us.csr -extensions req_ext -extfile ./myssl.cnf -out exchange01.rushworth.us.cer -days 365 -CA /ca/ca.cer -CAkey /ca/ca.key -sha256

Before doing anything else, verify that your SAN values are in the certificate

[lisa@linux02]# openssl x509 -in exchange01.rushworth.us.cer -text | grep -A1 Alternative
X509v3 Subject Alternative Name:
DNS:exchange01.rushworth.us, DNS:exchange01

If you are using the certificate in something that understands PEM nodes, you are set. If you are trying to get a certificate for a Windows server, create a PFX export of the public/private key pair and then import the PFX to your computer’s personal certificate store.

openssl pkcs12 -export -out exchange01.rushworth.us.pfx -inkey exchange01.rushworth.us.key -in exchange01.rushworth.us.cer

OpenSSL As A Trusted CA

There are wrappers for OpenSSL that provide certificate authority functionality, but I found myself spending a lot of time trying to get any to work. Since I only wanted to generate a few internal certificates (i.e. not something that needed a simple interface for non-techies), so I set up an OpenSSL certificate authority and used it to sign certificates.

First, generate a public/private keypair for your CA (use however many days you want, this is ten years:

openssl genrsa -aes256 -out ca.key 2048
openssl req -new -x509 -key ca.key -out ca.cer -days 3652 -sha256

Take ca.cer and publish it in our domain GPO as a trusted root certificate authority (Computer Configuration => Policies => Windows Settings => Security Settings => Public Key Policies => Trusted Root Certification Authorities)

If you are impatient, force client to update GPO. Otherwise wait. Eventually you will see your CA in the Windows computer’s certificate store as a trusted root certification authority.

Now generate certificate(s) against the CA (again use whatever value for days is reasonable for your purpose):

openssl genrsa -aes256 -out gitlab.rushworth.us.key 2048
openssl req -new -key gitlab.rushworth.us.key -out gitlab.rushworth.us.req
openssl x509 -req -in gitlab.rushworth.us.req -out gitlab.rushworth.us.cer -days 365 -CA /ca/ca.cer -CAkey /ca/ca.key -sha256 -CAcreateserial

On subsequent requests, you can omit the “-CAcreateserial” option.

In domain clients will trust your certificate. Non-domain clients will need to import the CA public key to their trust store.

Git Deployment

I ‘inherited’ the Git server at work — which means I had to learn how the back end component of Git works (beyond my file-system based implementation where there are just clients and a disk location). It is not as complicated as I feared. The chap who had deployed the Git backend at work chose Bonobo — since he no longer works for the company, I cannot just ask why this particular implementation. It’s Windows based and priced in our 0$ budget, and I am certain these were selling points. It seems quite stripped down compared to GitHub too — none of the issue tracking / Wiki / chat about it features. Which, for what my department does, is fine. We are not software developers. We have a lot of internal code for task automation, we have some internal code for departmental web sites, and we have some sample code we hand out to other developers (i.e. someone wants to start using LDAP or ADFS authentication, we can give them a sample implementation in their language). There aren’t feature requests. Generally speaking, there aren’t simultaneous development tasks on a project.

Since I deciphered the server implementation at work, I wanted to set up a Git server at home too. The limited feature set of Bonobo was off-putting. I wanted integrated issue tracking. Looking at the available opensource and free options, I selected GitLab. As a sandbox — poke around the server, see how it works and what features it offers — I wanted something ready-to-go. I noticed that there is a Docker container for the project. I helped a few friends who were testing Docker as a development and deployment methodology (I’ve even suggested it for my employer’s internal development staff … being able to develop and run an application with an integrated web server *without* needing the Windows permissions and configuration for a web server (and doing it all over again when your computer is replaced) seemed efficient. But I’d never actually used a Docker container before. It is incredibly easy.

Install docker — a bit obvious, but that was the most time consuming part of the process. I elected to install it on my Windows laptop for expediency. If we decide not to use GitLab, I haven’t thrown a bunch of unnecessary binaries on the server. Lenovo, as a default, does not enable virtualisation. Getting into the BIOS config tool (shift then click the power button, keep holding shift whilst you click restart) was the most time consuming bit of the installation.

Once Docker is installed, pull the container from the Docker store (docker pull gitlab/gitlab-ce). Then run it (docker run –detach –hostname gitlab.rushworth.us –publish 443:443 –publish 80:80 –publish 22:22 –name gitlab –restart always –volume /srv/gitlab/config://c/gldata/etc –volume /srv/gitlab/logs:/var/log/gitlab –volume /srv/gitlab/data://c/gldata/data –volume /svr/docker/gitlab/gitlab://c/gldata/gitlab gitlab/gitlab-ce:latest). You can remap ports (e.g. publish 8443:443) if needed.

Not quite there yet — you’ve got to edit the container config (docker exec -it gitlab vi /etc/gitlab/gitlab.rb) for your environment. Set a valid external url (external_url ‘http://gitlab.rushworth.us’). I also enabled LDAP authentication to test that out.

gitlab_rails[‘ldap_enabled’] = true

###! **remember to close this block with ‘EOS’ below**
gitlab_rails[‘ldap_servers’] = YAML.load <<-‘EOS’
main: # ‘main’ is the GitLab ‘provider ID’ of this LDAP server
label: ‘LDAP’
host: ‘ADHostname.rushworth.us’
port: 636
uid: ‘sAMAccountName’
method: ‘ssl’ # “tls” or “ssl” or “plain”
bind_dn: ‘cn=UserID,ou=SystemAccounts,dc=domain,dc=ccTLD’
password: ‘AccountPasswordGoesHere’
active_directory: true
allow_username_or_email_login: false
block_auto_created_users: false
base: ‘ou=ResourceUsers,dc=domain,dc=ccTLD’
user_filter: ‘(&(sAMAccountName=*))’ # Can add attribute value to restrict authorized users to GitLab access, we leave open to all valid user accounts in the OU. Should be able to authorize based on group membership using linked attribute value like (&(memberOf=cn=group,ou=groupOU,dc=domain,dc=ccTLD))
attributes:
username: [‘uid’, ‘userid’, ‘sAMAccountName’]
email: [‘mail’, ’email’, ‘userPrincipalName’]
name: ‘cn’
first_name: ‘givenName’
last_name: ‘sn’

EOS

The default is to retain a lot of log files — 30 days! This might be reasonable in a corporate environment, but even for production at home … that’s a lot of space dedicated to log files.

logging[‘logrotate_frequency’] = “daily” # rotate logs daily
logging[‘logrotate_rotate’] = 3 # keep 3 rotated logs
logging[‘logrotate_compress’] = “compress” # see ‘man logrotate’
logging[‘logrotate_method’] = “copytruncate” # see ‘man logrotate’

And finally configure SMTP for outbound mail. We don’t use authentication on our SMTP server; it controls relay based on source IP. We do use starttls, but the certificate is not going to be trusted without additional configuration … so I set the ssl verify mode to none.

gitlab_rails[‘smtp_enable’] = true
gitlab_rails[‘smtp_address’] = “smtp.hostname.ccTLD”
gitlab_rails[‘smtp_port’] = 25
# gitlab_rails[‘smtp_user_name’] = “smtp user”
# gitlab_rails[‘smtp_password’] = “smtp password”
# gitlab_rails[‘smtp_domain’] = “example.com”
# gitlab_rails[‘smtp_authentication’] = “login”
gitlab_rails[‘smtp_enable_starttls_auto’] = true
# gitlab_rails[‘smtp_tls’] = false

###! **Can be: ‘none’, ‘peer’, ‘client_once’, ‘fail_if_no_peer_cert’**
###! Docs: http://api.rubyonrails.org/classes/ActionMailer/Base.html
gitlab_rails[‘smtp_openssl_verify_mode’] = ‘none’

Once the config has been updated, restart the container (docker restart gitlab).

Access the web site and you’ll be prompted to set a password for the admin user, root. You can click the ‘ldap’ tab and log in with Active Directory credentials. Fin.

If we deploy this for a production system, I would set up SSL on the web site and possibly externalize the GitLab database to MySQL. The external database is more of an academic experiment because we already use MySQL (and I still don’t want  to learn about vacuuming PostgreSQL).

Apache Airflow — No Backfill

A lot of software seems to be designed to save the user from themselves. This is great 90% of the time when you mess up and really want their help (or when the software’s help is cosmetic … my gripe against auto-correcting smart quotes, as an example). But I seem to fall into the other 10% a lot. And I mean a LOT. Apache Airflow jobs try to grab new information all.of.the.time. It’s a feature called “backfill”, and I’m sure it helps all sorts of people do exactly what they really wanted done. Not me 🙁

Having updated to 1.8, though, I now see a configuration parameter to instruct a DAG not to do me any favors. Just do what you’re asked when you’re asked to do it: catchup = False

DAG('testjob', default_args=default_args, schedule_interval='0 * * * *', catchup=False)

Android Mail Client Malfunction On FierceXL

Both Scott and I have an odd issue with our FierceXL using the stock mail client to communicate with Exchange 2013 over the OWA interface. Randomly, one or more of the connected accounts stops receiving e-mail. We know OWA still works and is available from the phone — we can go into Chrome on the phone and log into OWA. There is absolutely no traffic coming across the reverse proxy / Exchange server from the phone IP. Switching between the cellular network and home WiFi has no impact.

I had just been rebooting my phone. Upon startup, communication is again seen on the reverse proxy server. A few seconds later, the backlog of new mail starts popping into the mail client. Scott recently discovered that you can resolve the issue by closing the mail app (bringing up the recent/running application list and swiping mail off of the screen) and re-opening it.

It appears something within the mail client is getting a thread hung — not the mail client en toto as I often cease receiving messages to one of three accounts. Ending the process and re-spawning it clears whatever is hung. Unfortunately, we have not had any updates for these phones since November of last year so there’s not a quick software fix that can be applied to resolve the issue.

The Peril Of Hosting Your Own Services

I love hosting my own services — home automation, file shares, backups, e-mail, web servers, DNS … bit of paranoia, a bit of control freak, and a bit of pride. But every now and again, hosting my own services causes problems because, well, vendors don’t develop processes around someone with servers in their house.

We got a new cable modem. Scott went to a web page (happened to be Google) and got redirected to the TWC activation page. Went through whatever, ended up calling into support, and finally our account was sorted. Woohoo! Everything works … umm, except I cannot search Google.

Turns out TWC manages their activation redirection by serving up bogus DNS info — their server IP instead of the real one. Which then got cached on our DNS server. No idea what TTL TWC set on their bogus data, but it was more than a minute or two. Had to clear the DNS server cache before we were able to hit Google sites again.

OK, Google

Chrome 58 was released last month – and since then, I’ve gotten a LOT of certificate errors. Especially internally (Windows CA signed certs @ home and @ work). It’s really annoying – yeah, we don’t have SAN dnsHost attributes defined. And I know the RFC says falling back to CN is deprecated (seriously, search https://tools.ietf.org/html/rfc2818 for subjectAltName) but the same text was in there in 1999 … so not exactly a new innovation in SSL policy. Fortunately there’s a registry key that will override this for now.

The problem I have with SAN certificates is exemplified in Google’s cert on the web server that hosts the chromium changes site:

Seriously – this certificate ensures that the web site is any of these hundred wild-carded hostnames … and the more places you use a certificate, the greater the possibility of it being compromised. I get why people like wildcards — UALR was able to buy one cert & use it across the entire organisation. Cost effective and easy. The second through nth guy who wanted an SSL cert didn’t need to go about establishing his credentials within the organisation. He didn’t have to figure out how to make a cert request or how to pay for it. Just ask the first guy for a copy of his public/private key pair. Or run everything through your load balancer on the wildcard certificate & trust whatever backend cert happens to be in place.

But the point of security design is not trusting large groups of people do act properly. To secure their data appropriately. To patch their systems, configure their system to avoid attacks, to replace the certificate EVERYWHERE every TIME someone leaves the organisation, and otherwise prevent a certificate installed on dozens of servers from being accessed by a malicious party. My personal security preference would be seeing a browser flag every time a cert has a wildcard or more than one SAN.

Exchange Online

We’re moving users to the magic in-the-cloud Exchange. Is this a cost effective solution? Well – that depends on how you look at the cost. The on prem cost includes a lot of money to external groups that are still inside the company. If the SAN team employs ten people … well, that’s a sunk cost if they’re administering our disk space or not. If we were laying people off because services moved out to magic cloud hosted locations … then there’s a cost savings. But that’s not reality. Point being, there’s no good comparison because the internal “costs” are inflated. Microsoft’s pricing to promote cloud adoption means EOL is essentially free with purchase too. I’m sure the MS cost will go up in the future — I remember them floating “leased” software back in the late 90’s (prelude to SaaS) and thinking that was a total racket. You move all your licensing to this convenient “pay for what you use” model. And once a plurality of customers have adopted the licensing scheme, start bumping up rates. It’s a significant undertaking to migrate over – but if I’m saving hundreds of thousands of dollars a year … worth it. Rates go up, and the extra fifty grand a year isn’t worth the cost and time for migrating back to on prem. And next year that fifty grand more isn’t worth it either. Economies of scale say MS (or Amazon, or whomever) can purchase ten thousand servers and petabytes of disk space for less money than I can get two thousand servers and a hundred terabytes … but they want to make a profit too. There might be a small cost savings in the long term, but nothing like the hundreds of thousands we’re being sold up front.

Regardless – business accounting isn’t my thing. A lot of it seems counter-productive if not outright nonsensical. There are actually features in Exchange Online that do not exist in the on prem solution. The one I discovered today is subaddressing. At home, we use the virtusertable in sendmail to map entire subdomains to a single mailbox. This means I can provide a functional e-mail address, on the fly, to a new company and have mail delivered into my mailbox. Works fine for a small number of people, but it is not a scalable solution. Some e-mail providers started using a delimiter after which any string was ignored. This means I could have a GMail account of DevNull@gmail.com but get mail as DevNull+SomeRandomString@gmail.com or DevNull+CompanyNameHere@gmail.com … great for identifying who is losing your e-mail address out in Internet-land. Also somewhat trivial to write a rule that takes +SomeCompromisedAddress and move it to trash. EOL lets us do that.

Another interesting feature that is available on prem but not convenient is free busy federation (now termed an “organisational relationship”). In previous iterations, both parties needed to establish firewall rules (and preferably a B2B connection) to transfer the free busy data. But two companies with MS tenants should be able to link up without having to enact firewall changes. We still connect to the tenant. The other party still connects to the tenant. It’s our two tenants that communicate via MS’s network. Something I’m interested in playing around with … might try to see if we can link our sandbox tenant up to the production one just to see what exactly is involved.

Git, Version Management, Branches, and Sub-modules

As we have increased in staff, we’ve gained a few new programmers. While it was easy enough for us to avoid stepping on each other’s toes, we have experienced several production problems that could be addressed by rethinking our repository configuration.

Current state: We have a monolithic repository for different batch servers. Each server has a clone of the repository, and the development equivalent has a clone of the same repository. The repository has top-level folders for each independent script. There is a SharedTools top-level folder for reusable functions.

Changes are made on forks located both on the development server and individuals’ computers, tested on the development server, then pushed to the repo. Under a CRQ, a pull is performed from the production server to elevate the new code. Glomming dozens of scripts into a single repository was simple and quick; but, with new people involved with development efforts, we have experienced challenges with changes being lost, unintentional elevation of code, and having UAT run against under-development code.

Pitfalls: Four people working on four different scripts are working in the same repository. We have had individuals developing on their laptop overwrite changes (force push is dangerous, even if force-with-lease is used), we have had individuals developing on the dev server commit other people’s edits (git add * isn’t a good idea in a shared environment – specifically add changed files to your commit), and we’ve had duplication of effort (which is certainly a problem outside of development efforts, and one that can be addressed outside of git).

We could address the issues we’ve seen through training and communication – ensure anyone contributing code to the repository adequately understands what force push means, appreciates what wildcards include, and generally have a more nuanced understanding of git than the one-hour training I provided last year. But I think we should consider the LOE and advantages of using a technical solution to ensure less experienced git users are able to successfully use our repositories.

Proposal – Functional Splits:

While we have a few more individuals with development experience, they are quite specifically Windows script developers (PowerShell, VBScript, etc). We could just stop using the Windows batch server and let the two or three Microsoft guys figure it out for themselves. This limits individual growth – I “don’t do” PowerShell development, the Windows guys don’t learn Linux. And, as the group changes over time, we have not addressed the underlying problem of multiple people working on the same codebase.

Proposal – Git Changes:

We can begin using branches for development efforts and reserve “master” for ready-for-deployment code. Doing so, we eliminate the possibility of inadvertently elevating code before it is ready – only commands targeted to “origin master” will be run on production servers.

Using descriptive branch names (Initials-ScriptFolderName-SummaryOfChange) will help eliminate duplicated efforts. If I notice we need to send a few mass mails with inline images, seeing “TJR-sendMassMail-AddInlineImages” in the branch list lets me know you’ve got it covered. And “TJR-sendMassMail-RecipientListFromLiveLDAPQuery” lets me know you’re working on something else and I’m setting myself up for merge challenges by working on my change right now. If both of our changes are high priority, we might choose to work through a merge operation. This would be an informed, upfront decision instead of a surprise message indicating that fast-forward merging is not possible.

In large development projects, branch management can become a full-time pursuit. I do not think that will be an issue in our case. Minimizing the number of branches used, and not creating branches based on branches, makes branch management a simpler task. We should be able to perform fast-forward merges to push code into master because our branches modify different files in the repository.

To begin a development effort, create a branch and push it to the git server. Make your changes within that branch, and ensure you keep your branch in sync with master – you cannot merge branches that are “behind” into master without force. Once you are finished with your development, merge your branch into master and delete your branch. This approach will require additional training to ensure everyone understands how to create, rebase, merge, and delete branches (and not to just force operations because it lets you complete your task).

Instead of using ‘master’ for production code, the inverse is equally viable: create a “stable” branch that is for production code and only pull that branch to PROD servers. I believe this approach is done to prevent accidental changes to prod code – you’ve got to intentionally target “origin stable” with an operation to impact production code.

Our single repository configuration is a detriment to using branches if development is performed on the DEV server. To illustrate the issue, create a BranchTesting repo and add a single file to master. Create a Branch1 branch in one command window and check it out. Create a Branch2 in a second command window and check it out. In your first command window, add a file and commit it. In your second command window, add a file and commit it. You will find that both files have been committed to Branch2.

How can we address this issue?

Develop on our individual workstations instead of the DEV server. Not sharing a file set for our development efforts eliminates the branch context switching problem. If you clone the repo to your laptop, Gary clones the repo to his laptop, and I clone the repo to my laptop … you can create TJR-sendMassMail-AddInlineImages on your computer, write and test the changes locally, commit the changes and pull them to the DEV server for more robust testing, and then merge your changes into master when you are ready to elevate the code. I can simultaneously create LJR-monitorLDAPReplication-AddOUD11Servers, do my thing, commit changes and pull them to the DEV server (first using “git branch” to determine if someone else is already testing their branch on the DEV server), and merge my stuff into master when I’m ready to elevate. Other than remembering to ensure you verify that DEV has master checked out (i.e. no one else is testing, so the resource is free), we do not have resource contention.

While it may not be desirable to fill up our laptop drives with the entire code set from six different application servers, sparse-checkout allows you to select the specific folders that will come down to your fork.

The advantage of this approach is that it has no initial LOE beyond training and process change. The repositories are left as-is, and we start using them differently.

Unfortunately, this approach may not be viable in some instances – when access to data sources is restricted by IP ACL, you may not be able to do more than linting on your laptop. It may not even be possible to configure a Windows laptop to run some of our code – some Linux requirements are difficult to address in Windows (the PKI website’s cert info check, for instance), and testing code on Windows may not ensure successful operation on the Linux hosts.

Break the monolithic repositories into discrete repositories and use submodules allow the multiple independent repositories to be “rolled up” into a top-level repository. Development is done in the submodule repositories. I can clone monitorLDAPReplication, you can clone sendMassMail, etc. changes can be made within our branches of these completely different repositories and merged into the individual repository’s master branch for release to the production environment. Release can be done for the superset (“–recurse-submodules”) or individual sub-modules.

This would require splitting a repository into its individual components and configuring the sub-module relationships. This can be a scripted operation, and it is an incremental change to the script I used to create the repositories and ingest code; but the LOE for implementation is a few days of script writing / testing. Training will be required to ensure individuals can register their submodules within the top-level repo, and we will need to accustom ourselves to maintaining individual repos.

Or just break monolithic repositories into discrete repositories. The level of effort is about the same initially, but no one needs to learn how to set up a new submodule. We lose single-repo conveniences, but there’s literally no association between our different script folders where someone working in X could inadvertently impact Y.