Category: System Administration

Preventing erronious use of the master branch on development servers

One of the web servers at work uses a refspec in the “git pull” command to map the remote development branch to the local remote-tracking master branch. This is fairly confusing (and it looks like the dev server is using the master branch unless you dig into how the pull is performed), but I can see how this prevents someone from accidentally typing something like “git checkout master” and really messing up the development environment. I can also see a dozen ways someone can issue what is a completely reasonable git command 99% of the time and really mess up the development environment.

While it is simple enough to just checkout the development branch, doing so does open us up to the possibility that someone will erroneously  deliver the production code to the development server and halt all testing. While you cannot create shell aliases for multi-word commands (or, more accurately, alias expansion is performed for the first word of a simple command is checked to see if it has an alias … so you’ll never get the multi-word command), you can define a function to intercept git commands and avoid running unwanted commands:

function git() { 
     case $* in 
         "checkout master" ) command echo "This is a dev server, do not checkout the master branch!" ;; 
         "pull origin master" ) command echo "This is a dev server, do not pull the master branch" ;; 
         * ) command git "$@" ;; 
     esac
}

Or define the desired commands and avoid running any others:

function git(){
     if echo "$@" | grep -Eq '^checkout uat$'; then
          command git $@
     elif echo "$@" | grep -Eq '^pull .+ uat$'; then
          command git $@
     else
          echo "The command $@ needs to be whitelisted before it can be run"
     fi
}

Either approach mitigates the risk of someone incorrectly using the master branch on the development server.

Postfix IPv6 Loopback Failure

I needed to send email messages from a PHP form, and the web server at work uses Postfix. So … I’m getting Postfix set up to relay mail for the first time in a decade or two 🙂 I thought I’d just have to edit /etc/postfix/main.cf and add “relayhost = [something.example.com]”.

Nope. The service fails to start with nothing particularly indicative — just a [FAILED] status from the init script. Attempting to start Postfix outside of the init script is far more informative:

[lisa@564240601ac2 init.d]# /usr/sbin/postfix start
postfix: fatal: parameter inet_interfaces: no local interface found for ::1

Turns out I’ve got to edit /etc/postfix/main.cf and tell it to use IPv4 only:

# Enable IPv4, and IPv6 if supported
inet_protocols = ipv4

 

Certificate Generation Script

I finally put together a script that gathers some basic information (hostname & SAN’s) and creates a certificate signed against my CA. I’ve got a base myssl.cnf file that ends with

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]

The script appends all of the alternate names to the myssl.cnf file.

#!/bin/bash

RED_DARK='\033[38;5;196m'
GREEN_DARK='\033[38;5;35m'
BLUE_DARK='\033[38;5;57m'
NC='\033[0m' # Reset

function getInput {
        echo -e "${BLUE_DARK}Please input the short hostname you wish to use (e.g. server123):${NC}"
        read HOST

        echo -e "${BLUE_DARK}Please input the domain name you wish to use with this hostname (e.g. rushworth.us):${NC}"
        read DOMAIN

        echo -e "${GREEN_DARK}Please enter any SAN values for this certificate, separated by spaces (must be fully qualified):${NC}"
        read SANS

        FQHOST="${HOST}.${DOMAIN}"

        echo -e "Short hostname: $HOST"
        echo -e "Fully qualified hostname: $FQHOST"
        echo -e "SAN: $SANS"

        echo -e "${RED_DARK}Is this correct? (Y/N):${NC}"
        read boolCorrect

        if [ $boolCorrect == 'Y' ] || [ $boolCorrect == 'y' ]
        then
                mkdir $HOST
                echo $HOST
                cp myssl.cnf "./$HOST/myssl.cnf"

                cd "./$HOST"

                echo "The following SANs will be used on this certificate: "
                echo "DNS.1 = ${FQHOST}"
                echo "DNS.1 = ${FQHOST}" >> ./myssl.cnf
                echo "DNS.2 = ${HOST}"
                echo "DNS.2 = ${HOST}" >> ./myssl.cnf

                if [ -n "$SANS" ]
                then
                        SANARRAY=( $SANS )
                        iSANCounter=2
                        for SANITEM in "${SANARRAY[@]}" ; do
                                let iSANCounter=iSANCounter+1
                                echo "DNS.${iSANCounter} = ${SANITEM}"
                                echo "DNS.${iSANCounter} = ${SANITEM}" >> ./myssl.cnf
                        done
                fi
                export strCertKeyPassword=Wh1t2v2rP144w9rd
                export strPFXPassword=123abc456
                openssl genrsa -passout env:strCertKeyPassword -aes256 -out $FQHOST.passwd.key 2048
                openssl req -new -key $FQHOST.passwd.key -passin env:strCertKeyPassword -config ./myssl.cnf -reqexts req_ext -out $FQHOST.csr -subj "/C=US/ST=Ohio/L=Cleveland/O=Rushworth/OU=Home/CN=$FQHOST"
                openssl x509 -req -in $FQHOST.csr -passin env:strCertKeyPassword -extensions req_ext -extfile ./myssl.cnf -out $FQHOST.cer -days 365 -CA /ca/ca.cer -CAkey /ca/ca.key -sha256
                openssl rsa -in $FQHOST.passwd.key -out $FQHOST.key -passin pass:$strCertKeyPassword -passin env:strCertKeyPassword
                openssl pkcs12 -export -out $FQHOST.pfx -inkey $FQHOST.key -in $FQHOST.cer -passout env:strPFXPassword

        else
                getInput
        fi
}

getInput

There’s an encrypted private key and a non-encrypted private key. Because I have some Windows servers — Exchange and Active Directory — I create a PFX file too.

 

GitLab – Using the Docker Executor for Testing

Setting up gitlab-runner to use a Docker executor: You need Docker running on the gitlab-runner host. In my sandbox, I have GitLab running as a Docker container. Instead of installing Docker in Docker, I have mounted the host Docker socket to the GitLab container. You’ll need to add the –privileged flag, and since I’m using Windows … my mount path is funky. But it works.

docker run –detach –hostname gitlab.rushworth.us –publish 443:443 –publish 80:80 –publish 22:22 –name gitlab -v //var/run/docker.sock:/var/run/docker.sock –privileged gitlab/gitlab-ee:latest

Register the runner using “docker-runner register”. I always specify the image in my CI YAML file, so the default image is immaterial … but I’ve encountered groups with an image that mirrors the production servers who set that image as the default.

Edit /etc/gitlab-runner/config.toml and change “privileged = false” to true.

Start the runner (docker-runner start). In the GitLab Admin Area, navigate to Overview => Runners and select the one we just created. When a project is updated, tags can be used to select an appropriate runner. Because most of my testing is done with the shell executor, the runner which uses the shell executor has no tags and the runner which uses the Docker executor is tagged with “runner-docker”. You can require all jobs include a tag to select the appropriate runner (which avoids someone accidentally forgetting a tag and having their project processed through the wrong runner).

An image – you’ll need an image. You can use base images from the Docker Hub registry or create your own image. You can add components in the before_script or use a Dockerfile to build an image from the parent image.

Now we’re ready to use the Docker executor! Create your CI YAML file.

If you are not using the default image, start with “image: <the image you want>”.

We don’t want phpunit in the running image, but I use it for testing. Thus, I need a before_script component to install the phpunit package.

If you’ve used a tag to restrict what is run in your Docker-executor based runner, add the appropriate tag. Include the tester command line.

.gitlab.yml:

image: gitlab.rushworth.us:4567/lisa/ljtestproject-dockerexecutor
stages:
- test

before_script:
# Install dependencies
- bash ci/docker_InstallReqs.sh

test_job:
stage: test
tags:
- runner-docker
script:
- phpunit --configuration phpunit_myapp.xml

Docker_InstallReqs.sh

#!/bin/bash
yum install php-phpunit-PHPUnit

Now when you commit changes to the repository, the Docker-executor based runner will be used for the CI/CD pipeline. A transient Docker container will be created with the image, your before_script will be executed, and then the test script will be run within the container.

 

GitLab – Using the built-in Docker Registry

GitLab has a built-in Docker registry that you can use for projects. With the Omnibus install (or a container based on the official Docker image), enabling the registry is as simple as adding a config line to your gitlab.rb (this assumes you have a SSL key at /etc/gitlab/ssl named with the fully qualified hostname and using .crt for the public key and .key for the private key

registry_external_url ‘https://gitlab.example.com:4567’

Then just tag an image to a project’s repository URL

docker tag ossautomation/cent68php56 gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor

Log in and push the image:

D:\git\ljtestproject-dockerexecutor>docker login gitlab.example.com:4567
Username: lisa
Password:
Login Succeeded

D:\git\ljtestproject-dockerexecutor>docker push gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor
The push refers to repository [gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor]
45c3e2f5d139: Pushing [=> ] 33.31MB/1.619GB

GitLab SSH Deployment Setup

Preliminary stuff – before setting up SSH deployment in your pipeline, you’ll need a user on the target box with permission to write to the files being published. You will need a public/private key pair.

On the target server, the project needs to be cloned into the deployment directory. The public key will need to be added to authorized_keys (or authorized_keys2 on older versions of Linux) file so the private key can be used for authentication.

To set up your GitLab project for SSH-based deployment, you need to add some variables to the project. In the project, navigate to Settings ==> CI/CD

Expand the “Variables” section. You will need to add the following key/value variable pairs:

Key Value
SSH_KNOWN_HOSTS Output of ssh-keyscan targetserver.example.com
SSH_PRIVATE_KEY Content of your private key
DEPLOYMENT_HOST Target hostname, e.g. targetserver.example.com
DEPLOYMENT_USER Username on target server
DEPLOYMENT_PATH Path to which project will be deployed on target server

Save the variables

I am managing both a production and development deployment within the pipeline, so I’ve got prod and dev specific variables. We use the same username for prod and dev; but the hostname, path, and target server public key are different.

If your repository is publicly readable, this is sufficient. If you have a private repository, you’ll need a way to authenticate and fetch the data. In this example, I am using a deployment token. Under Settings Repository, expand the “Deployment Tokens” section and create a deployment token. On my target servers, the remote is added as https://TokenUser:TokenSecret@gitlab.example.com/path/to/project.git instead of just https://gitlab.example.com/path/to/project.git

Once you have defined these variables within the project, use the variables in your CI/CD YAML. In this example, I am deploying PHP code to a web server. Changes to the development branch are deployed to the dev server, and changes to the master branch are deployed to the production server.

In the before_script, I set up the key-based authentication by adding the private key to my runner environment and adding the prod and dev target server’s public key to the runner environment.

- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS_DEV" > ~/.ssh/known_hosts
- echo "$SSH_KNOWN_HOSTS_PROD" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts

In the deployment component, username and host variables are used to connect to the target server via SSH. The commands run over that SSH session change directory into the deployment target path and use “git pull” to fetch and merge the updated code. This ensures the proper branch is pulled to the production and down-level environments.

production-deployment:
 stage: deploy
  script:
    - ssh $DEPLOYMENT_USER@$DEPLOYMENT_HOST_PROD "cd '$DEPLOYMENT_PATH_PROD'; git pull origin master"
  only:
    - master

development-deployment:
 stage: deploy
 script:
   - ssh $DEPLOYMENT_USER@$DEPLOYMENT_HOST_DEV "cd '$DEPLOYMENT_PATH_DEV'; git pull origin development"
 only:
   - development

Now when I make changes to the project code,

Assuming the tests still pass, the deployment will run

If you click on the deployment component, you can see what changes were pulled to the target server

And, yes, the updated files are on my target server.

 

Create a New Exchange Account with Powershell

We use unique e-mail addresses every time we give out our address. Because that occasionally means I need to reply to someone using that address, I set up a quick web form to send e-mail from any arbitrary address in my domain. Every now and again, though … we need to actually reply to a message (attach pictures, get some really specific formatting without wasting a lot of time coming up with the appropriate HTML, etc). That means making a new Exchange account and granting full access to the person who wants to send mail from that address.

New-Mailbox -Name "NewAccount" -Alias NewAccount -LastName "NewAccount" -UserPrincipalName NewAccount@example.com -Password (ConvertTo-SecureString -String 'haW29oihOI#192QHe983QHR9' -AsPlainText -Force)
Add-MailboxPermission -Identity "NewAccount@example.com" -User "me@example.com" -AccessRights FullAccess -InheritanceType All

Or adding a new primary SMTP address to an existing account:

set-mailbox -Identity NewAccount@example.com -PrimarySmtpAddress "ANewAddress@example.com" -EmailAddressPolicyEnabled $false

When adding a new address, the existing primary SMTP address becomes a secondary proxy address.

SSL Trust On Fedora

I have a CA on one of our Fedora boxes, and I use it to sign some of the internal certificates. I’ll probably stop doing that since the LetsEncrypt certs are free … but, for now, I’ve still got to set up a trust to my CA.

In /etc/pki/ca-trust/source/anchors, put a PEM file with the CA public key. Run update-ca-trust … the cert gets added to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and your system will trust the CA.

Using File System As A Git Repository

I’ve used GitLab for quite some time, and as a full featured CI/CD platform that also provides git functionality … it’s awesome. But it’s also serious overkill for someone who wants to coordinate code with another developer or two. Or just keep history for their code. Or a backup. To accomplish this, all you need is drive space. If you’ve got a file server, any folder on the share can be a git repository.

On the server, create a git repository and add something to it.

Z:\Temp\test>git init
Initialized empty Git repository in Z:/Temp/test/.git/

Z:\Temp\test>notepad testfile.txt

Z:\Temp\test>git add testfile.txt

Z:\Temp\test>git commit -m "Initial file upload"
[master (root-commit) 9a3ebe7] Initial file upload
1 file changed, 1 insertion(+)
create mode 100644 testfile.txt

Then on your client, either clone the repo from the drive path

C:\Users\lisa>git clone file://z:/temp/test
Cloning into 'test'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.

Or from the UNC path

C:\Users\lisa>git clone file://server01.example.com/data/temp/test
Cloning into 'test'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.

I prefer to use the UNC paths – if my drive letter fails to map for some reason, the UNC path is still available.

If you’ve got pre-existing code, there’s a bit of a different process. On the server, create an empty folder and use “git init” to initialize the empty repo. On the client where the code exists, run:

git init
git add *
git commit -m “Initial code upload”
git remote add origin git clone file://server01.example.com/data/temp/test
git push origin master

 

Finding Disabled Accounts In Active Directory

When using Active Directory (AD) as a source of user data, it’s useful to filter out disabled accounts. Unfortunately, AD has a lot of different security-related settings glomed together in the userAccountControl attribute. Which means there’s no single attribute/value combination you can use to ignore disabled accounts.

The decimal value you see for userAccountControl isn’t terribly useful, but display it in binary and each bit position has a meaning. The userAccountControl value is just the number with a bunch of bits set. Numbering the bits from left to right, here is what each one means.

Bit # Meaning
0 Unused – must be 0
1 Unused – must be 0
2 Unused – must be 0
3 Unused – must be 0
4 Unused – must be 0
5 ADS_UF_PARTIAL_SECRETS_ACCOUNT
6 ADS_UF_NO_AUTH_DATA_REQUIRED
7 ADS_UF_TRUSTED_TO_AUTHENTICATE_FOR_DELEGATION
8 ADS_UF_PASSWORD_EXPIRED
9 ADS_UF_DONT_REQUIRE_PREAUTH
10 ADS_UF_USE_DES_KEY_ONLY
11 ADS_UF_NOT_DELEGATED
12 ADS_UF_TRUSTED_FOR_DELEGATION
13 ADS_UF_SMARTCARD_REQUIRED
14 Unused – must be 0
15 ADS_UF_DONT_EXPIRE_PASSWD
16 Unused – must be 0
17 Unused – must be 0
18 ADS_UF_SERVER_TRUST_ACCOUNT
19 ADS_UF_WORKSTATION_TRUST_ACCOUNT
20 ADS_UF_INTERDOMAIN_TRUST_ACCOUNT
21 Unused – must be 0
22 ADS_UF_NORMAL_ACCOUNT
23 Unused – must be 0
24 ADS_UF_ENCRYPTED_TEXT_PASSWORD_ALLOWED
25 ADS_UF_PASSWD_CANT_CHANGE
26 ADS_UF_PASSWD_NOTREQD
27 ADS_UF_LOCKOUT
28 ADS_UF_HOMEDIR_REQUIRED
29 Unused – must be 0
30 ADS_UF_ACCOUNT_DISABLE
31 Unused – must be 0

Bit #30 indicates if the account is disabled — 1 if the account is disabled, 0 if the account is enabled. Simple and direct approach is to “and” the attribute value with 0b10 to extract just the bit we care about. When the and operation returns 0, the account is enabled. When it returns 2 (0x10), the account is disabled.