Category: System Administration

GitLab – Using the Docker Executor for Testing

Setting up gitlab-runner to use a Docker executor: You need Docker running on the gitlab-runner host. In my sandbox, I have GitLab running as a Docker container. Instead of installing Docker in Docker, I have mounted the host Docker socket to the GitLab container. You’ll need to add the –privileged flag, and since I’m using Windows … my mount path is funky. But it works.

docker run –detach –hostname gitlab.rushworth.us –publish 443:443 –publish 80:80 –publish 22:22 –name gitlab -v //var/run/docker.sock:/var/run/docker.sock –privileged gitlab/gitlab-ee:latest

Register the runner using “docker-runner register”. I always specify the image in my CI YAML file, so the default image is immaterial … but I’ve encountered groups with an image that mirrors the production servers who set that image as the default.

Edit /etc/gitlab-runner/config.toml and change “privileged = false” to true.

Start the runner (docker-runner start). In the GitLab Admin Area, navigate to Overview => Runners and select the one we just created. When a project is updated, tags can be used to select an appropriate runner. Because most of my testing is done with the shell executor, the runner which uses the shell executor has no tags and the runner which uses the Docker executor is tagged with “runner-docker”. You can require all jobs include a tag to select the appropriate runner (which avoids someone accidentally forgetting a tag and having their project processed through the wrong runner).

An image – you’ll need an image. You can use base images from the Docker Hub registry or create your own image. You can add components in the before_script or use a Dockerfile to build an image from the parent image.

Now we’re ready to use the Docker executor! Create your CI YAML file.

If you are not using the default image, start with “image: <the image you want>”.

We don’t want phpunit in the running image, but I use it for testing. Thus, I need a before_script component to install the phpunit package.

If you’ve used a tag to restrict what is run in your Docker-executor based runner, add the appropriate tag. Include the tester command line.

.gitlab.yml:

image: gitlab.rushworth.us:4567/lisa/ljtestproject-dockerexecutor
stages:
- test

before_script:
# Install dependencies
- bash ci/docker_InstallReqs.sh

test_job:
stage: test
tags:
- runner-docker
script:
- phpunit --configuration phpunit_myapp.xml

Docker_InstallReqs.sh

#!/bin/bash
yum install php-phpunit-PHPUnit

Now when you commit changes to the repository, the Docker-executor based runner will be used for the CI/CD pipeline. A transient Docker container will be created with the image, your before_script will be executed, and then the test script will be run within the container.

 

GitLab – Using the built-in Docker Registry

GitLab has a built-in Docker registry that you can use for projects. With the Omnibus install (or a container based on the official Docker image), enabling the registry is as simple as adding a config line to your gitlab.rb (this assumes you have a SSL key at /etc/gitlab/ssl named with the fully qualified hostname and using .crt for the public key and .key for the private key

registry_external_url ‘https://gitlab.example.com:4567’

Then just tag an image to a project’s repository URL

docker tag ossautomation/cent68php56 gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor

Log in and push the image:

D:\git\ljtestproject-dockerexecutor>docker login gitlab.example.com:4567
Username: lisa
Password:
Login Succeeded

D:\git\ljtestproject-dockerexecutor>docker push gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor
The push refers to repository [gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor]
45c3e2f5d139: Pushing [=> ] 33.31MB/1.619GB

GitLab SSH Deployment Setup

Preliminary stuff – before setting up SSH deployment in your pipeline, you’ll need a user on the target box with permission to write to the files being published. You will need a public/private key pair.

On the target server, the project needs to be cloned into the deployment directory. The public key will need to be added to authorized_keys (or authorized_keys2 on older versions of Linux) file so the private key can be used for authentication.

To set up your GitLab project for SSH-based deployment, you need to add some variables to the project. In the project, navigate to Settings ==> CI/CD

Expand the “Variables” section. You will need to add the following key/value variable pairs:

Key Value
SSH_KNOWN_HOSTS Output of ssh-keyscan targetserver.example.com
SSH_PRIVATE_KEY Content of your private key
DEPLOYMENT_HOST Target hostname, e.g. targetserver.example.com
DEPLOYMENT_USER Username on target server
DEPLOYMENT_PATH Path to which project will be deployed on target server

Save the variables

I am managing both a production and development deployment within the pipeline, so I’ve got prod and dev specific variables. We use the same username for prod and dev; but the hostname, path, and target server public key are different.

If your repository is publicly readable, this is sufficient. If you have a private repository, you’ll need a way to authenticate and fetch the data. In this example, I am using a deployment token. Under Settings Repository, expand the “Deployment Tokens” section and create a deployment token. On my target servers, the remote is added as https://TokenUser:TokenSecret@gitlab.example.com/path/to/project.git instead of just https://gitlab.example.com/path/to/project.git

Once you have defined these variables within the project, use the variables in your CI/CD YAML. In this example, I am deploying PHP code to a web server. Changes to the development branch are deployed to the dev server, and changes to the master branch are deployed to the production server.

In the before_script, I set up the key-based authentication by adding the private key to my runner environment and adding the prod and dev target server’s public key to the runner environment.

- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS_DEV" > ~/.ssh/known_hosts
- echo "$SSH_KNOWN_HOSTS_PROD" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts

In the deployment component, username and host variables are used to connect to the target server via SSH. The commands run over that SSH session change directory into the deployment target path and use “git pull” to fetch and merge the updated code. This ensures the proper branch is pulled to the production and down-level environments.

production-deployment:
 stage: deploy
  script:
    - ssh $DEPLOYMENT_USER@$DEPLOYMENT_HOST_PROD "cd '$DEPLOYMENT_PATH_PROD'; git pull origin master"
  only:
    - master

development-deployment:
 stage: deploy
 script:
   - ssh $DEPLOYMENT_USER@$DEPLOYMENT_HOST_DEV "cd '$DEPLOYMENT_PATH_DEV'; git pull origin development"
 only:
   - development

Now when I make changes to the project code,

Assuming the tests still pass, the deployment will run

If you click on the deployment component, you can see what changes were pulled to the target server

And, yes, the updated files are on my target server.

 

SSL Trust On Fedora

I have a CA on one of our Fedora boxes, and I use it to sign some of the internal certificates. I’ll probably stop doing that since the LetsEncrypt certs are free … but, for now, I’ve still got to set up a trust to my CA.

In /etc/pki/ca-trust/source/anchors, put a PEM file with the CA public key. Run update-ca-trust … the cert gets added to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and your system will trust the CA.

Using File System As A Git Repository

I’ve used GitLab for quite some time, and as a full featured CI/CD platform that also provides git functionality … it’s awesome. But it’s also serious overkill for someone who wants to coordinate code with another developer or two. Or just keep history for their code. Or a backup. To accomplish this, all you need is drive space. If you’ve got a file server, any folder on the share can be a git repository.

On the server, create a git repository and add something to it.

Z:\Temp\test>git init
Initialized empty Git repository in Z:/Temp/test/.git/

Z:\Temp\test>notepad testfile.txt

Z:\Temp\test>git add testfile.txt

Z:\Temp\test>git commit -m "Initial file upload"
[master (root-commit) 9a3ebe7] Initial file upload
1 file changed, 1 insertion(+)
create mode 100644 testfile.txt

Then on your client, either clone the repo from the drive path

C:\Users\lisa>git clone file://z:/temp/test
Cloning into 'test'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.

Or from the UNC path

C:\Users\lisa>git clone file://server01.example.com/data/temp/test
Cloning into 'test'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.

I prefer to use the UNC paths – if my drive letter fails to map for some reason, the UNC path is still available.

If you’ve got pre-existing code, there’s a bit of a different process. On the server, create an empty folder and use “git init” to initialize the empty repo. On the client where the code exists, run:

git init
git add *
git commit -m “Initial code upload”
git remote add origin git clone file://server01.example.com/data/temp/test
git push origin master

 

Finding Disabled Accounts In Active Directory

When using Active Directory (AD) as a source of user data, it’s useful to filter out disabled accounts. Unfortunately, AD has a lot of different security-related settings glomed together in the userAccountControl attribute. Which means there’s no single attribute/value combination you can use to ignore disabled accounts.

The decimal value you see for userAccountControl isn’t terribly useful, but display it in binary and each bit position has a meaning. The userAccountControl value is just the number with a bunch of bits set. Numbering the bits from left to right, here is what each one means.

Bit # Meaning
0 Unused – must be 0
1 Unused – must be 0
2 Unused – must be 0
3 Unused – must be 0
4 Unused – must be 0
5 ADS_UF_PARTIAL_SECRETS_ACCOUNT
6 ADS_UF_NO_AUTH_DATA_REQUIRED
7 ADS_UF_TRUSTED_TO_AUTHENTICATE_FOR_DELEGATION
8 ADS_UF_PASSWORD_EXPIRED
9 ADS_UF_DONT_REQUIRE_PREAUTH
10 ADS_UF_USE_DES_KEY_ONLY
11 ADS_UF_NOT_DELEGATED
12 ADS_UF_TRUSTED_FOR_DELEGATION
13 ADS_UF_SMARTCARD_REQUIRED
14 Unused – must be 0
15 ADS_UF_DONT_EXPIRE_PASSWD
16 Unused – must be 0
17 Unused – must be 0
18 ADS_UF_SERVER_TRUST_ACCOUNT
19 ADS_UF_WORKSTATION_TRUST_ACCOUNT
20 ADS_UF_INTERDOMAIN_TRUST_ACCOUNT
21 Unused – must be 0
22 ADS_UF_NORMAL_ACCOUNT
23 Unused – must be 0
24 ADS_UF_ENCRYPTED_TEXT_PASSWORD_ALLOWED
25 ADS_UF_PASSWD_CANT_CHANGE
26 ADS_UF_PASSWD_NOTREQD
27 ADS_UF_LOCKOUT
28 ADS_UF_HOMEDIR_REQUIRED
29 Unused – must be 0
30 ADS_UF_ACCOUNT_DISABLE
31 Unused – must be 0

Bit #30 indicates if the account is disabled — 1 if the account is disabled, 0 if the account is enabled. Simple and direct approach is to “and” the attribute value with 0b10 to extract just the bit we care about. When the and operation returns 0, the account is enabled. When it returns 2 (0x10), the account is disabled.

 

Upgrading Oracle SQL Developer

I’ve been using an Oracle database more in my new position … which means I’ve got the Oracle SQL Developer tool installed on my computer. My first upgrade was available yesterday … and it didn’t work. Not like threw an error, but like double click on the executable and nothing happens. It silently exits.

Turns out there’s something in appdata that needs to be cleared. I don’t run multiple versions of SQL Developer, so I could just blow away “%userprofile%\appdata\roaming\SQL Developer” and “%userprofile%\appdata\roaming\sqldeveloper” to clear whatever needs to be cleared. Click the icon and the program finally runs.

Setting up gitlab-runner

CLI on the GitLab server:

# Set up the GitLab Repo
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh

# Install package
yum install gitlab-runner

# verify runner is executable
ll `which gitlab-runner`
# If needed, flag it executable – shouldn’t be a problem with RPM installations, but it’s been a problem for me with manual installs
#chmod ugo+x /usr/local/bin/gitlab-runner

# Register a runner
gitlab-runner register

# use URL & token from http://<GITLABSERVER>/admin/runners
# add tags, if you want to use tags to assign runner
# executor: shell (and docker, if you want to use docker executors. The shell executor is the simplest case, so I am starting there)

# start the runner
gitlab-runner start

On the GitLab Web GUI:

Admin section => Overview => Runners. Click pencil to edit the runner and uncheck “Lock to current projects”, and (unless you want to use tagging) check “Run untagged jobs”

** I was getting an error in every pipeline job saying the git command was not found.

For most other commands, you can append to the path in the before_script section of your .gitlab-ci.yml

before_script:
– export PATH=$PATH:/opt/whatever/odd/path/bin

But that doesn’t work in this case because we’re not getting that far: the bootstrap “stuff” cannot fetch the project to see the before script. Git, on my system, was part of the GitLab package. I just created a symlink into a “normal” binary location:

root@gitlab:~# which git
/opt/gitlab/embedded/bin/git
root@gitlab:~# ln -s /opt/gitlab/embedded/bin/git /usr/bin/git

And we’ve got successful test execution:

 

Software Flow Control and vim

In the early 90’s, one of the things I liked about Microsoft’s ecosystem was that they developed a standard for keyboard shortcuts. In most applications, developed by Microsoft or not, you could hit ctrl-p to print or ctrl-x to exit. Or ctrl-s to save. It’s quite convenient when I’m using Windows applications, but hitting ctrl-s to save without really thinking about it hangs vim. Hangs like “go into another shell and kill vim & that ssh session”. This is because ctrl-s, in Linux, means XOFF — the software flow control command that means “hi, I’m a thing from 1968 and my buffer is getting full. chill out for a bit and let me catch up”. Recovery is simple enough, send XON — “hi, that thing from 1968 again, and I’m all caught up. send me some more stuff”. That’s ctrl-q.

But there aren’t many slow anything‘s involved in computing these days, which means XON/XOFF isn’t the most useful of features for most people (* if you’ve got real serial devices attached … you may not be “most people” here). Instead of remembering ctrl-q gets gets vim back without killing it, just disable START/STOP control. Thing is it’s not really vim that’s using flow control — it’s the terminal emulator — so the “fix” isn’t something you’ll have to do in vim. In your ~/.bashrc or ~/.bash_profile (or globally in something like /etc/profile.d/disableFlowControl.sh)

# Disable XON/XOFF flow control so ctrl-s doesn’t hang vim
stty -ixon

If you can add -ixoff to avoid ctrl-q from meaning XON too … but I don’t bother since “start sending me data” doesn’t seem to hang anything.

Generating a keytab file without domain admin permissions

Most of the application owners I encountered wanted someone online with them when they had to change their Kerberos service principal password. Not because I really needed to generate the keytab file, but “just in case”. A warm fuzzy feeling, good thoughts being sent their way. Whatever. I was up at dark-o-clock, so I’d generate the keytab the right way and we’d all be asleep in twenty minutes. What’s the wrong way? Well, in a stand-alone AD … that’s really just mapping the UPN to the wrong thing or failing to chose the encryption type wisely. But with AD accounts managed by an identify management platform and a notification package registered on the DCs to update said identity management platform when passwords were changed? I joined a lot of emergency calls either at 7AM following their keytab update or half an hour after the change completed. And 7AM was only because the app didn’t happen to have any 3rd shift users.

Keytab files have a key version number (kvno). Generate keytab and set the account password, you’ve got a file with KVNO version 5. Except IDM picks up the password change, tweaks up the managed accounts, and the actual AD object msDS-KeyVersionNumber is 6. And auth on your site falls over about half an hour after you complete your change (replication time!). So what’s the right way? Don’t make changes to the account. If you’re changing the password, change the password. And then generate a keytab.

 

I’ve created a sample account, ljrtest, used setspn to set an SPN value for my lisa.sandbox.rushworth.us site, and configured the account to support AES 128 and 256 bit encryption.

To generate a keytab file without updating the UPN or attempting to set the account password, use:

ktpass /out ljrtest.keytab /princ HTTP/lisa.sandbox.rushworth.us@rushworth.us -SetUPN /mapuser ljrtest /crypto AES256-SHA1 /ptype KRB5_NT_PRINCIPAL /pass DevNull -SetPass /target dc.rushworth.us

KTPASS is part of the RSAT utilities — on Win10 with the Oct 2018 update (or newer), this is now a “Feature on Demand” and can be added  through “Apps & Features” by clicking “optional features” and selecting the ADS RSAT pack.

There are a few other utilities available — ktab from the JDK or ktutil on Linux — if you cannot install the RSAT pack.