Month: September 2019

Building a Docker Image From a Parent Image

Docker maintains a registry of pre-built images that may be all you need. Put some thought into it, though – don’t just trust any image you find on the registry. When a pre-built image doesn’t meet your needs, you can make your own image based on any base image.

I create a new folder to hold the build instructions, additional configuration files, and notes about the build process. In that folder, create a file named “Dockerfile” which controls the image build.

You’ll need to specify the base image for your build using the FROM directive. I have examples here for installing sqlite3 on both a CentOS and Ubuntu base image. Add LABELs indicating the purpose of the image and who maintains it. Use the “RUN” directives to provide instructions to modify the base image to your needs – install what you want, create folders or files, clean up anything you don’t want.

The ENTRYPOINT is what runs when the container starts – it can be an executable, as in this case, or a script file.

If you need to expose ports for inter-container communication, add a list of ports to expose. SQLite is self-contained, but a microservice environment may have “EXPOSE 21443” to allow other containers to communicate with it on port 21443. Note this is different than binding the container port to a host – my Apache web server container has 443 bound; but that’s done in the “docker run” line where the container is built, not in the Dockerfile for the image.

Example – SQLite3 on CentOS

FROM centos:latest

LABEL "version": "latest"
LABEL "description": "CentOS server running SQLite3"
LABEL "maintainer": "DockerImages@lisa.rushworth.us"

RUN yum -y install glibc.i686
RUN yum -y install zlib.i686
RUN yum -y install wget
RUN yum -y install unzip

RUN cd /root && wget https://sqlite.org/2019/sqlite-tools-linux-x86-3290000.zip && unzip -j -d /usr/bin -o /root/sqlite-tools-linux-x86-3290000.zip && rm -f /root/sqlite-tools-linux-x86-3290000.zip

RUN rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN mkdir -p /db

WORKDIR /db

ENTRYPOINT [ "sqlite3" ]

Example – SQLite3 on Ubuntu

FROM ubuntu:latest

LABEL "version": "latest"
LABEL "description": "Ubuntu server running SQLite3"
LABEL "maintainer": "DockerImages@lisa.rushworth.us"

RUN DEBIAN_FRONTEND=noninteractive apt-get -yq update

RUN DEBIAN_FRONTEND=noninteractive apt-get -yq install sqlite3

RUN rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN mkdir -p /db

WORKDIR /db

ENTRYPOINT [ "sqlite3" ]

Save your Dockerfile and build your container using “docker build -t image/name .”

Use “docker images” to confirm the image has been successfully created.

And start a container using your image:

docker run -it --name ljrimagetest ljr/sqllite3

If you are having problems starting your container, change the entrypoint to something like “/bin/bash” – this will drop you to the host’s command line instead of the application. From there, you can troubleshoot your launch problems and sort the problem. The CentOS sqlite install, as an example, required glib and zlib components to run. Rather than trying something, rebuilding the image, launching a container, and looping until sqlite3 launched … I used bash as my entrypoint, installed packages until the application ran, and then modified the Dockerfile and rebuilt the image.

At this point, you can tag the image and upload it to a registry. I use my GitLab server to store Docker images. There is a DockerImages repository

To tag the image, use

docker tag ljr/sqllite3 gitlab.rushworth.us:4567/lisa/dockerimages/sqllite3:latest

And upload the image to your registry (you may need to authenticate first)

docker push gitlab.rushworth.us:4567/lisa/dockerimages/sqllite3:latest

If you are using GitLab as your registry, navigate to the repository. Select Packages => Container Registry

Here’s my Sqlite3 image

 

Mutually Assured Destruction

During the arms race — NATO/WARSAW tension, there was a concept of mutually assured destruction. You may have lots of nuclear warheads, but so do I. So no one can use any of them because we’ll wipe out the entire planet. I’ve been thinking about that in terms of climate change … you chose to do something, you’re going to destroy us all. But … there’s no single collective ‘you’ and no single collective action like, say, the Soviet or American government deciding to launch. And, in the nuclear assured destruction case, the destruction was well understood and immediate. We didn’t deal particularly well with the earlier, easier to comprehend, case of mutually assured destruction. We just kept dumping money into arms and then negotiating to retire out-of-date (and sometimes old-to-the-point-of-getting-dangerous) weapons. How in the world do we hope to force mass short-term changes to ward off hypothetical long-term damages.

GitLab – Using the built-in Docker Registry

GitLab has a built-in Docker registry that you can use for projects. With the Omnibus install (or a container based on the official Docker image), enabling the registry is as simple as adding a config line to your gitlab.rb (this assumes you have a SSL key at /etc/gitlab/ssl named with the fully qualified hostname and using .crt for the public key and .key for the private key

registry_external_url ‘https://gitlab.example.com:4567’

Then just tag an image to a project’s repository URL

docker tag ossautomation/cent68php56 gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor

Log in and push the image:

D:\git\ljtestproject-dockerexecutor>docker login gitlab.example.com:4567
Username: lisa
Password:
Login Succeeded

D:\git\ljtestproject-dockerexecutor>docker push gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor
The push refers to repository [gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor]
45c3e2f5d139: Pushing [=> ] 33.31MB/1.619GB

GitLab SSH Deployment Setup

Preliminary stuff – before setting up SSH deployment in your pipeline, you’ll need a user on the target box with permission to write to the files being published. You will need a public/private key pair.

On the target server, the project needs to be cloned into the deployment directory. The public key will need to be added to authorized_keys (or authorized_keys2 on older versions of Linux) file so the private key can be used for authentication.

To set up your GitLab project for SSH-based deployment, you need to add some variables to the project. In the project, navigate to Settings ==> CI/CD

Expand the “Variables” section. You will need to add the following key/value variable pairs:

Key Value
SSH_KNOWN_HOSTS Output of ssh-keyscan targetserver.example.com
SSH_PRIVATE_KEY Content of your private key
DEPLOYMENT_HOST Target hostname, e.g. targetserver.example.com
DEPLOYMENT_USER Username on target server
DEPLOYMENT_PATH Path to which project will be deployed on target server

Save the variables

I am managing both a production and development deployment within the pipeline, so I’ve got prod and dev specific variables. We use the same username for prod and dev; but the hostname, path, and target server public key are different.

If your repository is publicly readable, this is sufficient. If you have a private repository, you’ll need a way to authenticate and fetch the data. In this example, I am using a deployment token. Under Settings Repository, expand the “Deployment Tokens” section and create a deployment token. On my target servers, the remote is added as https://TokenUser:TokenSecret@gitlab.example.com/path/to/project.git instead of just https://gitlab.example.com/path/to/project.git

Once you have defined these variables within the project, use the variables in your CI/CD YAML. In this example, I am deploying PHP code to a web server. Changes to the development branch are deployed to the dev server, and changes to the master branch are deployed to the production server.

In the before_script, I set up the key-based authentication by adding the private key to my runner environment and adding the prod and dev target server’s public key to the runner environment.

- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS_DEV" > ~/.ssh/known_hosts
- echo "$SSH_KNOWN_HOSTS_PROD" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts

In the deployment component, username and host variables are used to connect to the target server via SSH. The commands run over that SSH session change directory into the deployment target path and use “git pull” to fetch and merge the updated code. This ensures the proper branch is pulled to the production and down-level environments.

production-deployment:
 stage: deploy
  script:
    - ssh $DEPLOYMENT_USER@$DEPLOYMENT_HOST_PROD "cd '$DEPLOYMENT_PATH_PROD'; git pull origin master"
  only:
    - master

development-deployment:
 stage: deploy
 script:
   - ssh $DEPLOYMENT_USER@$DEPLOYMENT_HOST_DEV "cd '$DEPLOYMENT_PATH_DEV'; git pull origin development"
 only:
   - development

Now when I make changes to the project code,

Assuming the tests still pass, the deployment will run

If you click on the deployment component, you can see what changes were pulled to the target server

And, yes, the updated files are on my target server.

 

VSCode Tab Key Not Working

Tab suddenly stop tabbing in VSCode? Try hitting ctrl+m — evidently there’s another ‘mode’ for the tab key where it changes focus instead of tabbing. Very cool and useful when used deliberately. Very “huh?!?!” when you accidentally hit ctrl+m 🙂

From https://code.visualstudio.com/docs/getstarted/keybindings

Ctrl+M Toggle Use of Tab Key for Setting Focus editor.action.toggleTabFocusMode

Of course if that wasn’t your problem … focus mode is turned on now & you’ll want to hit ctrl+m again to change back to tab mode!

Corrupted Spreadsheets From PHPSpreadsheet (andPHPExcel)

I need to deliver Excel files to the browser, so used php://output as the save location. Does exactly what I want except …

Excel says it has a problem with some of the file content. It’s recoverable – click “Yes” and you’ll see all of the spreadsheet data. But no one is going to want to run a repair on every single file they download from my site!

I confirmed the buffer was being cleared, that I didn’t have any extraneous PHP errors getting inserted into the spreadsheet data. My output was clean – it was also corrupt. I’d actually started using the old PHPExcel module, installed and changed over to PHPSpreadsheet because I know PHPExcel is not maintained. But the problem persisted. I started reading through the docs for PHPSpreadsheet to see if I could find a hint.

https://phpspreadsheet.readthedocs.io/en/latest/topics/recipes/#redirect-output-to-a-clients-web-browser

Caution:

Make sure not to include any echo statements or output any other contents than the Excel file. 
There should be no whitespace before the opening <?php tag and at most one line break after the closing ?> tag 
(which can also be omitted to avoid problems). 
Make sure that your script is saved without a BOM (Byte-order mark) because this counts as echoing output. 
The same things apply to all included files. 
Failing to follow the above guidelines may result in corrupt Excel files arriving at the client browser, 
and/or that headers cannot be set by PHP (resulting in warning messages).

Do I have more than one newline after the closing “?>” tag? Sure do!

Got rid of the extra newline, and the downloaded file is fine.

 

Create a New Exchange Account with Powershell

We use unique e-mail addresses every time we give out our address. Because that occasionally means I need to reply to someone using that address, I set up a quick web form to send e-mail from any arbitrary address in my domain. Every now and again, though … we need to actually reply to a message (attach pictures, get some really specific formatting without wasting a lot of time coming up with the appropriate HTML, etc).

If the Exchange powershell snap-in is not loaded (the new-mailbox command is not found), load the snap-in:

Add-PSSnapIn -Name Microsoft.Exchange.Management.PowerShell.SnapIn

Then create a new Exchange account and granting full access to the person who wants to send mail from that address.

New-Mailbox -Name "NewAccount" -Alias NewAccount -LastName "NewAccount" -UserPrincipalName NewAccount@example.com -Password (ConvertTo-SecureString -String 'haW29oihOI#192QHe983QHR9' -AsPlainText -Force)
Add-MailboxPermission -Identity "NewAccount@example.com" -User "me@example.com" -AccessRights FullAccess -InheritanceType All

Or adding a new primary SMTP address to an existing account:

set-mailbox -Identity NewAccount@example.com -PrimarySmtpAddress "ANewAddress@example.com" -EmailAddressPolicyEnabled $false

When adding a new address, the existing primary SMTP address becomes a secondary proxy address.

Oracle Query Returns JSON

I’m using Oracle a lot in my new job; and, in the process, I am learning about a lot of neat Oracle database features. Today, I discovered JSON_OBJECT:

SELECT JSON_OBJECT
(KEY 'CDID" VALUE C1.circuit_design_id,
KEY 'NODE_STATUS' VALUE circuit_position.CIRCUIT_NODE_STATUS, KEY "CDID3' VALUE circuit_position.circuit_design_id_3) jsonCircuitResults
FROM circuit C1
LEFT OUTER JOIN circuit_position on C1.circuit_design_id = circuit_position.circuit_design_id
WHERE C1.circuit_design_id = '54535525' and circuit_position.circuit_design_id_3 is not null;

The response is JSON –

 

SSL Trust On Fedora

I have a CA on one of our Fedora boxes, and I use it to sign some of the internal certificates. I’ll probably stop doing that since the LetsEncrypt certs are free … but, for now, I’ve still got to set up a trust to my CA.

In /etc/pki/ca-trust/source/anchors, put a PEM file with the CA public key. Run update-ca-trust … the cert gets added to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and your system will trust the CA.