GitLab – Using the built-in Docker Registry

GitLab has a built-in Docker registry that you can use for projects. With the Omnibus install (or a container based on the official Docker image), enabling the registry is as simple as adding a config line to your gitlab.rb (this assumes you have a SSL key at /etc/gitlab/ssl named with the fully qualified hostname and using .crt for the public key and .key for the private key

registry_external_url ‘https://gitlab.example.com:4567’

Then just tag an image to a project’s repository URL

docker tag ossautomation/cent68php56 gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor

Log in and push the image:

D:\git\ljtestproject-dockerexecutor>docker login gitlab.example.com:4567
Username: lisa
Password:
Login Succeeded

D:\git\ljtestproject-dockerexecutor>docker push gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor
The push refers to repository [gitlab.example.com:4567/lisa/ljtestproject-dockerexecutor]
45c3e2f5d139: Pushing [=> ] 33.31MB/1.619GB

GitLab SSH Deployment Setup

Preliminary stuff – before setting up SSH deployment in your pipeline, you’ll need a user on the target box with permission to write to the files being published. You will need a public/private key pair.

On the target server, the project needs to be cloned into the deployment directory. The public key will need to be added to authorized_keys (or authorized_keys2 on older versions of Linux) file so the private key can be used for authentication.

To set up your GitLab project for SSH-based deployment, you need to add some variables to the project. In the project, navigate to Settings ==> CI/CD

Expand the “Variables” section. You will need to add the following key/value variable pairs:

Key Value
SSH_KNOWN_HOSTS Output of ssh-keyscan targetserver.example.com
SSH_PRIVATE_KEY Content of your private key
DEPLOYMENT_HOST Target hostname, e.g. targetserver.example.com
DEPLOYMENT_USER Username on target server
DEPLOYMENT_PATH Path to which project will be deployed on target server

Save the variables

I am managing both a production and development deployment within the pipeline, so I’ve got prod and dev specific variables. We use the same username for prod and dev; but the hostname, path, and target server public key are different.

If your repository is publicly readable, this is sufficient. If you have a private repository, you’ll need a way to authenticate and fetch the data. In this example, I am using a deployment token. Under Settings Repository, expand the “Deployment Tokens” section and create a deployment token. On my target servers, the remote is added as https://TokenUser:TokenSecret@gitlab.example.com/path/to/project.git instead of just https://gitlab.example.com/path/to/project.git

Once you have defined these variables within the project, use the variables in your CI/CD YAML. In this example, I am deploying PHP code to a web server. Changes to the development branch are deployed to the dev server, and changes to the master branch are deployed to the production server.

In the before_script, I set up the key-based authentication by adding the private key to my runner environment and adding the prod and dev target server’s public key to the runner environment.

- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS_DEV" > ~/.ssh/known_hosts
- echo "$SSH_KNOWN_HOSTS_PROD" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts

In the deployment component, username and host variables are used to connect to the target server via SSH. The commands run over that SSH session change directory into the deployment target path and use “git pull” to fetch and merge the updated code. This ensures the proper branch is pulled to the production and down-level environments.

production-deployment:
 stage: deploy
  script:
    - ssh $DEPLOYMENT_USER@$DEPLOYMENT_HOST_PROD "cd '$DEPLOYMENT_PATH_PROD'; git pull origin master"
  only:
    - master

development-deployment:
 stage: deploy
 script:
   - ssh $DEPLOYMENT_USER@$DEPLOYMENT_HOST_DEV "cd '$DEPLOYMENT_PATH_DEV'; git pull origin development"
 only:
   - development

Now when I make changes to the project code,

Assuming the tests still pass, the deployment will run

If you click on the deployment component, you can see what changes were pulled to the target server

And, yes, the updated files are on my target server.

 

VSCode Tab Key Not Working

Tab suddenly stop tabbing in VSCode? Try hitting ctrl+m — evidently there’s another ‘mode’ for the tab key where it changes focus instead of tabbing. Very cool and useful when used deliberately. Very “huh?!?!” when you accidentally hit ctrl+m 🙂

From https://code.visualstudio.com/docs/getstarted/keybindings

Ctrl+M Toggle Use of Tab Key for Setting Focus editor.action.toggleTabFocusMode

Of course if that wasn’t your problem … focus mode is turned on now & you’ll want to hit ctrl+m again to change back to tab mode!

Corrupted Spreadsheets From PHPSpreadsheet (andPHPExcel)

I need to deliver Excel files to the browser, so used php://output as the save location. Does exactly what I want except …

Excel says it has a problem with some of the file content. It’s recoverable – click “Yes” and you’ll see all of the spreadsheet data. But no one is going to want to run a repair on every single file they download from my site!

I confirmed the buffer was being cleared, that I didn’t have any extraneous PHP errors getting inserted into the spreadsheet data. My output was clean – it was also corrupt. I’d actually started using the old PHPExcel module, installed and changed over to PHPSpreadsheet because I know PHPExcel is not maintained. But the problem persisted. I started reading through the docs for PHPSpreadsheet to see if I could find a hint.

https://phpspreadsheet.readthedocs.io/en/latest/topics/recipes/#redirect-output-to-a-clients-web-browser

Caution:

Make sure not to include any echo statements or output any other contents than the Excel file. 
There should be no whitespace before the opening <?php tag and at most one line break after the closing ?> tag 
(which can also be omitted to avoid problems). 
Make sure that your script is saved without a BOM (Byte-order mark) because this counts as echoing output. 
The same things apply to all included files. 
Failing to follow the above guidelines may result in corrupt Excel files arriving at the client browser, 
and/or that headers cannot be set by PHP (resulting in warning messages).

Do I have more than one newline after the closing “?>” tag? Sure do!

Got rid of the extra newline, and the downloaded file is fine.

 

Oracle Query Returns JSON

I’m using Oracle a lot in my new job; and, in the process, I am learning about a lot of neat Oracle database features. Today, I discovered json_array. Wrap the selected columns in json_array followed by a name for the result set

SELECT json_array(C1.circuit_design_id,

circuit_position.CIRCUIT_NODE_STATUS, circuit_position.circuit_design_id_3) jsonCircuitResults

FROM circuit C1

LEFT OUTER JOIN circuit_position on C1.circuit_design_id = circuit_position.circuit_design_id

WHERE C1.circuit_design_id = ‘54535525’ and circuit_position.circuit_design_id_3 is not null;

The response is JSON –

 

SSL Trust On Fedora

I have a CA on one of our Fedora boxes, and I use it to sign some of the internal certificates. I’ll probably stop doing that since the LetsEncrypt certs are free … but, for now, I’ve still got to set up a trust to my CA.

In /etc/pki/ca-trust/source/anchors, put a PEM file with the CA public key. Run update-ca-trust … the cert gets added to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and your system will trust the CA.

On the climate

I liked the format of CNN’s climate town hall event. The successful way of allowing candidates to convey a lot of information was basically an accident of the DNC’s stubbornness — they didn’t want to host a debate on climate change (why?!?!), and they own any situation where more than one candidate appears on a stage and talks (or something like that). So the only option available was to provide each candidate a chunk of time — *not* a debate. Sucks that it wasn’t freely available like the debates were, but we signed up for a free trial of YouTube TV to watch it. And we’ll get a free trial of something else for MSNBC’s climate debate later this month (Weld is supposed to be on the schedule).

I’m glad CNN had large time blocks for each candidate — they could have given everyone ten minutes and not had time for follow-up. And it was interesting to see Buttigieg use another approach to the question of why environmentalism is important — because ‘existential threat’ is not likely to convert any minds. But couching it in terms of our stewardship of God’s creations — that actually has potential to appeal to people who don’t care about species going extinct, habitat loss, clean air, clean water, their kids future. What does God think of how you are treating his creations? I cannot imagine that sort of digression coming into a debate format. There were a few forays into a nuclear discussion too — Yang’s liquid fluoride thorium reactor investment, Sanders assertion that we’ve got enough radioactive waste already. I’d have liked to see someone offering to invest in reactors fueled with used rods — not as part of our overall energy strategy, but because *something* needs to be done with the existing waste.

Possibly due to the origin of the event, possibly just demographics … but they had some good questions too. I loved seeing food policy repeatedly brought up as a component of climate change mitigation. And some realization that it’s not as easy as picking up a guy from a Gulf oil rig, sending him to classes for a few weeks, and dropping him off at his job on the solar farm. People get their identities tied up in their job — “what they do” — and that’s just as important as addressing the training and logistics of training people for new jobs.

It was great to see Booker acknowledging that he doesn’t know much about geoengineering and would have to research it before having an opinion. That’s a reasonable order of things. And I liked that Yang brought up geoengineering as a component of the solution — geoengineering is something I’d researched from the ML / data modeling side, and it was good to see it put forth as something other than nuc’ing hurricanes. I’d have liked to see the problems with capitalist enterprise being the source of technological solutions (although Yang touches on the issue in ditching GDP as the sole measurement of economic success).

Biden’s whole performance was expectedly underwhelming. I’m not sure where he got his rep as a great people-person politician … because WTF is the point of arguing about whose state is getting it worse? Biden’s digression into base readiness is something I wish had follow-up. He started down a path, then drew back saying something like “I cannot get into that”. Which made me wonder what exactly *that* was. He might have been talking about the DoD Climate Report from Jan 2019 and not wanted to get into it as a digression wasting his free airtime. But my mind went down the path … a VP’s got high level clearance. Does he keep it after leaving office? Can he still request briefings from DNI? Can he *not* get into it?

Cooper had a card about Andrew Goldman because they knew the question was going to be asked — it was basically a “gotcha” setup. But the exchange highlighted how Biden is the embodiment of what people hate about politicians. The “depends on what the meaning of the word ‘is’ is” legal technicality dodge. My takeaway is that Biden knew the guy had been a co-founder of a natural gas extraction company and runs a hedge fund with diversified investments. But the dude *isn’t* the CEO or active board member of a 100% fossil fuel company, so I’m following the letter of the agreement. How’d he make his first million? How’d he make his subsequent hundreds of millions? Not relevant. The alternative is that Biden’s campaign is inept, and they don’t do background research on people hosting high-dollar fundraising events. I believe that about as much as I believe a former model didn’t put any thought into the words on the back of the jacket she was wearing.

Other candidates did better jobs answering ‘gotcha’ questions. Sanders refused to pledge that no taxpayer will shoulder the 16 trillion dollar burden for his environmental plan (and I love the idea of TVA’ing renewable energy production, which I hadn’t heard before) because *some* taxpayers WILL pay more. Yang’s response about electric cars was great – it would have been a great place to talk about Porsche’s Taycan (another pure electric car with sub three-second 0-60 time) event earlier in the day. But telling people that they’re going to LOVE driving an electric car is spot on. Warren won’t take the light-bulb bait and wants to focus on the oil industry, the electric power industry and the building industry (although it would have been nice if *she* had enumerated them)

Some didn’t seem well prepared for obvious gotcha’s. Booker gets a bit cornered by the “taking away my burgers” question … because he’s a vegan and doesn’t want to seem like one of *those* vegans? Klobuchar doesn’t have a good response about the dairy or cattle industry because she may need to run again in Minnesota. But the Biden exchange seemed like an ambush in that it wasn’t an obvious question. Cooper returning to the Goldman topic for a closing clarification was bad for Biden too. The last impression you got of Biden was him being defensive about possibly taking money from an LNG guy. Which at least gives the appearance that Biden is losing his de facto choice status. Could you have seen CNN going after Clinton like that last time around?

The whole thing got me thinking that the next president has a chance of making some progress in spite of Moscow Mitch. Piece-meal some of it through budget reconciliation. There’s SCOTUS precedent that the Executive branch can take money from one place and allocate it elsewhere for something that has been intentionally not funded by Congress (yet another point where Trump probably doesn’t even understand the ramifications of his short-term ‘win’). Withhold subsidies from one place (fossil fuels, meat, dairy) and use that money toward other initiatives (increased solar/wind rebates, increased veggie subsidies, electric car rebates). Adjust dietary guidelines and national school lunch requirements (it’s be a huge uproar, but imagine the impact of schools doing meatless Monday). Sue the companies that have internal documents from *decades* ago indicating that they knew burning fossil fuels was environmentally destructive. It’s no different than the tobacco companies — well, it’s worse because even if you lived on a remote island for your entire life and never encountered anyone who bought anything from ExxonMobile … you’re impacted by their products — so why *don’t* we have trillion dollar settlements from them funding the public solar/wind power company?

Accessing MobyVM

I needed to map an addition port into an existing Docker container. Now I know the right thing to do is to create a new container and do it right this time but GitLab’s container has problems running on the Windows Docker Desktop. Permission-based problems that I’m not particularly included to attempt to sort out just to run a simple sandbox. Which means I’d need to drop my config file back in place & recreate my sandbox projects. And since I’m using CI/CD variables which don’t export … recreating the sandbox projects is a bit of a PITA.

On Linux, I can fix this by editing the config.v2.json and hostconfig.json files … but this is Windows running a funky Hyper-V Linux. And it turns out you can access the files on this MobyVM.

docker run -it --rm --privileged --pid=host justincormack/nsenter1

Now I’m able to cd into /var/lib/docker/containers, find the full ID for my GitLab container and cd into it, and edit the two config files. If it is running, you need to stop the container prior to editing the config files.

config.v2.json — add the port to “ExposedPorts”

chStdin”:false,”AttachStdout”:false,”AttachStderr”:false,”ExposedPorts”:{“22/tcp”:{},”443/tcp”:{},”80/tcp”:{},”4567/tcp”:{}},”Tty”:fal…

hostconfig.json — add the port to “PortBindings”

ult”,”PortBindings”:{“22/tcp”:[{“HostIp”:””,”HostPort”:”22″}],”443/tcp”:[{“HostIp”:””,”HostPort”:”443″}],”80/tcp”:[{“HostIp”:””,”HostPort”:”80″}],”4567/tcp”:[{“HostIp”:””,”HostPort”:”4567″}]},”Res…

 

Stop the Windows Docker service, start it, then start the container again. Voila! The new port for the container registry is there without recreating the container.

Using File System As A Git Repository

I’ve used GitLab for quite some time, and as a full featured CI/CD platform that also provides git functionality … it’s awesome. But it’s also serious overkill for someone who wants to coordinate code with another developer or two. Or just keep history for their code. Or a backup. To accomplish this, all you need is drive space. If you’ve got a file server, any folder on the share can be a git repository.

On the server, create a git repository and add something to it.

Z:\Temp\test>git init
Initialized empty Git repository in Z:/Temp/test/.git/

Z:\Temp\test>notepad testfile.txt

Z:\Temp\test>git add testfile.txt

Z:\Temp\test>git commit -m "Initial file upload"
[master (root-commit) 9a3ebe7] Initial file upload
1 file changed, 1 insertion(+)
create mode 100644 testfile.txt

Then on your client, either clone the repo from the drive path

C:\Users\lisa>git clone file://z:/temp/test
Cloning into 'test'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.

Or from the UNC path

C:\Users\lisa>git clone file://server01.example.com/data/temp/test
Cloning into 'test'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.

I prefer to use the UNC paths – if my drive letter fails to map for some reason, the UNC path is still available.

If you’ve got pre-existing code, there’s a bit of a different process. On the server, create an empty folder and use “git init” to initialize the empty repo. On the client where the code exists, run:

git init
git add *
git commit -m “Initial code upload”
git remote add origin git clone file://server01.example.com/data/temp/test
git push origin master

 

Finding Disabled Accounts In Active Directory

When using Active Directory (AD) as a source of user data, it’s useful to filter out disabled accounts. Unfortunately, AD has a lot of different security-related settings glomed together in the userAccountControl attribute. Which means there’s no single attribute/value combination you can use to ignore disabled accounts.

The decimal value you see for userAccountControl isn’t terribly useful, but display it in binary and each bit position has a meaning. The userAccountControl value is just the number with a bunch of bits set. Numbering the bits from left to right, here is what each one means.

Bit # Meaning
0 Unused – must be 0
1 Unused – must be 0
2 Unused – must be 0
3 Unused – must be 0
4 Unused – must be 0
5 ADS_UF_PARTIAL_SECRETS_ACCOUNT
6 ADS_UF_NO_AUTH_DATA_REQUIRED
7 ADS_UF_TRUSTED_TO_AUTHENTICATE_FOR_DELEGATION
8 ADS_UF_PASSWORD_EXPIRED
9 ADS_UF_DONT_REQUIRE_PREAUTH
10 ADS_UF_USE_DES_KEY_ONLY
11 ADS_UF_NOT_DELEGATED
12 ADS_UF_TRUSTED_FOR_DELEGATION
13 ADS_UF_SMARTCARD_REQUIRED
14 Unused – must be 0
15 ADS_UF_DONT_EXPIRE_PASSWD
16 Unused – must be 0
17 Unused – must be 0
18 ADS_UF_SERVER_TRUST_ACCOUNT
19 ADS_UF_WORKSTATION_TRUST_ACCOUNT
20 ADS_UF_INTERDOMAIN_TRUST_ACCOUNT
21 Unused – must be 0
22 ADS_UF_NORMAL_ACCOUNT
23 Unused – must be 0
24 ADS_UF_ENCRYPTED_TEXT_PASSWORD_ALLOWED
25 ADS_UF_PASSWD_CANT_CHANGE
26 ADS_UF_PASSWD_NOTREQD
27 ADS_UF_LOCKOUT
28 ADS_UF_HOMEDIR_REQUIRED
29 Unused – must be 0
30 ADS_UF_ACCOUNT_DISABLE
31 Unused – must be 0

Bit #30 indicates if the account is disabled — 1 if the account is disabled, 0 if the account is enabled. Simple and direct approach is to “and” the attribute value with 0b10 to extract just the bit we care about. When the and operation returns 0, the account is enabled. When it returns 2 (0x10), the account is disabled.