Category: Coding

Renaming a Branch in Git

I finally had a situation where I needed to rename a branch in git. When I was the only one involved in a development effort (or even looking at it!), it didn’t really matter if I typo’d something. Exchange and Exchagne … I know what I meant. But working under a more formal development process, I started naming my branch after the issue ID. And managed to typo the first one. Sigh!

# Check out the incorrectly named branch

git checkout OSSA166

# Rename it with the correct name

git branch -m OSSA163

# See what you’ve got — the local one is right now, but the remote is still incorrectly named

git branch -a

* OSSA163
master
remotes/origin/HEAD -> origin/master
remotes/origin/OSSA166
remotes/origin/master
remotes/origin/uat

# Push a change to rename the remote one too

git push origin :OSSA166 OSSA163

Total 0 (delta 0), reused 0 (delta 0)

To ssh://git.example.com/path/to/my/repo.git

– [deleted]         OSSA166
* [new branch]      OSSA163 -> OSSA163

# And see what you’ve got again

git branch -a

* OSSA163
master
remotes/origin/HEAD -> origin/master
remotes/origin/OSSA163
remotes/origin/master
remotes/origin/uat

 

Docker Desktop for Windows – Bind Mounts

I’ve been trying to set up a Docker container running an older CentOS, Apache, and PHP version as a sandbox for work. This would allow me to update code on my local computer, test changes, and then pull the changes to the development server for UAT testing. Setting up the base container was easy enough — installed a VM, tar’d off the system, and imported it as a Docker image. There’s a lot of optimization that could/should be done, but I was aiming for proof of concept at this stage.

I am using bind mounts for the website configuration and code — the website conf file in conf.d, the SSL certificates, and the vhtml folder which houses the web code. This means I can tweak the site config & code in my IDE, reload Apache in Docker, and validate my changes. It worked great until I connected to the company VPN. Attempting to access the mounted data just hangs. Nothing. Drop the VPN, and the files are there again.

There are two problems — firstly, the default VPN configuration does not allow access to local network resources. And, it seems, the Docker NAT is a local network resource. We use Cisco AnyConnect. In the settings, I checked off “Allow local (LAN) access when using VPN (if configured)”. Note the if configured — the server-side settings need to allow use of local resources when connected via VPN. Fortunately, people with WiFi printers complained about having to disconnect the VPN every time they wanted to print something; and accessing local resources is permitted in our profile.

Unfortunately, I still couldn’t access files on my mount points. Docker Desktop shared out my drive, and the server network mounts the CIFS share. With my domain credentials. An Active Directory domain which is most certainly not registered in the VPN DNS servers.

[root@5542506m1a5e /]# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QMCCTMGPBHQFW66ARPWHSQMWQL:/var/lib/docker/overlay2/l/IQ2YIH47ZXTN55PGH3BWUKFPTT,upperdir=/var/lib/docker/overlay2/d072c94532976a4196174751c57359139501739001e7b9d50de59041c768a307/diff,workdir=/var/lib/docker/overlay2/d072c94532976a4196174751c57359139501739001e7b9d50de59041c768a307/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
...
//10.0.75.1/D on /etc/httpd/certs type cifs (rw,relatime,vers=3.02,sec=ntlmsspi,cache=strict,username=myuid,domain=mydomain,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.75.1,file_mode=0755,dir_mode=0777,iocharset=utf8,nounix,serverino,mapposix,nobrl,mfsymlinks,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
...
tmpfs on /sys/firmware type tmpfs (ro,relatime)

To use the share when connected via the VPN, I needed to use the credentials of a local account here. Beyond creating a local administator-level account, you may need to add read/write permissions for that new account to your %userprofile% directory — inheritence is generally disabled & only the individual user has access to the folder.

Once there’s a local account set up to work, you’ve got to tell Docker to use it. In the settings, select “Shared Drives”. Use “Reset credentials” to open a prompt for the logon credentials that will be used to mount the shared volume.

o

Start the Docker container, VPN into the company network, and I’ve got a fully functional sandbox in a Docker container.

Sparse Checkout With Git

I’ve encountered a few repositories that are huge. Unwieldy huge, and stuffed with files that aren’t relevant to what I need. The straight-forward solution is to use multiple repositories — that’s what I do at work with my code samples. There’s a different repo for each language because the PHP developers really don’t care what the C# code looks like. The Java developers don’t need a copy of the Python code. But there are advantages to having a single repository that may preclude you from taking the simple solution. Git sub-modules are an interesting approach — combining multiple repositories into a single functional unit. But that’s a pretty big change to an existing repo. And, if you participate in open source projects, it may not be your decision anyway.

There’s another option for selectively cloning when you’re working with a large repo — an option that doesn’t require any changes to the repository. An end user can perform a sparse checkout — essentially use a filter like .gitignore to select or deselect certain files/folders from being pulled into the local working directory. The file is named sparse-checkout and is located in .git\info — unlike a .gitignore file which indicates what shouldn’t get included, sparse-checking controls what is included (if you want an entire repo except one folder, use !path/to/folder/**)

The sparse-checkout file used to get just the core components of Scott’s OpenHAB helper libraries plus the OpenWeatherMap community scripts is:

.github/**
Core/**
Community/OpenWeatherMap/**

To use sparse checkout, set the core.sparseCheckout config value to true. You can add sparse checkout to a repo you’ve already cloned and use

git read-tree -mu HEAD

to “clean up” unwanted files. Or you can set up sparse checkout before you clone the repo

D:\tmp>mkdir ljrtest

D:\tmp>cd ljrtest

D:\tmp\ljrtest>git init
Initialized empty Git repository in D:/tmp/ljrtest/.git/

D:\tmp\ljrtest>git remote add origin https://github.com/openhab-scripters/openhab-helper-libraries

D:\tmp\ljrtest>git config core.sparseCheckout true

D:\tmp\ljrtest>copy ..\sparse-checkout .git\info\
1 file(s) copied.

D:\tmp\ljrtest>git pull origin master
remote: Enumerating objects: 3591, done.
remote: Total 3591 (delta 0), reused 0 (delta 0), pack-reused 3591R ), 7.00 MiB | 6.95 MiB/s
Receiving objects: 100% (3591/3591), 9.26 MiB | 7.22 MiB/s, done.
Resolving deltas: 100% (1786/1786), done.
From https://github.com/openhab-scripters/openhab-helper-libraries
* branch master -> FETCH_HEAD
* [new branch] master -> origin/master

D:\tmp\ljrtest>dir
Volume in drive D is DATA
Volume Serial Number is D8E9-3B61

Directory of D:\tmp\ljrtest

07/03/2019 09:07 AM <DIR> .
07/03/2019 09:07 AM <DIR> ..
07/03/2019 09:07 AM <DIR> .github
07/03/2019 09:07 AM <DIR> Community
07/03/2019 09:07 AM <DIR> Core
0 File(s) 0 bytes
5 Dir(s) 386,515,042,304 bytes free

D:\tmp\ljrtest>dir .\Community
Volume in drive D is DATA
Volume Serial Number is D8E9-3B61

Directory of D:\tmp\ljrtest\Community

07/03/2019 09:07 AM <DIR> .
07/03/2019 09:07 AM <DIR> ..
07/03/2019 09:07 AM <DIR> OpenWeatherMap
0 File(s) 0 bytes
3 Dir(s) 386,515,042,304 bytes free

Using sparse checkout, no one else has to do anything. Configure your client to get the files you want, and you’re set.

 

Two Approaches to Using PIP Through an Integrated Authenticated Proxy

The proxy at work uses integrated authentication. While BASIC auth prompts happily let you use a proxy of http://uid:pass@proxy:port, ours does not. There are two ways I’ve managed to use pip to install packages.

Proxied Proxy

The easiest approach is to use something that can handle the authenticated proxy, like Fiddler, as an intermediary. I do the same thing with perl’s PPM, docker pull … basically anywhere I’ve got to use a proxy that wants to pass through a proxy username.

Select Tools => Options to open the configuration dialog. Make sure you are handling SSL traffic — if not, check the box to “Capture HTTPS CONNECTs, “Decrypt HTTPS traffic”, and “Ignore server certificate errors” (it’s only unsafe if you don’t understand what you’re doing … don’t log into your bank account bouncing traffic through this config!)

On the “Connections” tab, check the port on which Fiddler is listening. If you cannot install Fiddler on the same box where you want to use pip, you’ll need to check off “Allow remote computers to connect” (and you won’t use localhost as the proxy hostname). Click OK, start capturing traffic (F12), and we’re ready to go.

Use the PIP command line to install the package but proxy the request through your Fiddler instance. In this example, Fiddler is installed on the local box and uses port 8888.

pip –trusted-host pypi.org –trusted-host files.pythonhosted.org –proxy http://localhost:8888 install SomePackage

This is nice because pip will automatically resolve dependencies. Not great if you’re not allowed to install your own software and cannot get Fiddler installed.

Dependency Nighmare

Back in the early days of Linux (think waaaay before package managers working against online repositories), we called this “dependency hell” — navigating dependency chains to get all of the required “stuff” on the box so you can install the thing you actually wanted.

Make a folder for all these wheels we’re going to end up downloading so it’s easy to clean up once we’re done. Search PyPi for the package you want. On the package page, select ‘Download Files’ and then download the whl

Use “pip install something.whl” to attempt installing it. If you’re lucky, you’ve got all the dependencies and the package will install. If you don’t have all of the dependencies, you’ll get an error telling you the first missing one. Go back to the pypi website & get that. Use “pip install somethingelse.whl” to install it and maybe get a dependency error here too. Once you get a dependency installed, try the package you want again. Got another error? Start downloading and installing again. Eventually, though, you’ll get all of the dependencies satisfied and be able to install the package you actually want.

Adding Python to Fedora Alternatives

Gimp installed Python 2.7). Which, of course, took over my system so nothing was using Python 3 anymore. We’ve used ‘alternatives’ to manage the Java installation, and I thought that might be a good solution in case I ever need to use Python 2

Add both Python versions to alternatives:

[lisa@fedora ~]# sudo alternatives –install /usr/bin/python python /usr/bin/python3.7 1
[lisa@fedora ~]# sudo alternatives –install /usr/bin/python python /usr/bin/python2.7 2

Select which one you want to use:

[lisa@fedora ~]# sudo alternatives –config python

There are 2 programs which provide ‘python’.

Selection Command
———————————————–
1 /usr/bin/python3.7
+ 2 /usr/bin/python2.7

Enter to keep the current selection[+], or type selection number: 1

And, of course, repeat the process for PIP:

[lisa@fedora ~]# sudo alternatives –install /usr/bin/pip pip /usr/bin/pip2.7 2
[lisa@fedora ~]# sudo alternatives –install /usr/bin/pip pip /usr/bin/pip3.7 1

[lisa@fedora ~]# sudo alternatives –config pip

Iterating through files/folders with spaces in name using find in bash

Ran into a problem using Sphinx to document some Python modules and scripts that Scott put together for OpenHAB. They’re making some changes to the files to get Sphinx to process them, thus making copies of the original code. Problem is, some of the folders just weren’t showing up in the copy. Needed to change IFS in order to tokenize the find results into full paths that don’t break on spaces in file or folder names.

SAVEDIFS=$IFS
IFS=$(echo -en "\n\b")

for DIRNAME in $(find "$COMMUNITY_DIR" -maxdepth 1 -type d 2>/dev/null); do
     echo $DIRNAME
done
IFS=$SAVEDIFS

Git Commands

There are a few git commands that we use when working with the OpenHAB and helper library repositories. The OpenHAB Eclipse project sets uses a split push/pull repository where the *fetch* repo is the organization and the *push* repo is your personal repo. This is reasonable because you do not have permissions to write to the organizational repository. You can use the same split-repository setup for other projects. Clone the project either from the organization’s repo, and then change the push URL to your personal repository.

# Show list of remotes
[lisa@linux ~]# git remote -v
origin https://github.com/openhab-scripters/openhab-helper-libraries (fetch)
origin https://github.com/openhab-scripters/openhab-helper-libraries (push)

# Set push remote to PERSONAL repository
[lisa@linux ~]# git remote set-url –push origin https://github.com/ljr55555/openhab-helper-libraries

# Show list of remotes — verification step
[lisa@linux ~]# git remote -v
origin https://github.com/openhab-scripters/openhab-helper-libraries (fetch)
origin https://github.com/ljr55555/openhab-helper-libraries (push)

While the split repository setup prevents accidentally attempting to push changes to a repo to which you lack write access, I find it a little confusing. Instead, I add specific repos for ORG (the organizational repo) and my personal repo.
The drawback to this configuration is that you *can* attempt to push changes directly to the organization repo — which will either yield an error because you lack access or will inadvertently publish code in the org repo because you don’t lack access.

# Add ORG repo with organizational repo URL
[lisa@linux ~]# git remote add ORG https://github.com/openhab-scripters/openhab-helper-libraries
# Add LJR repo with personal fork URL
[lisa@linux ~]# git remote add LJR https://github.com/ljr55555/openhab-helper-libraries
[lisa@linux ~]# git remote -v
LJR https://github.com/ljr55555/openhab-helper-libraries (fetch)
LJR https://github.com/ljr55555/openhab-helper-libraries (push)
ORG https://github.com/openhab-scripters/openhab-helper-libraries (fetch)
ORG https://github.com/openhab-scripters/openhab-helper-libraries (push)
origin https://github.com/openhab-scripters/openhab-helper-libraries (fetch)
origin https://github.com/ljr55555/openhab-helper-libraries (push)

# Scenario: Someone has updated ORG master branch
# I want to incorporate those changes into PERSONAL master branch and push them into my repo
[lisa@linux ~]# git checkout master # Switch to your local master branch
[lisa@linux ~]# git fetch ORG/master # Get changes from Organization master
[lisa@linux ~]# git rebase ORG/master # Apply those changes to local master
[lisa@linux ~]# git push –force LJR master # Overwrite personal repo master with updated info

# Scenario: Someone has updated ORG master branch.
# I want to incorporate those changes in PERSONAL lucid-migration branch
[lisa@linux ~]# git checkout master # Switch to your local master branch
[lisa@linux ~]# git fetch ORG/master # Get changes from Organization master
[lisa@linux ~]# git rebase ORG/master # Apply those changes to local master
[lisa@linux ~]# git checkout lucid-migration # Switch back to your local lucid-migration branch
[lisa@linux ~]# git rebase –preserve-merges master # Rebase your local lucid-migration (checked out branch) onto local master
[lisa@linux ~]# git push –force-with-lease LJR lucid-migration # Overwrite personal repo lucid-migration branch with updated info

 

And a few misc commands that I want to remember
# Check username and email
[lisa@linux ~]# git config –list

# Set username and e-mail address
[lisa@linux ~]# git config –global user.name “FIRST_NAME LAST_NAME”
[lisa@linux ~]# git config –global user.email “MY_NAME@example.com”

# merge gone bad, bail!
[lisa@linux ~]# git merge –abort

# Forgot to add sign-off on commit
[lisa@linux ~]# git commit –amend

Using sed to insert lines into a file

I’ve used sed to replace file content — use a regex to replace the sendmail.cf line that routes mail directly with a smarthost directive

sed -i -e 's/^DS/DS\\\[mailTWB.example.com\\\]/' $strSendmailDirectory/etc/mail/sendmail.cf

But I’ve needed to prepend text to a file. Turns out sed acn do that. In fact, you can insert strings at any line number. Using “sed -i ‘5s;^;StringsToInsert\n;’ filename.xtn will insert “StringsToInsert\n” at line 5. To prepend text to a file, use “1s”

[lisa@fedora tmp]# cat test.txt;sed -i ‘5s;^;NewLine1\nNewLine2\n;’ test.txt;cat test.txt
Line 1
Line 2
Line 3
Line 4
Line 5
Line 6
**********
Line 1
Line 2
Line 3
Line 4
NewLine1
NewLine2
Line 5
Line 6
**********

 

I’ve also come across an oddity in the Win32 sed — the method I usually use to blow away everything after a newline for some reason blows away everything after the first line. Works fine on RHEL7 and Fedora29, so the quick solution is “run it from the Linux box”.

C:\temp>cat input.txt
line 1
line 2

line 3
line 4
line 5
C:\temp>sed -i ‘/^$/q’ input.txt&cat input.txt
line 1

Did you know … you can open files in VSCode over SSH!?

The plug-in is a preview and you need to use VS Code Insiders to install it … but you can open files and folders directly from a *n?x server via SSH. This is a great way to circumvent Samba quirks (changing the case of a file name, filemode differences between the Samba share and the local files causing all files to be marked as changed, etc) – and can even eliminate the need to load file sharing servers like Samba in the first place.

Once the plug-in is installed, a “Remote – SSH” icon appears in the left-hand menu bar. There is a single configuration option for a file containing host definitions. You’ll want to set up key-based authentication and include the path to the authorized private key in your host config.

Right-clicking a host will allow you to open a file or folder within the current VSCode window or launch a new window.

One caveat – you are running git commands from the context of the remote machine … this means you’ll need a user name set up there or your commits show up with the local logged on username and username@hostname address.