Category: Coding

Sparse Checkout With Git

I’ve encountered a few repositories that are huge. Unwieldy huge, and stuffed with files that aren’t relevant to what I need. The straight-forward solution is to use multiple repositories — that’s what I do at work with my code samples. There’s a different repo for each language because the PHP developers really don’t care what the C# code looks like. The Java developers don’t need a copy of the Python code. But there are advantages to having a single repository that may preclude you from taking the simple solution. Git sub-modules are an interesting approach — combining multiple repositories into a single functional unit. But that’s a pretty big change to an existing repo. And, if you participate in open source projects, it may not be your decision anyway.

There’s another option for selectively cloning when you’re working with a large repo — an option that doesn’t require any changes to the repository. An end user can perform a sparse checkout — essentially use a filter like .gitignore to select or deselect certain files/folders from being pulled into the local working directory. The file is named sparse-checkout and is located in .git\info — unlike a .gitignore file which indicates what shouldn’t get included, sparse-checking controls what is included (if you want an entire repo except one folder, use !path/to/folder/**)

The sparse-checkout file used to get just the core components of Scott’s OpenHAB helper libraries plus the OpenWeatherMap community scripts is:

.github/**
Core/**
Community/OpenWeatherMap/**

To use sparse checkout, set the core.sparseCheckout config value to true. You can add sparse checkout to a repo you’ve already cloned and use

git read-tree -mu HEAD

to “clean up” unwanted files. Or you can set up sparse checkout before you clone the repo

D:\tmp>mkdir ljrtest

D:\tmp>cd ljrtest

D:\tmp\ljrtest>git init
Initialized empty Git repository in D:/tmp/ljrtest/.git/

D:\tmp\ljrtest>git remote add origin https://github.com/openhab-scripters/openhab-helper-libraries

D:\tmp\ljrtest>git config core.sparseCheckout true

D:\tmp\ljrtest>copy ..\sparse-checkout .git\info\
1 file(s) copied.

D:\tmp\ljrtest>git pull origin master
remote: Enumerating objects: 3591, done.
remote: Total 3591 (delta 0), reused 0 (delta 0), pack-reused 3591R ), 7.00 MiB | 6.95 MiB/s
Receiving objects: 100% (3591/3591), 9.26 MiB | 7.22 MiB/s, done.
Resolving deltas: 100% (1786/1786), done.
From https://github.com/openhab-scripters/openhab-helper-libraries
* branch master -> FETCH_HEAD
* [new branch] master -> origin/master

D:\tmp\ljrtest>dir
Volume in drive D is DATA
Volume Serial Number is D8E9-3B61

Directory of D:\tmp\ljrtest

07/03/2019 09:07 AM <DIR> .
07/03/2019 09:07 AM <DIR> ..
07/03/2019 09:07 AM <DIR> .github
07/03/2019 09:07 AM <DIR> Community
07/03/2019 09:07 AM <DIR> Core
0 File(s) 0 bytes
5 Dir(s) 386,515,042,304 bytes free

D:\tmp\ljrtest>dir .\Community
Volume in drive D is DATA
Volume Serial Number is D8E9-3B61

Directory of D:\tmp\ljrtest\Community

07/03/2019 09:07 AM <DIR> .
07/03/2019 09:07 AM <DIR> ..
07/03/2019 09:07 AM <DIR> OpenWeatherMap
0 File(s) 0 bytes
3 Dir(s) 386,515,042,304 bytes free

Using sparse checkout, no one else has to do anything. Configure your client to get the files you want, and you’re set.

 

Two Approaches to Using PIP Through an Integrated Authenticated Proxy

The proxy at work uses integrated authentication. While BASIC auth prompts happily let you use a proxy of http://uid:pass@proxy:port, ours does not. There are two ways I’ve managed to use pip to install packages.

Proxied Proxy

The easiest approach is to use something that can handle the authenticated proxy, like Fiddler, as an intermediary. I do the same thing with perl’s PPM.

Select Tools => Options to open the configuration dialog. Make sure you are handling SSL traffic — if not, check the box to “Capture HTTPS CONNECTs, “Decrypt HTTPS traffic”, and “Ignore server certificate errors” (it’s only unsafe if you don’t understand what you’re doing … don’t log into your bank account bouncing traffic through this config!)

On the “Connections” tab, check the port on which Fiddler is listening. If you cannot install Fiddler on the same box where you want to use pip, you’ll need to check off “Allow remote computers to connect” (and you won’t use localhost as the proxy hostname). Click OK, start capturing traffic (F12), and we’re ready to go.

Use the PIP command line to install the package but proxy the request through your Fiddler instance. In this example, Fiddler is installed on the local box and uses port 8888.

pip –trusted-host pypi.org –trusted-host files.pythonhosted.org –proxy http://localhost:8888 install SomePackage

This is nice because pip will automatically resolve dependencies. Not great if you’re not allowed to install your own software.

Dependency Hell

Back in the early days of Linux (think mid 90’s, waaaay before package managers working against online repositories), we called this “dependency hell” — navigating dependency chains to get all of the required “stuff” on the box so you can install the thing you actually wanted.

Make a folder for all these wheels we’re going to end up downloading so it’s easy to clean up once we’re done. Search PyPi for the package you want. On the package page, select ‘Download Files’ and then download the whl

Use “pip install something.whl” to attempt installing it. If you’re lucky, you’ve got all the dependencies and the package will install. If you don’t have all of the dependencies, you’ll get an error telling you the first missing one. Go back to the pypi website & get that. Use “pip install somethingelse.whl” to install it and maybe get a dependency error here too. Once you get a dependency installed, try the package you want again. Got another error? Start downloading and installing again. Eventually, though, you’ll get all of the dependencies satisfied and be able to install the package you actually want.

Adding Python to Fedora Alternatives

Gimp installed Python 2.7). Which, of course, took over my system so nothing was using Python 3 anymore. We’ve used ‘alternatives’ to manage the Java installation, and I thought that might be a good solution in case I ever need to use Python 2

Add both Python versions to alternatives:

[lisa@fedora ~]# sudo alternatives –install /usr/bin/python python /usr/bin/python3.7 1
[lisa@fedora ~]# sudo alternatives –install /usr/bin/python python /usr/bin/python2.7 2

Select which one you want to use:

[lisa@fedora ~]# sudo alternatives –config python

There are 2 programs which provide ‘python’.

Selection Command
———————————————–
1 /usr/bin/python3.7
+ 2 /usr/bin/python2.7

Enter to keep the current selection[+], or type selection number: 1

And, of course, repeat the process for PIP:

[lisa@fedora ~]# sudo alternatives –install /usr/bin/pip pip /usr/bin/pip2.7 2
[lisa@fedora ~]# sudo alternatives –install /usr/bin/pip pip /usr/bin/pip3.7 1

[lisa@fedora ~]# sudo alternatives –config pip

Iterating through files/folders with spaces in name using find in bash

Ran into a problem using Sphinx to document some Python modules and scripts that Scott put together for OpenHAB. They’re making some changes to the files to get Sphinx to process them, thus making copies of the original code. Problem is, some of the folders just weren’t showing up in the copy. Needed to change IFS in order to tokenize the find results into full paths that don’t break on spaces in file or folder names.

SAVEDIFS=$IFS
IFS=$(echo -en "\n\b")

for DIRNAME in $(find "$COMMUNITY_DIR" -maxdepth 1 -type d 2>/dev/null); do
     echo $DIRNAME
done
IFS=$SAVEDIFS

Git Commands

There are a few git commands that we use when working with the OpenHAB and helper library repositories. The OpenHAB Eclipse project sets uses a split push/pull repository where the *fetch* repo is the organization and the *push* repo is your personal repo. This is reasonable because you do not have permissions to write to the organizational repository. You can use the same split-repository setup for other projects. Clone the project either from the organization’s repo, and then change the push URL to your personal repository.

# Show list of remotes
[lisa@linux ~]# git remote -v
origin https://github.com/openhab-scripters/openhab-helper-libraries (fetch)
origin https://github.com/openhab-scripters/openhab-helper-libraries (push)

# Set push remote to PERSONAL repository
[lisa@linux ~]# git remote set-url –push origin https://github.com/ljr55555/openhab-helper-libraries

# Show list of remotes — verification step
[lisa@linux ~]# git remote -v
origin https://github.com/openhab-scripters/openhab-helper-libraries (fetch)
origin https://github.com/ljr55555/openhab-helper-libraries (push)

While the split repository setup prevents accidentally attempting to push changes to a repo to which you lack write access, I find it a little confusing. Instead, I add specific repos for ORG (the organizational repo) and my personal repo.
The drawback to this configuration is that you *can* attempt to push changes directly to the organization repo — which will either yield an error because you lack access or will inadvertently publish code in the org repo because you don’t lack access.

# Add ORG repo with organizational repo URL
[lisa@linux ~]# git remote add ORG https://github.com/openhab-scripters/openhab-helper-libraries
# Add LJR repo with personal fork URL
[lisa@linux ~]# git remote add LJR https://github.com/ljr55555/openhab-helper-libraries
[lisa@linux ~]# git remote -v
LJR https://github.com/ljr55555/openhab-helper-libraries (fetch)
LJR https://github.com/ljr55555/openhab-helper-libraries (push)
ORG https://github.com/openhab-scripters/openhab-helper-libraries (fetch)
ORG https://github.com/openhab-scripters/openhab-helper-libraries (push)
origin https://github.com/openhab-scripters/openhab-helper-libraries (fetch)
origin https://github.com/ljr55555/openhab-helper-libraries (push)

# Scenario: Someone has updated ORG master branch
# I want to incorporate those changes into PERSONAL master branch and push them into my repo
[lisa@linux ~]# git checkout master # Switch to your local master branch
[lisa@linux ~]# git fetch ORG/master # Get changes from Organization master
[lisa@linux ~]# git rebase ORG/master # Apply those changes to local master
[lisa@linux ~]# git push –force LJR master # Overwrite personal repo master with updated info

# Scenario: Someone has updated ORG master branch.
# I want to incorporate those changes in PERSONAL lucid-migration branch
[lisa@linux ~]# git checkout master # Switch to your local master branch
[lisa@linux ~]# git fetch ORG/master # Get changes from Organization master
[lisa@linux ~]# git rebase ORG/master # Apply those changes to local master
[lisa@linux ~]# git checkout lucid-migration # Switch back to your local lucid-migration branch
[lisa@linux ~]# git rebase –preserve-merges master # Rebase your local lucid-migration (checked out branch) onto local master
[lisa@linux ~]# git push –force-with-lease LJR lucid-migration # Overwrite personal repo lucid-migration branch with updated info

 

And a few misc commands that I want to remember
# Check username and email
[lisa@linux ~]# git config –list

# Set username and e-mail address
[lisa@linux ~]# git config –global user.name “FIRST_NAME LAST_NAME”
[lisa@linux ~]# git config –global user.email “MY_NAME@example.com”

# merge gone bad, bail!
[lisa@linux ~]# git merge –abort

# Forgot to add sign-off on commit
[lisa@linux ~]# git commit –amend

Using sed to insert lines into a file

I’ve used sed to replace file content — use a regex to replace the sendmail.cf line that routes mail directly with a smarthost directive

sed -i -e 's/^DS/DS\\\[mailTWB.example.com\\\]/' $strSendmailDirectory/etc/mail/sendmail.cf

But I’ve needed to prepend text to a file. Turns out sed acn do that. In fact, you can insert strings at any line number. Using “sed -i ‘5s;^;StringsToInsert\n;’ filename.xtn will insert “StringsToInsert\n” at line 5. To prepend text to a file, use “1s”

[lisa@fedora tmp]# cat test.txt;sed -i ‘5s;^;NewLine1\nNewLine2\n;’ test.txt;cat test.txt
Line 1
Line 2
Line 3
Line 4
Line 5
Line 6
**********
Line 1
Line 2
Line 3
Line 4
NewLine1
NewLine2
Line 5
Line 6
**********

 

I’ve also come across an oddity in the Win32 sed — the method I usually use to blow away everything after a newline for some reason blows away everything after the first line. Works fine on RHEL7 and Fedora29, so the quick solution is “run it from the Linux box”.

C:\temp>cat input.txt
line 1
line 2

line 3
line 4
line 5
C:\temp>sed -i ‘/^$/q’ input.txt&cat input.txt
line 1

Did you know … you can open files in VSCode over SSH!?

The plug-in is a preview and you need to use VS Code Insiders to install it … but you can open files and folders directly from a *n?x server via SSH. This is a great way to circumvent Samba quirks (changing the case of a file name, filemode differences between the Samba share and the local files causing all files to be marked as changed, etc) – and can even eliminate the need to load file sharing servers like Samba in the first place.

Once the plug-in is installed, a “Remote – SSH” icon appears in the left-hand menu bar. There is a single configuration option for a file containing host definitions. You’ll want to set up key-based authentication and include the path to the authorized private key in your host config.

Right-clicking a host will allow you to open a file or folder within the current VSCode window or launch a new window.

One caveat – you are running git commands from the context of the remote machine … this means you’ll need a user name set up there or your commits show up with the local logged on username and username@hostname address.

 

HTML Opacity v/s Alpha

I am building a page that allows employees to search for public MS Teams groups — for some reason, Teams uses a ‘starts with’ search, and our staff rarely manages to find the public Teams that are out there. I wanted the list of teams and descriptions to have a visible line separation, and a table border looked bad with the enterprise color scheme. I decided to use even/odd table rows to display a slightly lighter background color. I set an opacity on the background so the actual background image is still visible.

My font colors changed! The opacity applied to the text as well.

tr:nth-child(even) {background-color: rgb(52,52,52); opacity: 0.5;}

Instead of setting an opacity on the row, I added an alpha channel to the row background color without impacting the text within table cells.

tr:nth-child(even) {background-color: rgba(52,52,52,0.5);}

 

 

List locally installed Python modules

I’ve been helping someone else get an Azure bot running on their system … which involves a lot of “what do I have that you don’t” … for which listing locally installed python modules is incredibly helpful.

python -c “import pkg_resources; print([(d.project_name, d.version) for d in pkg_resources.working_set])”

 

[lisa@cent6 ljr]# python -c “import pkg_resources; print([(d.project_name, d.version) for d in pkg_resources.working_set])”
[(‘scipy’, ‘1.2.0’), (‘scikit-learn’, ‘0.20.2’), (‘PyYAML’, ‘3.13’), (‘PyMySQL’, ‘0.9.3’), (‘pycares’, ‘2.4.0’), (‘numpy’, ‘1.16.0’), (‘multidict’, ‘4.5.2’), (‘Cython’, ‘0.29.4’), (‘coverage’, ‘4.5.2’), (‘aiohttp’, ‘3.0.9’), (‘yarl’, ‘1.3.0’), (‘wrapt’, ‘1.11.1’), (‘vcrpy’, ‘2.0.1’), (‘typing’, ‘3.6.6’), (‘sklearn’, ‘0.0’), (‘singledispatch’, ‘3.4.0.3’), (‘sharepy’, ‘1.3.0’), (‘requests-toolbelt’, ‘0.9.1’), (‘requests-oauthlib’, ‘1.2.0’), (‘pytest’, ‘4.1.1’), (‘pytest-cov’, ‘2.6.1’), (‘pytest-asyncio’, ‘0.10.0’), (‘PyJWT’, ‘1.7.1’), (‘py’, ‘1.7.0’), (‘pluggy’, ‘0.8.1’), (‘oauthlib’, ‘3.0.1’), (‘nltk’, ‘3.4’), (‘msrest’, ‘0.4.29’), (‘more-itertools’, ‘5.0.0’), (‘isodate’, ‘0.6.0’), (‘ConfigArgParse’, ‘0.14.0’), (‘certifi’, ‘2018.11.29’), (‘botframework-connector’, ‘4.0.0a6’), (‘botbuilder-schema’, ‘4.0.0a6’), (‘botbuilder-azure’, ‘4.0.0a6’), (‘azure-devtools’, ‘1.1.1’), (‘azure-cosmos’, ‘3.0.0’), (‘attrs’, ‘18.2.0’), (‘atomicwrites’, ‘1.2.1’), (‘async-timeout’, ‘2.0.1’), (‘aiodns’, ‘1.2.0’), (‘botbuilder-core’, ‘4.0.0a6’), (‘systemd-python’, ‘234’), (‘smartcols’, ‘0.3.0’), (‘setools’, ‘4.1.1’), (‘rpm’, ‘4.14.2.1’), (‘gpg’, ‘1.11.1’), (‘cryptography’, ‘2.3’), (‘cffi’, ‘1.11.5’), (‘urllib3’, ‘1.24.1’), (‘SSSDConfig’, ‘2.0.0’), (‘slip’, ‘0.6.4’), (‘slip.dbus’, ‘0.6.4’), (‘six’, ‘1.11.0’), (‘setuptools’, ‘40.4.3’), (‘sepolicy’, ‘1.1’), (‘requests’, ‘2.20.0’), (‘PySocks’, ‘1.6.8’), (‘pyparsing’, ‘2.2.0’), (‘pyOpenSSL’, ‘18.0.0’), (‘pykickstart’, ‘3.16’), (‘PyGObject’, ‘3.30.4’), (‘pycparser’, ‘2.14’), (‘ply’, ‘3.9’), (‘pip’, ‘18.1’), (‘ordered-set’, ‘2.0.2’), (‘isc’, ‘2.0’), (‘IPy’, ‘0.81’), (‘iotop’, ‘0.6’), (‘iniparse’, ‘0.4’), (‘idna’, ‘2.7’), (‘distro’, ‘1.3.0’), (‘decorator’, ‘4.3.0’), (‘chardet’, ‘3.0.4’), (‘asn1crypto’, ‘0.24.0’)]