Category: System Administration

Docker Desktop for Windows – Bind Mounts

I’ve been trying to set up a Docker container running an older CentOS, Apache, and PHP version as a sandbox for work. This would allow me to update code on my local computer, test changes, and then pull the changes to the development server for UAT testing. Setting up the base container was easy enough — installed a VM, tar’d off the system, and imported it as a Docker image. There’s a lot of optimization that could/should be done, but I was aiming for proof of concept at this stage.

I am using bind mounts for the website configuration and code — the website conf file in conf.d, the SSL certificates, and the vhtml folder which houses the web code. This means I can tweak the site config & code in my IDE, reload Apache in Docker, and validate my changes. It worked great until I connected to the company VPN. Attempting to access the mounted data just hangs. Nothing. Drop the VPN, and the files are there again.

There are two problems — firstly, the default VPN configuration does not allow access to local network resources. And, it seems, the Docker NAT is a local network resource. We use Cisco AnyConnect. In the settings, I checked off “Allow local (LAN) access when using VPN (if configured)”. Note the if configured — the server-side settings need to allow use of local resources when connected via VPN. Fortunately, people with WiFi printers complained about having to disconnect the VPN every time they wanted to print something; and accessing local resources is permitted in our profile.

Unfortunately, I still couldn’t access files on my mount points. Docker Desktop shared out my drive, and the server network mounts the CIFS share. With my domain credentials. An Active Directory domain which is most certainly not registered in the VPN DNS servers.

[root@5542506m1a5e /]# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QMCCTMGPBHQFW66ARPWHSQMWQL:/var/lib/docker/overlay2/l/IQ2YIH47ZXTN55PGH3BWUKFPTT,upperdir=/var/lib/docker/overlay2/d072c94532976a4196174751c57359139501739001e7b9d50de59041c768a307/diff,workdir=/var/lib/docker/overlay2/d072c94532976a4196174751c57359139501739001e7b9d50de59041c768a307/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
...
//10.0.75.1/D on /etc/httpd/certs type cifs (rw,relatime,vers=3.02,sec=ntlmsspi,cache=strict,username=myuid,domain=mydomain,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.75.1,file_mode=0755,dir_mode=0777,iocharset=utf8,nounix,serverino,mapposix,nobrl,mfsymlinks,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
...
tmpfs on /sys/firmware type tmpfs (ro,relatime)

To use the share when connected via the VPN, I needed to use the credentials of a local account here. Beyond creating a local administator-level account, you may need to add read/write permissions for that new account to your %userprofile% directory — inheritence is generally disabled & only the individual user has access to the folder.

Once there’s a local account set up to work, you’ve got to tell Docker to use it. In the settings, select “Shared Drives”. Use “Reset credentials” to open a prompt for the logon credentials that will be used to mount the shared volume.

o

Start the Docker container, VPN into the company network, and I’ve got a fully functional sandbox in a Docker container.

Cleaning Up Unused Docker Images

I’ve been using Docker for quite some time, but never had unused container images. This is partially because I installed a new hard drive and started from a blank slate, but also because I haven’t needed to use many different images to build my containers.

I’ve changed jobs recently and wanted to set up a container to mirror our web server. Which meant trying to get a CentOS 6.8 container going. Except there isn’t one from Cent anymore. And I don’t exactly trust random-dude-from-the-Internet’s OS. Download it and poke around without running it, sure … but that’s not a platform on which I can do my development.

And that means I’ve got a few images that I do not need. To view the list of images, use “docker images -a”

 

D:\docker>docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
openhab/openhab snapshot 8a4749c86ff3 4 weeks ago 527MB
docker4w/nsenter-dockerd latest 2f1c802f322f 9 months ago 187kB
centos/php-56-centos7 latest 92ed8b3a7cb4 15 months ago 617MB
…
13652604711/centos6.8-ssh latest 59ab169b5158 2 years ago 289MB

Then use “docker rmi imagename” to remove any unnecessary ones.

D:\docker>docker rmi centos/php-56-centos7
Untagged: centos/php-56-centos7:latest
Untagged: centos/php-56-centos7@sha256:f3c95020fa870fcefa7d1440d07a2b947834b87bdaf000588e84ef4a599c7546
Deleted: sha256:92ed8b3a7cb4d56d3a1c58386d966f22736010a292a81a72dddbc4ffc7cae3fd
Deleted: sha256:bdcb229c59ed69d26750cd0d24362670e1fa2ae9be6ef19aa3e7c5571a4a8503
Deleted: sha256:90eb7fca62f6c0febd9cc21544269029ff231f39f16054ba6b0ca93ec1037d97
Deleted: sha256:cdcf05e149fc6cb2801f7f93dce3acb54465fe6c46a16dd6135aa74d79bedffa
Deleted: sha256:139498a5907a4d17cf07b1400bdbdb4db5e9f1ac4e3985aac2b374eaa712d5fb
Deleted: sha256:5f0780b14e43db37e84162e0045657203ac1e9fb531cc3e879fa464eda013e79
Deleted: sha256:7e117241875497974bb56f09e6340e142a9acaa11af76917afab345acc25b5c1
Deleted: sha256:4b170488c295918f4d7618c2cd0b9b428d55ec952dd6a715593e3af34e538d94
Deleted: sha256:1e889f7360c52d1b20f93335382290445e4f257f08ccef01694837572842e95f
Deleted: sha256:43e653f84b79ba52711b0f726ff5a7fd1162ae9df4be76ca1de8370b8bbf9bb0

D:\docker>docker rmi 13652604711/centos6.8-ssh
Untagged: 13652604711/centos6.8-ssh:latest
Untagged: 13652604711/centos6.8-ssh@sha256:41bbe66ac18f199efac325d0d4bcb5d0390ec501ca82d6d1ce223df8a050be3a
Deleted: sha256:59ab169b5158a172079e2a89442936bc49292ea951f2eb9acb688a0ee34f95e1
Deleted: sha256:12d850520660ec9de87e84735a7067e663db282245502820f09dae5c937a93d2
Deleted: sha256:6b5c6954e3d511934786375730a068d0f013dcc99356a341a8c5d268a3b1cf3d

Un-killable Process

Scott had a Dolphin instance veg out on him. Not much for it other than killing the process. Except it didn’t kill. Now SIGTERM (15) I don’t expect to kill a vegged out process, but SIGKILL (9)? I’ve never seen that fail. A little research later, and I’ve discovered “uninterruptible sleep”.  Which, at 11PM is starting to sound really good to me. But not something I associate with computer programs. Essentially, processes that are waiting on I/O very briefly pop into this state and pop out of the state when the I/O operation completes. Code needs to have timeouts to prevent the application from getting stuck waiting for I/O. And, evidently, Scott has discovered a scenario in which Dolphin does not have a timeout.

How can you tell that your process is stuck in uninterruptible sleep? Use “ps u” (or “ps aux” for all processes) and check the “STAT” column.

From “man ps”:

PROCESS STATE CODES
Here are the different values that the s, stat and state output
specifiers (header "STAT" or "S") will display to describe the state of
a process:

D uninterruptible sleep (usually IO)
I Idle kernel thread
R running or runnable (on run queue)
S interruptible sleep (waiting for an event to complete)
T stopped by job control signal
t stopped by debugger during the tracing
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z defunct ("zombie") process, terminated but not reaped by its parent

For BSD formats and when the stat keyword is used, additional
characters may be displayed:

< high-priority (not nice to other users)
N low-priority (nice to other users)
L has pages locked into memory (for real-time and custom IO)
s is a session leader
l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
+ is in the foreground process group

Rebooting clears the process (or sorting whatever is blocking the I/O operation). But there are processes that “kill -9” won’t terminate.

Recreating Grub Bootloader After the Windows Install Wipes It

Setting up the dual-boot Windows/Fedora system was straight-forward on my laptop. I installed Windows, then installed Linux and grub mkconfig found Windows and included it in the menu. Scott already had Fedora, and we needed to repair his Windows installation. Which, of course, blew away grub. Easy enough to get back, provided you’ve got a Live USB installation from which to boot.

Boot the Live media and use fdisk to find the Linux partition (in our installations, /boot is contained within the root partition).

[root@fedora02 ~]# fdisk -l
Disk /dev/sda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xd847fbc2

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1026047 1024000 500M 83 Linux
/dev/sda2 1026048 20971519 19945472 9.5G 8e Linux LVM

Mount that partition somewhere:

mkdir /mnt/mycomputer
mount /dev/sda2 /mnt/mycomputer

Add bind mounts so /dev and /proc are in there

mount –bind /dev /mnt/mycomputer/dev
mount –bind /proc /mnt/mycomputer/proc

Chroot yourself into the mount point

chroot /mnt/mycomputer

Now you can reinstall grub. You don’t want the partition (e.g. /dev/sda2) but the disk. The following commands install the grub2 bootloader and reboot.

grub2-install /dev/sda
reboot

If you forget to pay attention on boot and thus inadvertently end up in the default operating system (<G>), edit /etc/default/grub and increase “GRUB_TIMEOUT=5”. Build the grub config — this should identify your Windows partition and include it in the menu

grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot again, and you’ll be able to select between Windows and Linux.

Using grub rescue to boot machine and repair MBR

Using grub rescue to boot machine and repair MBR

Use “ls” to find your partition list:
ls
(hd0) (hd0,msdos3) (hd0,msdos2) (hd0,msdos1)

Check the content of each to find your Linux partition:
ls (hd0,msdos3)/
ls (hd0,msdos2)/
ls (hd0,msdos1)/

You want the one with the /boot folder. In our case, this is (hd0,msdos3). The following commands will boot your Linux OS.

set prefix=(hd0,msdos3)/boot/grub2
set root=(hd0,msdos3)
insmod normal
normal

<root password for maintenance>

Use “df” or “mount” to figure out which disk holds the Linux partition. In our case, it is /dev/sdb. You don’t want the partition (e.g. /dev/sdb2) but the disk. The following commands install the grub2 bootloader and build a config file.

grub2-install /dev/sdb
grub2-mkconfig -o /boot/grub2/grub.cfg

Linux – Finding “Missing” Disk Space

You can have ‘stuff’ in a folder, mount a partition to that folder, and have disk space used for which you cannot account. Using a tool like df, you will see that 10GB of your 10GB disk is used, but using du the individual folders only account for 5GB.

Rather than randomly umounting partitions to see if there’s anything in the mount point folder, you can bind mount root to another location and check the disk utilization on the new mount.

 

[root@fedora123 ~]# mount -o bind / /mnt/fakeout/
[root@fedora123 ~]# du -sh `ls /mnt/fakeout | grep -v mnt`
0 bin
0 boot
280K ca
8.0K cacert.pem
4.0K careq.pem
0 certs
0 crl
0 dev
44M etc
52K home
0 index.txt
0 lib
0 lib64
0 media
0 newcerts
0 openhab
724K opt
4.0K private
0 proc
847M root
0 run
0 sbin
227M srv
0 sys
4.0K tmp
3.0G usr
4.0K var
[root@fedora123 ~]# umount fakeout

GPO Changes Not Reflected On Computer

A year or three ago, we had set up a group policy to display logon information in the Windows welcome screen — last logon time, information about any bad passwords since your last successful logon. It’s nice, but you are unable to log in if the information is unavailable. So we disabled the setting (just clearing a GPO setting doesn’t always remove the config from anywhere it’s already set … you’ve got to change to the desired setting). Aaaaand … one server still shows the logon info. Or, more accurately, fails to display it and prevents the domain account from logging on. Luckily it’s a member computer and we can just log in with local accounts.

I thought about just editing the local computer policy (which would have priority anyway) to disable the logon info, but the computer policy could not be opened. It threw a strange access denied error. I could edit the local user policy. Just not the computer policy.

It seems that the local computer policy got corrupted. After deleting registry.pol from c:\Windows\System32\GroupPolicy\Machine … I am able to modify the local computer security policy. GPO settings from the domain are also applied as expected. WooHoo! I can sign in using domain IDs again!

Two Approaches to Using PIP Through an Integrated Authenticated Proxy

The proxy at work uses integrated authentication. While BASIC auth prompts happily let you use a proxy of http://uid:pass@proxy:port, ours does not. There are two ways I’ve managed to use pip to install packages.

Proxied Proxy

The easiest approach is to use something that can handle the authenticated proxy, like Fiddler, as an intermediary. I do the same thing with perl’s PPM, docker pull … basically anywhere I’ve got to use a proxy that wants to pass through a proxy username.

Select Tools => Options to open the configuration dialog. Make sure you are handling SSL traffic — if not, check the box to “Capture HTTPS CONNECTs, “Decrypt HTTPS traffic”, and “Ignore server certificate errors” (it’s only unsafe if you don’t understand what you’re doing … don’t log into your bank account bouncing traffic through this config!)

On the “Connections” tab, check the port on which Fiddler is listening. If you cannot install Fiddler on the same box where you want to use pip, you’ll need to check off “Allow remote computers to connect” (and you won’t use localhost as the proxy hostname). Click OK, start capturing traffic (F12), and we’re ready to go.

Use the PIP command line to install the package but proxy the request through your Fiddler instance. In this example, Fiddler is installed on the local box and uses port 8888.

pip –trusted-host pypi.org –trusted-host files.pythonhosted.org –proxy http://localhost:8888 install SomePackage

This is nice because pip will automatically resolve dependencies. Not great if you’re not allowed to install your own software and cannot get Fiddler installed.

Dependency Nighmare

Back in the early days of Linux (think waaaay before package managers working against online repositories), we called this “dependency hell” — navigating dependency chains to get all of the required “stuff” on the box so you can install the thing you actually wanted.

Make a folder for all these wheels we’re going to end up downloading so it’s easy to clean up once we’re done. Search PyPi for the package you want. On the package page, select ‘Download Files’ and then download the whl

Use “pip install something.whl” to attempt installing it. If you’re lucky, you’ve got all the dependencies and the package will install. If you don’t have all of the dependencies, you’ll get an error telling you the first missing one. Go back to the pypi website & get that. Use “pip install somethingelse.whl” to install it and maybe get a dependency error here too. Once you get a dependency installed, try the package you want again. Got another error? Start downloading and installing again. Eventually, though, you’ll get all of the dependencies satisfied and be able to install the package you actually want.

Adding Python to Fedora Alternatives

Gimp installed Python 2.7). Which, of course, took over my system so nothing was using Python 3 anymore. We’ve used ‘alternatives’ to manage the Java installation, and I thought that might be a good solution in case I ever need to use Python 2

Add both Python versions to alternatives:

[lisa@fedora ~]# sudo alternatives –install /usr/bin/python python /usr/bin/python3.7 1
[lisa@fedora ~]# sudo alternatives –install /usr/bin/python python /usr/bin/python2.7 2

Select which one you want to use:

[lisa@fedora ~]# sudo alternatives –config python

There are 2 programs which provide ‘python’.

Selection Command
———————————————–
1 /usr/bin/python3.7
+ 2 /usr/bin/python2.7

Enter to keep the current selection[+], or type selection number: 1

And, of course, repeat the process for PIP:

[lisa@fedora ~]# sudo alternatives –install /usr/bin/pip pip /usr/bin/pip2.7 2
[lisa@fedora ~]# sudo alternatives –install /usr/bin/pip pip /usr/bin/pip3.7 1

[lisa@fedora ~]# sudo alternatives –config pip