Testing udev Rules

The first step of testing a udev rule is to determine the actual device you want to test. Get the info for the /dev/thing and find the real /device/path/…. (note, make sure you’re not in a “looking at parent” section — you want the one all the way at the top)

[lisa@linuxhost dev]# udevadm info -a /dev/ttyUSB0 | more
...
looking at device '/devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0':
...

Once you have the device, use udevadm in test mode and you will see the results from all of your udev rules. Including group & permission mask applied to the device.

[lisa@linuxhost dev]# udevadm test /devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0 | more
calling: test
version 238
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
...
47437 strings (385590 bytes), 43385 de-duplicated (347383 bytes), 4053 trie nodes used
PROGRAM 'usb_modeswitch --symlink-name /devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0 10c4 8a2a ' /usr/lib/udev/rules.d/40-usb_modeswitch.rules:10
starting 'usb_modeswitch --symlink-name /devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0 10c4 8a2a '
Process 'usb_modeswitch --symlink-name /devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0 10c4 8a2a ' succeeded.
GROUP 18 /usr/lib/udev/rules.d/50-udev-default.rules:25
IMPORT builtin 'hwdb' /usr/lib/udev/rules.d/60-serial.rules:7
IMPORT builtin 'usb_id' /usr/lib/udev/rules.d/60-serial.rules:8
/sys/devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0: if_class 255 protocol 0
IMPORT builtin 'hwdb' /usr/lib/udev/rules.d/60-serial.rules:8
IMPORT builtin 'path_id' /usr/lib/udev/rules.d/60-serial.rules:15
LINK 'serial/by-path/pci-0000:00:02.0-usb-0:1:1.0-port0' /usr/lib/udev/rules.d/60-serial.rules:17
IMPORT builtin skip 'usb_id' /usr/lib/udev/rules.d/60-serial.rules:19
LINK 'serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_90F0016B-if00-port0' /usr/lib/udev/rules.d/60-serial.rules:24
GROUP 18 /etc/udev/rules.d/99-server.rules:5
MODE 0666 /etc/udev/rules.d/99-server.rules:5
LINK 'ttyUSB-5' /etc/udev/rules.d/99-server.rules:5
handling device node '/dev/ttyUSB0', devnum=c188:0, mode=0666, uid=0, gid=18
preserve permissions /dev/ttyUSB0, 020666, uid=0, gid=18
preserve already existing symlink '/dev/char/188:0' to '../ttyUSB0'
found 'c188:0' claiming '/run/udev/links/\x2fserial\x2fby-id\x2fusb-Silicon_Labs_HubZ_Smart_Home_Controller_90F0016B-if00-port0'
creating link '/dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_90F0016B-if00-port0' to '/dev/ttyUSB0'
preserve already existing symlink '/dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_90F0016B-if00-port0' to '../../ttyUSB0'
found 'c188:0' claiming '/run/udev/links/\x2fserial\x2fby-path\x2fpci-0000:00:02.0-usb-0:1:1.0-port0'
creating link '/dev/serial/by-path/pci-0000:00:02.0-usb-0:1:1.0-port0' to '/dev/ttyUSB0'
preserve already existing symlink '/dev/serial/by-path/pci-0000:00:02.0-usb-0:1:1.0-port0' to '../../ttyUSB0'
found 'c188:0' claiming '/run/udev/links/\x2fttyUSB-5'
creating link '/dev/ttyUSB-5' to '/dev/ttyUSB0'
preserve already existing symlink '/dev/ttyUSB-5' to 'ttyUSB0'
...

Or as a one-liner:

udevadm test `udevadm info -a /dev/ttyUSB0 | grep "looking at device" | sed "s/looking at device '//" | sed "s/'://"`

Just say the word

It worries me — the did he / didn’t he “say the n word” question once again surrounding Trump. Not because I think he did or did not use the term, but because the discussion is meaningless. Trump will deny saying it — hell, he denies writing things that are archived in his Twitter account. He denies saying things that even when told there is a publicly available recording of him saying it. The racists among his supporters will see a wink with that denial. Some willfully blind supports will believe the denial. Opponents will assume the tape exists. Whatever.

Words are powerful, but not in the way this debate seems to imply. Not saying a specific word does not magically cure the social, political, and economic problems in this country any more than having a president say radical Islamic terrorism magically solved the real social, political, and economic problems that lead to terrorist attacks.

But what makes a single word the arbiter of racism? I had a physical education instructor who was sexist. The fact he called every girl in his class “chick” didn’t make him sexist. His belief that we were less capable because of our gender, that our time in physical education would best be spent sitting on the bleachers fixing up our nails, that we did not have the mental capacity to be taught. That made him sexist. Had the man respectfully called me by my proper name every single time … he would still have been a sexist asshole who had no business teaching school children.

Someone who made his political name demanding a president show him some ID, who knowingly called Mexican emigrants a bunch of rapists, who thinks shutting down Mosques bears consideration, who refers to hut-dwellers from shit-hole countries … words are powerful, but refraining from uttering a specific word does not negate his racism. Maybe the dude was not a racist (discrimination against tenants and such says otherwise, but just pretend). Maybe Trump’s political persona is a role he’s playing. Willing injuring others by inciting racist violence. Scapegoating others for serious economic problems. Fomenting a social environment where racist actions are acceptable. Maybe that’s not racist. It’s still horrendous behavior.

Zoneminder and PHP 7.2

After updating to php 7.2, ZoneMinder completely stopped working. Fortunately there were lots of entries in the error_log file

[Fri Aug 10 15:44:19.880809 2018] [php7:error] [pid 5293] [client 127.0.0.1:46958] PHP Fatal error:  Cannot use 'Object' as class name as it is reserved in /usr/share/zoneminder/www        /api/lib/Cake/Core/Object.php on line 30
[Fri Aug 10 15:44:19.889737 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/lib/zoneminder/templogs/cake_error.log) [function.file-put-contents]: failed to open stream: No such file or directory in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on         line 142
[Fri Aug 10 15:44:19.889850 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/log/zonemindererror.log) [function.file-put-contents]: failed to open stream: Permission denied in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on line 142
[Fri Aug 10 15:44:19.890176 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/lib/zoneminder/templogs/cake_error.log) [function.file-put-contents]: failed to open stream: No such file or directory in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on         line 142
[Fri Aug 10 15:44:19.890221 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/log/zonemindererror.log) [function.file-put-contents]: failed to open stream: Permission denied in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on line 142
[Fri Aug 10 15:44:19.890594 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/lib/zoneminder/templogs/cake_error.log) [function.file-put-contents]: failed to open stream: No such file or directory in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on         line 142
[Fri Aug 10 15:44:19.890637 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/log/zonemindererror.log) [function.file-put-contents]: failed to open stream: Permission denied in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on line 142
[Fri Aug 10 15:44:19.890960 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/lib/zoneminder/templogs/cake_error.log) [function.file-put-contents]: failed to open stream: No such file or directory in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on         line 142
[Fri Aug 10 15:44:19.891021 2018] [php7:warn] [pid 5293] [client 127.0.0.1:46958] PHP Warning:  file_put_contents(/var/log/zonemindererror.log) [function.file-put-contents]: failed to open stream: Permission denied in /usr/share/zoneminder/www/api/lib/Cake/Log/Engine/FileLog.php on line 142
[Fri Aug 10 15:44:19.892818 2018] [php7:error] [pid 5293] [client 127.0.0.1:46958] PHP Fatal error:  Uncaught Error: Class 'Controller' not found in /usr/share/zoneminder/www/api/li        b/Cake/Error/ExceptionRenderer.php:174\nStack trace:\n#0 /usr/share/zoneminder/www/api/lib/Cake/Error/ExceptionRenderer.php(92): ExceptionRenderer->_getController(Object(InternalErr        orException))\n#1 /usr/share/zoneminder/www/api/lib/Cake/Error/ErrorHandler.php(126): ExceptionRenderer->__construct(Object(InternalErrorException))\n#2 /usr/share/zoneminder/www/ap        i/lib/Cake/Error/ErrorHandler.php(284): ErrorHandler::handleException(Object(InternalErrorException))\n#3 /usr/share/zoneminder/www/api/lib/Cake/Error/ErrorHandler.php(213): ErrorHa        ndler::handleFatalError(64, 'Cannot use 'Obj...', '/usr/share/zone...', 30)\n#4 /usr/share/zoneminder/www/api/lib/Cake/Core/App.php(970): ErrorHandler::handleError(64, 'Cannot use '        Obj...', '/usr/share/zone...', 30, Array)\n#5 /usr/share/zoneminder/www/api/lib/Cake/Core/App.php(943): App::_checkFatalError()\n#6 [internal function]: App::shutdown()\n#7 {main}\n          thrown in /usr/share/zoneminder/www/api/lib/Cake/Error/ExceptionRenderer.php on line 174

Looks like CakePHP used class names that are now reserved words. Unfortunately, you cannot just drop the updated CakePHP files into ZoneMinder (I tried). Until the repository package is updated, you’ve got to build ZoneMinder from source. Or grab the testing RPM from the zoneminder.com repo. 1.31.45-1.13 works. Don’t forget to run “perl /usr/bin/zmupdate.pl” to update the database.

Then I had to throw the database connection into from the config file into /usr/share/zoneminder/www/api/app/Config/database.php (default array) because I do not use the default connection info.

Once I had the updated ZoneMinder along with the newer CakePHP that the ZM folks have in their repo … we’ve got ZoneMinder again.

Sendmail: Giving everyone else a chance

We will occasionally get slammed with thousands of messages from a specific sender (usually one we’ve engaged to e-mail all of our employees, and of course they do it at 11 in the morning instead of some low volume off-hours time) which delays mail from all recipients. You can use the sendmail command line to flush the mail queue for messages other than those from a specific sender or other than those to a specific sender.

Using sendmail -qSdomain.gTLD will attempt to delivery messages where the sender matches domain.gTLD. Adding a ‘not’ in front of the ‘S’ attempts to deliver messages where the sender is not the specified domain.

sendmail -q\!Sbadguy.gTLD -v

That ensures all of the other mail is cleared through the queue even though ten thousand messages from a single sender still need to be processed. In real life, bypassing everything with the ‘mail from’ of @em-sj-77.mktomail.com is just

sendmail -q\!Sem-sj-77.mktomail.com -v

Shell Scripting: “File Exist” Test With Wildcards

Determining if a specific file exists within a shell script is straight-forward:

if [ -f filename.txt ]; then DoSomething; fi

The -f verifies that a regular file exists. You might want -s (exists and size is greater than zero), -w (exists and is writable), -e (a regular or special file exists), etc. But the test comes from the “CONDITIONAL EXPRESSIONS” section of the bash man page and is simply used in an if statement.

What if you don’t know the exact name of the file? Using the text “if [ -f *substring*.xtn ]” seems like it works. If there is no matching file, the condition evaluates to FALSE. If there is one matching file the condition evaluates to TRUE. But when there are multiple matching files, you get an error because there are too many parameters

[lisa@fc test]# ll
total 0
[lisa@fc test]# if [ -f *something*.txt ]; then echo "The file exists"; fi
[lisa@fc test]# touch 1something1.txt
[lisa@fc test]# if [ -f *something*.txt ]; then echo "The file exists"; fi
The file exists
[lisa@fc test]# touch 2something2.txt
[lisa@fc test]# if [ -f *something*.txt ]; then echo "The file exists"; fi
-bash: [: 1something1.txt: binary operator expected

Beyond throwing an error … we are not executing the code-block meant to be run when the condition is TRUE. In a shell script, execution will continue past the block as if the condition evaluated to FALSE (i.e. the script does not just abnormally end on the error, making the failure more obvious).

To test for the existence of possibly multiple files matching a pattern, we can evaluate the number of files returned from ls. I include 2>/dev/null to hide the error which will otherwise be displayed when there are zero matching files.

[lisa@fc test]# ll
total 0
[lisa@fc test]# if [ $(ls *something*.txt 2>/dev/null | wc -l) -gt 0 ]; then echo "Some matching files are found."; fi
[lisa@fc test]# touch 1something1.txt
[lisa@fc test]# if [ $(ls *something*.txt 2>/dev/null | wc -l) -gt 0 ]; then echo "Some matching files are found."; fi
Some matching files are found.
[lisa@fc test]# touch 2something2.txt
[lisa@fc test]# if [ $(ls *something*.txt 2>/dev/null | wc -l) -gt 0 ]; then echo "Some matching files are found."; fi
Some matching files are found.
[lisa@fc test]#

Now we have a test that evaluates to TRUE when there are one or more matching files in the path.

Facts Develop

Back when John Kerry was running for President, the “flip flopper” charge was a epitomized by his Marshall University statement “I actually did vote for the $87 billion, before I voted against it” regarding the Iraq war. Which quickly became “I was for the war before I was against it” — a charge that implies one’s beliefs have changed as a matter of political expediency.

It amazed me that no one ever responded to such criticism with “yes, given the information we were presented in 2003, I supported the war. With the new information that has come to light in the past year (or fifteen)? I believe the pretense was false, and as such have reconsidered my support. As president here’s what I would do to ensure intelligence is not tainted by politics.” Which (a) admits to being a person who is capable of analyzing new evidence and changing their opinion when new information invalidates previous “knowledge” and (b) focuses the question on something most people agree is a bad thing.

In the intervening decade, I’ve almost gotten to the point I’d be willing to vote for an individual who would just admit that they’ve changed their mind when new information was discovered over someone with whom I share ideological beliefs. Aaaaand then there’s Trump. Whose former lawyer, Jay Sekulow, announced that facts develop.

It’s an ironic turn of phrase, given my mental association with the innocent party (i.e. person who had no way to know otherwise) being able to discover new facts and changing their mind. Facts don’t develop when you are the person with the facts. What you tell everyone is developing … and somehow you are hoping we fear being labeled a flip-flopper enough to avoid changing our opinion as these new facts are revealed. You hope cognitive bias keeps our beliefs entrenched when hundreds of little revisions are made to “facts”.

Reinitializing The Exchange Content Index Database

When you search your inbox by copying a word from a message subject and searching for it by subject … but get nothing back, that’s a good indication that the content index database has gone corrupt. With Exchange 2013, you can manually reinitialize that database as follows:

Stop-Service MSExchangeFastSearch
Stop-Service HostControllerService

rename “C:\program files\microsoft\Exchange Server\V15\Mailbox\Mailbox Database 1440585757\1CDB1E55-A129-46BC-84EF-2DDAE27B808C12.1.Single” “c:\program files\microsoft\Exchange Server\V15\Mailbox\Mailbox Database 1440585757\1CDB1E55-A129-46BC-84EF-2DDAE27B808C12.1.Single.bad”

Start-Service MSExchangeFastSearch
Start-Service HostControllerService

# Wait a bit for the content indexing process to start
Get-MailboxDatabaseCopyStatus | FL Name,*Index*

ContentIndexState of “Crawling” means it’s still working on it. Healthy means it’s done.

Alternative Fact: The Darknet Market

Alternative Fact: From Trump @ one of this continual campaign rallies, this time in Florida: “You know, if you go out and you want to buy groceries, you need a picture on a card, you need ID.”

“The only time you don’t need it, in many cases, is when you want to vote for a president, when you want to vote for a senator, when you want to vote for a governor or a congressman. It’s crazy.”
Real Fact: Unless the grocery stores, home improvement centers, general merchandise retailers, and coffee shops around me have joined up to create a dark non-web to protest this crazy requirement to show ID before purchasing anything, there are limited situations in which one needs to present ID to make a purchase (hell, they don’t even require a signature for a 200$ grocery purchase anymore). And groceries are not one of those cases (barring, possibly, those using government support to make said purchase … but that’s fraud prevention {i.e. making sure SNAP recipient Fred is the one purchasing this stuff} and not an attempt to check out your awesome drivers license photo before you can purchase green beans. Maybe beer (verifying age restriction). Maybe smokes (again, age restriction). Guns, probably. Maybe even ammo. Occasionally for a high-value purchase. Frequently if you are asking for a discount — senior, student, military.
The worst part about his argument is that the same example could have been presented factually to support his desired outcome. Who cares who you are if you want to fork over some cash to purchase some oranges. Go right ahead. If you want the military discount, we need to see some ID that proves you are were in the military. If you want to use Fred’s SNAP benefits to purchase those oranges, you need to prove you are Fred. ID is required to ensure honesty. To prevent fraud. And shouldn’t we demand honesty in our elections? Shouldn’t we try to eliminate fraudulent voting?
Now I think the argument is a bunch of rubbish – unless the dude with an ID printing machine is going to set up shop at local employers and print out free IDs on people’s lunch breaks, unless the ID office is going to be open 24×7 so people don’t have to take a day off work to renew their free ID, unless they’re going to cruise around rural America printing IDs on farms … requiring an ID seems more like disqualifying eager voters than ensuring the sanctity of the election process.

Kubernetes – Using Manifest Files To Deploy

While I’ve manually configured Kubernetes to create proof-of-concept sandbox systems, that is an inefficient approach for production systems. The kubectl create command can use a yaml file (or json, if you prefer that format). I maintain my manifests in a Git repository so changes can be tracked (and so I know where to find them!). Once you have a manifest, it’s a matter of using “kubectl create -f manifest1.yaml” or “kubectl create -f manifest1.yaml -f manifest2.yaml … -f manifestN.yaml” to bring the ‘stuff’ online.

But what goes into a manifest file? The following file would deploy four replicas of the sendmail container that is retained in my private registry.

If the namespaces have not been created,

apiVersion: v1
kind: Namespace
metadata:
  name: sendmailtest

The command “kubectl get ns” will list namespaces on the cluster. Note that ‘deployment’ items were not available in the v1 API, so the v1beta1 version needs to be specified to configure a deployment.

---
apiVersion: v1
kind: Pod
metadata:
  name: sendmail
  namespace: sendmailtest
  labels:
    app: sendmail
spec:
  containers:
    - name: sendmail
      image: sendmail
      ports:
        - containerPort: 25
---
apiVersion: v1beta1
kind: Deployment
metadata:
  name: sendmail
spec:
  replicas: 4
  template:
    metadata:
      labels:
        app: sendmail
    spec:
      containers:
        - name: sendmail
          image: 'rushworthdockerregistry/sendmail/sendmail/sendmail:latest'
          ports:
            - containerPort: 25

---
apiVersion: v1
kind: Service
metadata:
  name: sendmail-service
  labels:
    name: sendmail-service
spec:
  ports:
    - port: 25
      targetPort: 25
      protocol: TCP
      name: smtp
  selector:
    app: sendmail
  type: ClusterIP

 

Microservice Adoption

I worry that companies are deconstructing their monolithic applications into microservices because it’s trendy. In fact, there are places where microservices don’t make sense but rather impart additional complexity to an application that is not enhanced by the benefits of microservices. While some challenges to microservice adoption are transient or can be addressed through business decisions … some are fundamental aspects of the architecture.

Microservices are (relatively) new. Whereas a company that has built and run many monolithic applications has network, hypervisor, OS, deployment, and application experts … unless the company hires in a container orchestration / API gateway expert or brings in a consulting team (real world experience has been “learning it” was left up to employee initiative and the global archive of IT knowledge that is the Internet), there isn’t a deep knowledge base to support the framework. Not an insurmountable problem, and frankly no different than how virtualization was introduced — there weren’t hypervisor experts at the time, no one really understood sizing/scaling intricacies. It was learned, but the first 6-12 months were rough. High availability applications were physically designed to withstand failure. Our data centre has two unique circuits run to each rack – and dual power-supply servers are plugged into both the “A” and “B” circuits. Same with network – there’s a team that goes through two different switches. In switching to VM’s … we had to identify where this server runs (i.e. what is it’s host)? Is every component of a redundant system co-located on a single hypervisor or in SAN-booted VMs are they stored on a single SAN frame? Microservices will have a similar challenge — where is it running, can the service as a whole survive a fault? How do we recover from a major data centre failure?

Some of the places where I see microservices making development and operations more complex can be eliminated by business policies. Allowing individual service teams to dictate their own development language can reduce mobility between teams — the Java guru for service A will spend time researching the c# equiv if they move over work on service B. And while it is possible to publish a general coding standard that covers all languages (how variables will be named, what comment blocks should look like, etc) there are nuances to each language that make a shared standard impossible. Using multiple development languages limits employee mobility, and it also reduces a company’s ability to shift employees around to cover temporary resource shortfalls. While planned absences can be accounted for when selecting work for the next cycle, emergencies happen.

Breaking an application into small component services can create challenges in troubleshooting issues. There may be few who have an end-to-end understanding of the application. Where the monolithic application X getting munged information means the development team for App X needs to debug and sort the issue … ten interacting microservices can mean ten groups saying nothing’s wrong on their side and it’s everyone else’s problem. I’ve seen that occur frequently in infrastructure support — app guys says it is the server, server guy says it’s the hypervisor, hypervisor guy says it’s the SAN, SAN guy says it’s all good and someone should check with the network guys to see how those load balancers are doing.

Fundamentally, microservice architecture introduces additional components to run the application — the API gateway and container orchestration are functions that simply don’t exist in a monolithic application. These services themselves, as well as the supporting technologies that allow these services to function, add additional complexity.

As an example, the networking configuration behind making microservices available are not, in my experience, something with which developers are familiar. This is not a problem when dev teams require out-of-box functionality and said functionality is working properly. I became involved with container orchestration system because a friend’s dev team encountered failures where kube-proxy did not create the required iptables rules — a quick and easy thing for a Linux/Unix admin to identify and troubleshoot, but not something that concerned application developers in monolithic deployments. Since then, the dev team sought to use multiple network interfaces and the Kubernetes CNI plugin did not support that feature.

For an application where individual components have different utilization rates, microservice architecture makes sense. Thinking about a company that runs a major promotion. There will (hopefully) be a flood of customers browsing the web site. The components that handle browsing and search functions need to grow significantly. The component that handles existing user authentication, new user registration, customer checkout, inventory update, and shipping quote generation components don’t need to scale at the same level — only a fraction of the web traffic will actually convert to sales. So there’s no need to spin up new hosts in the web farm to handle users browsing product information.

For an application where individual components require frequent updates, microservice architecture makes sense. Is there a component that suffers frequent failures where having a pool of microservices available would increase the application’s uptime?