Category: Technology

Programming in Unknown Languages

I’ve often thought that the immersion method of learning a language was setting yourself up for failure – it isn’t like knowing the fundamentals of grammar and pronunciation in English helps you in any way when you find yourself in Karnataka trying to communicate in Sanskrit. There are rather complex algorithms that attempt to derive meaning from an unknown language, but apart from body language, pointing, and gesturing … that’s not something I can manage in real-time as someone speaks to me.

*Programming* languages, on the other hand, I am finding are rather easily learnt by immersion. I know several programming languages quite well – C/C++, F77/F90, perl, and php. I know a dozen or so other languages well enough to get by.

Some of our home automation scripts are written in CoffeeScript (which is evidently a way to write JavaScript without *actually* knowing JavaScript) – and I would never be able to write the program. But to come into the middle of the conversation (i.e. to take someone else’s non-functional code and try to fix it), I can glean enough of the language to debug and fix code. And there’s always Google for any syntax I cannot guess.

I wonder if someone who is fluent in multiple disparate languages (knowing half a dozen Romance languages doesn’t really give you a good base of knowledge – I mean someone who speaks Italian, Hindi, Cantonese, Swahili, and some Levantine dialect of Arabic) is able to do something similar — they know enough words to pretty much guess what words mean & enough different language structures to guess words in their context.

Reverse Proxying WebSockets to An MQTT Server

If you are trying to reverse proxy OpenHab – that’s over here. This post is about maintaining your own private MQTT server and making it accessible through a reverse proxy.

We want to be able to update our presence automatically (without publishing our location information to the Internet). Scott found a program called OwnTracks that uses an MQTT server – and there’s an MQTT binding from OpenHab that should be able to read in the updates.

We didn’t want to publish our home automation server to the Internet, but we do want to send updates from the cellular data network when we leave home. To accomplish this, I set up a reverse proxy on our Apache server.

The first step is to get an MQTT server up and working — we Installed a mosquitto package from Fedora’s dnf repository

Once it is installed, create a directory for the persistence file & chown the folder to mosquitto uid

Generate a bunch of certs using the ot-tools (git clone https://github.com/owntracks/tools.git). I edited the generate-CA.sh file in the ot-tools/tools/TLS folder prior to running the script. It will more or less work as-is, but modifying the organisation names makes a cert with your name on it. Not that anyone will notice. Or care 🙂 Modifying the IPLIST and HOSTLIST, on the other hand, will get you a cert that actually matches your hostname — which isn’t a problem for something that doesn’t verify host name information, but saves trouble if you get your hostnames to match up.
IPLIST & HOSTLIST
CA_ORG and CA_DN

Then use generate-CA.sh to generate a CA cert & a server cert. Copy these files into /etc/mosquitto/

Edit the config (/etc/mosquitto/mosquitto.conf) – LMGTFY to find settings you want. Specify a location for the persistence file, password file, and add in the websockets listeners (& ssl certs for the secure one)
persistence_file /var/lib/mosquitto/mosquitto.db

password_file /etc/mosquitto/passwd

listener 9001
protocol websockets

listener 9002
protocol websockets
cafile /etc/mosquitto/ca.crt
certfile /etc/mosquitto/mosquittohost.rushworth.us.crt
keyfile /etc/mosquitto/mosquittohost.rushworth.us.key

Add some users
/usr/bin/mosquitto_passwd /etc/mosquitto/passwd WhateverUID

Start mosquitto
mosquitto -c /etc/mosquitto/mosquitto.conf

Monitor mosquitto for the owntracks ‘stuff’
mosquitto_sub -h mosquittohost.rushworth.us -p 1883 -v -t ‘owntracks/#’ -u WhateverUID -P PWDHereToo

Setting up the reverse proxy
The big sticking point I had was that the Apache WebSockets reverse proxy has a problem (https://bz.apache.org/bugzilla/show_bug.cgi?id=55320) which is marked as closed. Fedora has 2.4.23, so I expected it was sorted. However using tshark to capture the traffic showed that the relayed traffic is still being send as clear.

Downloaded the exact same rev from Apache’s web site and checked the mod_proxy_wstunnel.c file for the changes in the bug report and found they were indeed committed. In spite of the fact I *had* 2.4.23, I decided to build it and see if the mod_proxy_wstunnel.so was different.

Make sure you have all the devel libraries (openssl-devel for me … run the config line and it’ll tell you whatever else you need)

Get apr and apr-util from Apache & store to ./srclib then gunzip & untar them. Rename the version-specific folders to just apr and apr-util

Once you have everything, configure and make
./configure –prefix=/usr/local/apache –with-included-apr –enable-alias=shared –enable-authz_host=shared –enable-authz_user=shared –enable-deflate=shared –enable-negotiation=shared –enable-proxy=shared –enable-ssl=shared –enable-reqtimeout=shared –enable-status=shared –enable-auth_basic=shared –enable-dir=shared –enable-authn_file=shared –enable-autoindex=shared –enable-env=shared –enable-php5=shared –enable-authz_default=shared –enable-cgi=shared –enable-setenvif=shared –enable-authz_groupfile=shared –enable-mime=shared –enable-proxy_http=shared –enable-proxy_wstunnel=shared

Rename your mod_proxy_wstunnel.so to something like mod_proxy_wstunnel.so.bak and the grab mod_proxy_wstunnel.so that just got built.

Grab the CA public key & the server public and private keys that were generated earlier & place them whereever you store your SSL certs on your Apache server

Create a new site config for this reverse proxy – SSL doesn’t do host headers so you need a unique port. Clear text you can use a host header. Don’t forget to add listen’s to your httpd.conf and ssl.conf files!

ProxyRequests Off
<VirtualHost #.#.#.#:##>
ServerName mosquitto.rushworth.us
ServerAlias mosquitto
DocumentRoot “/var/www/vhtml/mosquitto”

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
SetEnv proxy-initial-not-pooled
SetEnv proxy-initial-not-pooled 1

ProxyPreserveHost On
ProxyTimeOut    1800

ProxyPass               /       ws://mosquittohost.rushworth.us:9001/
ProxyPassReverse        /       ws://mosquittohost.rushworth.us:9001/
</VirtualHost>

<VirtualHost #.#.#.#:##>
ServerName mosquitto.rushworth.us
ServerAlias mosquitto
DocumentRoot “/var/www/vhtml/mosquitto”

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
SetEnv proxy-initial-not-pooled
SetEnv proxy-initial-not-pooled 1

ProxyPreserveHost On
ProxyTimeOut    1800

SSLEngine On
SSLProxyEngine On
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLCertificateFile /etc/httpd/conf/ssl/mosquittohost.rushworth.us.crt        # These are the public and private key components
SSLCertificateKeyFile /etc/httpd/conf/ssl/mosquittohost.rushworth.us.key        #     generated from generate-CA.sh earlier.
SSLCertificateChainFile /etc/httpd/conf/ssl/ca.crt                # This is the public key of the CA generated by generate-CA.sh

ProxyPass               /       wss://mosquittohost.rushworth.us:9002/
ProxyPassReverse        /       wss://mosquittohost.rushworth.us:9002/
</VirtualHost>

Reload apache. Create a DNS hostname internally and externally to direct the hostname to your reverse proxy server.

Configure the client — generate a key for yourself & merge it into a p12 file (make sure your ca cert files are still in the directory – if you *moved* them into /etc/mosquitto … copy them back:
sh generate-CA.sh client lisa
openssl pkcs12 -export -in lisa.crt -inkey lisa.key -name “Lisa’s key” -out lisa.p12
You’ll need to supply a password for the p12 file.

Put the ca.crt (*public* key) file and your p12 file somewhere on your phone (or Google Drive).

Client config – Install Owntracks from Play Store
Preferences – Connection
Mode:    Private MQTT
Host:    hostname & port used in your **SSL** config. Select use WebSockets
Identification:    uid & password created above. Device ID is used as part of the MQTT path (i.e. my lisa device is /owntracks/userid/lisa). Tracker ID is within the data itself
Security:    Use TLS, CA certificate is the ca.crt created above. Client cert is the p12 file – you’ll need to enter the same password used to create the file

If it isn’t working, turn off TLS & change the port to your clear text port. This will allow you to isolate an SSL-specific problem or a more general service issue. Once you know everything is working, you can drop the clear text reverse proxy component.

Voila – reverse proxied WebSockets over to Mosquitto for OwnTracks.

Linux Primer

We’ve got a few new people at work who don’t have any Linux experience, and I was asked to do a quick crash course on some super fundamental logging in / navigating / restarting service stuff so their first on call rotation wouldn’t be quite so stressful. Publishing the overview here in case it is useful for anyone else.

Linux Primer:

Connecting – We use both putty and Cygwin to connect to our Linux hosts via SSH (secure socket shell). Each has its own advantages and disadvantages – try them both and see which you prefer. If you need X redirection (you need the GUI ‘stuff’ to magic itself onto your computer), use Cygwin-X.

Logging In – Our Linux hosts authenticate users via cusoldap.windstream.com, so (assuming you are set up for access to the specific host) you will use your CSO userID and password to log in.

  • We often use a jump box – log into the jump box with your ID or using a key exchange. From there, we have key exchanges with our other boxes that allow us to connect without entering credentials again.
  • You can set up key exchanges on your own ID too – even from your Windows desktop – and avoid typing passwords.

Once you are logged in, you can start a screen session. Normally, anything you are running is terminated if your SSH session terminates (e.g. if you use Cygwin or Putty to connect to a box from your laptop that is VPN’d into the network & your VPN drops … everything you were doing in the SSH session is terminated.). You can use screen to set up a persistent session – you can reconnect to that session should your SSH connection get interrupted, other people can connect to the session to monitor a long running script, or multiple people can connect to the session and all see the same thing (screen sharing).

To start a new screen session, screen -S SessionName where SessionName is something that identifies the screen session as yours (e.g. LJRPasswordResync was the session I used when resyncing all employee and contractor passwords for OIDM – this includes both my initials and the function I’m running in the session). To see the currently running sessions, use screen –ls

[lisa@server810 ~]# screen -ls

There is a screen on:

8210.LJR        (Detached)

1 Socket in /tmp/screens/S-lisa.

The output contains both a session ID number (green) and a session name (blue) separated by a full stop. You can use either to connect to a screen session (the name is case sensitive!). To reconnect, use screen –x SessionName or screen –x SessionID

To determine if you are currently in a screen session, look at the upper left hand corner of your Putty window. The title will change to include screen when you are in a screen session. Alternately echo the STY environment variable. If you get nothing, it is not a screen session. If you get output, it is the PID and name of your current screen session.

[lisa@server810 ~]# echo $STY
43116.LJR

SUDO – The sudo command lets you execute commands that your ID is not normally privileged to run. There is configuration to sudo (maintained by ITSecurity) that defines what you can run through sudo. If, for example, you are unable to edit a file but are permitted to sudo vim … editing a file using “vi /path/to/file.xtn” will throw an error if you attempt to save changes, but running “sudo vi /path/to/file.xtn” would allow you to save changes to the file.

Substitute user – The command su lets you substitute a uidnumber for yours – this means you become that user.

Combining SUDO and SU – Once we are logged into LX810 with our user ID, we can use sudo su – root to become root without actually knowing the root password. The “space dash space” in the su command means the user’s environment is loaded. If you omit the space dash space, you’ll still be logged in as the root user, but your environment will be left in place.

Generally speaking, allowing sudo to root is a bad idea (i.e. don’t do this even though you’ll see it on a lot of our old servers). This is because root has full access to everything and running the shell as root is insecure and typos can be disastrous.

Navigating – You are in a DOS-like command line interface. The interface is known as a shell – root on LX810 is a bash shell. The default for a CUSO ID is the korn shell (/bin/ksh) – you can change your shell in your LDAP account to /bin/bash (or /bin/csh for the C shell) and subsequent logons will use the new shell. You can try each one and see which you prefer, you can use korn because it is the default from CUSO, or you can use bash because it matches the instructions I write.

From a file system navigation perspective, you will be in the logon user’s home directory. If you aren’t sure where you are in the file system, type pwd and the present working directory will be output.

To see what is in a directory, use ls … there are additional parameters you can pass (in Linux parameters are passed with a dash or two dashes). Adding -a lists *all* files (including the hidden ones, any file where the name starts with a full stop is a hidden file). Adding -l provides a long listing (file owners, sizes, modified dates). Adding -h lists file sizes in human readable format. You can pass each parameter separately (ls –a –l –h) or by concatenating them together (ls –alh)

You can use wc to count the number of lines either in a file (wc –l /path/to/file.xtn) or the output of ls (ls –al | wc –l) – this is useful on our sendmail servers when you have received a queue length alert and done something to clear out some of the queue. In sendmail particularly, there are two files for each message so you need to divide the line count by 2.

To change to a different directory, use cd – e.g. cd /etc/mail will change the working directory to /etc/mail.

To delete a file, use rm /path/to/file.xtn – this is the safe way to run it, it will prompt for confirmation for each file being deleted. You can use wildcards (rm /path/to/files*) to delete multiple files. You can add a -f parameter to not be prompted – which is more dangerous as you may have typed the wrong thing and it’ll be deleted without prompting. You can add a –r parameter for recursive (get rid of everything under a path). Not too dangerous as long as you have the prompt coming up – but if you use –r in conjunction with –f (rm –rf) … you can do a lot of damage. Absolute worst case would be recursive force delete from / … which would mean every file on disk goes away. Don’t do that J

If you are not sure where a file you need is located, you can use either find or locate. The locate command is not always installed, so you would need to use the find command. Locate uses an index database – so it’s quicker, but it doesn’t know about files created/deleted since the index was updated.

To use locate, use locate -i filename where filename is some part of the filename. The -i performs a case insensitive search – if you know the proper casing, you do not need to include this parameter.

To use find, you need to indicate the root of the search (if you have no clue, use ‘/’ which is the top level directory) as well as the exact file name that you want (not a substring of the file name like locate will let you do). Finding a file named audit.log that is somewhere on the disk would be find / -name audit.log

Customizing shell environment – You can customize your shell environment. The system-wide shell environment settings are in /etc and are specific to the shell. For a bash shell, it is /etc/bashrc

Individual user settings are in a hidden file within their home directory. For the bash shell, the user specific settings are in $HOME/.bashrc ($HOME is a variable for the current logon user’s home directory).

For a shared account, adding things to $HOME/.bashrc isn’t the best idea – your preferred settings can differ from someone else’s preferences. We make our own rc file in $HOME for the shared account (I actually set my .bashrc as world-readable and linked the shared ID $HOME/.ljlrc to my personal .bashrc file so I only have to remember to edit one file). You can load your personal preferences using source $HOME/.yourrc or you can load someone else’s preferences by sourcing their file in the shared account’s home directory (source $HOME/.ljlrc will load in mine).

Service Control – Most of our Linux systems still use systemd (init.d scripts) to start and stop services. You can find the scripts in /etc/init.d – these are readable text scripts. All scripts will have a start and stop command, and many have restart and status as additional commands. To control a service, you can use service servicename command, /sbin/service servicename command or /etc/init.d/servicename command – same thing either way. If you are controlling the service through sudo, though, you need to use the technique that is permitted to your UID in the sudo configuration.

If you use a command that isn’t implemented in the script, you will get usage information. You can use a semicolon to chain commands (like the & operator in DOS) – so /etc/init.d/sendmail restart is the same thing as running /etc/init.d/sendmail stop;/etc/init.d/sendmail start

Process utilization – To see what the processor and memory utilization is like on a box (as well as which processes are causing this utilization), use top. When top has launched, the first few lines give you the overall usage. The load average (blue below) tells you the load during the last one, five, and fifteen minutes – 1.00 is 100% on a single core system, 2.00 is 100% on a two core system, etc. Over the 100% number for a system means stuff got queued waiting for CPU cycles to become available.

The current CPU utilization (green below) breaks out usage by user tasks, system tasks, nice’d processes (generally nothing here), idle, io wait, hardware irq, software irq.

The memory usage (red below) shows used and free memory.

top – 13:58:30 up 486 days,  2:16,  9 users,  load average: 0.34, 0.24, 0.25

Tasks: 162 total,   1 running, 161 sleeping,   0 stopped,   0 zombie

Cpu(s):  0.4% us,  0.1% sy,  0.0% ni, 99.5% id,  0.0% wa,  0.0% hi,  0.0% si

Mem:   4147208k total,  2107876k used,  2039332k free,    62372k buffers

Swap:  2064376k total,     1352k used,  2063024k free,  1167652k cached

 

The process list can be sorted by whatever you need – if the box is CPU-bound, type an upper case C to sort by CPU usage. If it is memory bound, type an upper case M to sort by memory usage.

PID USER      PR  NI %CPU    TIME+  %MEM  VIRT  RES  SHR S COMMAND

23190 root      15   0    1   5:43.81 14.9  608m 605m 2872 S perl

14225 root      16   0    0   7:14.20  1.7  170m  69m  60m S cvd

14226 root      16   0    0   1:30.32  1.4  147m  57m  50m S EvMgrC

4585 root      16   0    0 212:01.99  1.1  230m  43m 6368 S dsm_om_connsvc3

4003 root      16   0    0   2729:44  0.6  171m  24m 3364 S dsm_sa_datamgr3

24552 root      16   0   13   0:36.16  0.3 17804  12m 2900 S perl

 

The first column shows the PID (process ID). Some commands as listed in top are obvious what they actually are (httpd is the apache web server, for instance) and others aren’t (perl, above, doesn’t really tell us *what* is using the CPU). To determine what the PID actually is, use ps –efww | grep PID#

[lisa@server810 Sendmail-CheckQSize]# ps -efww | grep 23190

root     23190 23187  0 01:23 ?        00:05:44 /usr/bin/perl /home/NDSSupport/Scripts/osrOCSProvisioning/_syncIMEnabledFromCSO.pl

root     24645 16640  0 14:10 pts/10   00:00:00 grep 23190

 

You will see the full command that is running – in this case a particular perl script. Note that you may also find your grep command in the list … depends a bit on timing if it shows up or not.

You may need to restart a service to clear something that has a memory leak. You may need to stop the process outside of the service control (e.g. stopping the sendmail service doesn’t shut down all current threads). To stop a process, use kill PID# … this is basically asking a process nicely to stop. It will clean up its resources and shut down cleanly. use ps –efww to see if the process is still running. If it still is, use kill -9 PID# which is not asking nicely. Most things to which a process is connected will clean up their own resources after some period of client inactivity (i.e. you aren’t causing a huge number of problems for someone else by doing this) but it is cleaner to use kill without the “do it NOW!!!” option first.

Tail and Grep – Tail is a command that outputs the last n lines of a file. It has a parameter that outputs new lines as they get appended to the file. On *n?x systems, you can use tail –F /path/to/file.xtn and lines will be output as they show up. This is particularly useful on log files where the system is continually adding new info at the bottom of the file. We put Windows ports of these utilities on our Windows servers – but the Windows port of tail does not support –F (there’s a good reason that has to do with the difference between Unix-like and Windows file systems). You can use tail –f instead – if the log file rolls (gets moved to another file and a new file is started) you won’t continue to get output like you will with –F … but you can ctrl-c to terminate the tail & start it again to start seeing the new lines again.

Grep is a command line search program. You can use grep to find lines in a file containing a string (or regex pattern, but learning regex is a question for LMGTFY.com) – to find all of the mail addressed to or from me in a sendmail log, grep –i rushworth /var/log/maillog – the dash i means case insensitive search.

Grep will also search piped input instead of a file – this means you can send the output of tail to grep and display only the lines matching the pattern for which you search.

tail -f /var/log/maillog | grep –i rushworth will output new lines of the maillog as they come in, but only display the ones with my name.

VIM – The non-visual text editor is vim – which is usually invoked using ‘vi’, but vi is an actual program that is like but not exactly the same as vim (vim is improved vi). The vim installation contains a very nice tutorial – invoked by running vimtutor

VIM has both a command mode and an editing mode. When in command mode, different keys on the keyboard have different functions. There are “quick reference” guides and “cheat sheets” online for vim – most people I know have a quick ref guide or cheat sheet taped next to their computer for quite some time before vim commands become well known.

History – Linux maintains a history of commands run in a session. This spans logons (you’ll see commands run last week even through you’ve logged on and off six times between then) but when there are multiple sessions for the same user, there can be multiple history files. Which is all a way of saying you may not see something you expect to see, or you may see things you don’t expect. The output of history shows the command history for the current logon session. You can pipe the output to grep and find commands in the history – for example, if you don’t remember how to start a service, you can use history | grep start and get all commands that contain the string start

[lisa@server855 ~]# history | grep start

7  service ibmslapd start

15  service ibmslapd restart

42  service ibmslapd start

56  service ibmslapd restart

71  service ibmslapd start

95  service ibmslapd start

107  service ibmslapd start

115  service ibmslapd restart

289  service ibmslapd start

303  service ibmslapd start

408  service ibmslapd start

419  service ibmslapd start

430  service ibmslapd start

443  service ibmslapd start

If a command fails, it will still be in the history (all of my typo’s are in there!), but if you see the same command a number of times … it’s probably correct. You can copy/paste the command if you need to edit it to run (or even to run it as-is). You can run the exact command again by typing bang followed by the line number of the history output (!115 with the history above would re-run “service ibmslapd restart”).

Symbolic Links

Linux symbolic links are nothing like Windows shortcuts, although I see people saying that. Shortcuts are independent files that contain a path to the referenced file. Linux sym links are just pointers to the inode for the file. They are the file, just allowing it to be used in a different location. This is a bit like memory addressing in programming — anything that reads from the memory address will get the same data, and anything that writes to the memory address. When you do a long list (ls -al or just ll), you will see both the file name and the file to which it points:

lrwxrwxrwx 1 root root 19 Aug 17 13:54 ljrtest -> /tmp/dnsexit-ip.txt

The “l” at the start of the line indicates that it is a link.

Text-to-speech (TTS) On Fedora

Scott has been setting up our OpenHAB server, and the latest project was controlling our network speakers. You can play Internet radio stations to the speakers, you can stream music from the NAS … but we also want to be able to play announcements. For that, we needed a text to speech engine.

Festival is in Fedora’s yum repository, but everything I’ve read about Festival says the output is robotic. Which is likely fun at first, but tiring after the first three or four times. Even if you have it say “beep, boop” at the end.

SVox (Nuance, which a long LONG time ago was spun off from Stanford Research Labs) has an open-source version of their text to speech product. Not in convenient package form, but close. Someone maintains a shell install script. Download the script:

wget https://raw.githubusercontent.com/stevenmirabito/asterisk-picotts/master/picotts-install.sh

Then read through it and make sure it’s not doing anything untoward. It wasn’t. Ran the script and a minute later, we can use “pico2wave -w /tmp/ljr.wav “I am your TTS engine”

Quick. Easy. And now we’ve got a wave file to send to the speaker (and remove when we’re done!)

Using BC And Command Substitution In OpenHAB’s Exec Binding

My husband has been setting up OpenHAB to control our home automation. Our dimmers are very direct – there’s a z-Wave binding that you set to 100 if you want it at 100%, set it to 18 if you want it at 18%, and so on. We have a handful of Zigbee bulbs, though, which are not so direct. We are controlling these bulbs through a Wink hub by running a curl command with the exec binding.

The OpenHAB exec binding runs a shell with a command string passed in from the -c parameter. Thus far, I have not found anything that runs within a shell not work in the exec binding. This includes command substitution {I personally use the backtick format instead of the $(command) format, but I expect the later to be equally functional}.

What is command substitution (without having to read the Open Group Base Specifications linked above)? If you run

kill `pidof java`

the shell takes the component within the backticks, evaluates it, and then takes the standard output and places that into the command. When “pidof java” returns “938 984 1038”, the command above becomes “kill 938 984 1038”.

We want to set the value to the OpenHab value (0-100) scaled to the Wink value (0-255 for GE Link bulbs) using command substitution with bc (an arbitrary precision calculator language). To evaluate a mathematical expression, echo the expression text and pipe it to bc. To set a bulb to 75% of its maximum brightness, our post data is “nodeId=a&attrId=aprontest -u -m9 -t2 -v`echo 2.55*75/1|bc`”.

Notice the divide by 1 at the end — that’s to turn a decimal value into an integer. If you use just 2.55*75, you post a value of 191.25 which throws an error. In bc’s language, / returns the quotient — this isn’t *rounding* but rather truncating the decimal portion( i.e. bc 9.99999/1 = 9).

We configure the OpenHAB item to take the selected value (the %2$s below), scale the value with bc, and insert the result into the curl command. We use a similar technique to read the data from Wink and present the scaled value through OpenHAB.

The item entry in our sitemap.items file:

Dimmer  DS_Pantry_Bulb_Level                                            “Bulb (Pantry Downstairs) [%d]”                                 <slider>        (gZigbeeBulb,gDS_Pantry,gLight)                                                                                 { exec=”<[/bin/sh@@-c@@echo `/usr/bin/curl \”http://wink.hub.address/set_dev_value.php\” -s -d \”nodeId=a&attrId=aprontest -l -m9;\”|grep Level|grep -oP \”\\d+\\D+\\K\\d+\\D+\\K\\d+\”` /2.55|bc:3600000:] >[*:/bin/sh@@-c@@/usr/bin/curl \”http://wink.hub.address/set_dev_value.php\” -s -d \”nodeId=a&attrId=aprontest -u -m9 -t2 -v`echo 2.55*%2$s/1|bc`;\”]”}

Parsing JSON In JavaScript

We’ve been trying to get our BloomSky data parsed and reflected in OpenHAB — we can automatically turn the lights on when there is motion *and* the luminescence is lower than some desired value.  Bloomsky has an API which allows us to retrieve JSON formatted data from our weather station. I never worked with JSON before – I’d heard the term, but didn’t actually know what it was … but I needed to parse it in a JavaScript transform. Does JavaScript do JSON? D’oh! Turns out JSON is an abbreviation for JavaScript Object Notation, and JavaScript parses JSON data really well.

Still need to turn my example web code into a transform that runs from OpenHAB, but getting values out of a JSON formatted string is as easy as using the “parse” function:

<html>
	  <head>
	    <script>
	      function parseMyData() {
		var input = '{"DeviceID":"83237E","LAT":41.226644299999997,"LON":-81.7224322,"ALT":292.78720092773438,"UTC":-4,"DST":1,"Searchable":true,"RegisterTime":1464494138,"CityName":"Hinckley","StreetName":"Bellus Road","FullAddress":"Bellus Road, Hinckley, Ohio, US","DeviceName":"Buzzard Cam 01","BoundedPoint":null,"NumOfFollowers":5,"Data":{"Temperature":80.528000000000006,"ImageURL":"http://storage.googleapis.com/bloomsky-img/eaB1rJytnZSmm5y3qJ1krJqwmJmtoJU=.jpg","Humidity":50,"Night":false,"ImageTS":1465938980,"Luminance":3445,"TS":1465938980,"Rain":false,"Pressure":29.087148500000001,"Voltage":2613,"UVIndex":"1"},"Point":{},"VideoList":["http://storage.googleapis.com/bloomsky-video/eaB1rJytnZSmm5y3_-4_2016-06-09.mp4","http://storage.googleapis.com/bloomsky-video/eaB1rJytnZSmm5y3_-4_2016-06-10.mp4","http://storage.googleapis.com/bloomsky-video/eaB1rJytnZSmm5y3_-4_2016-06-11.mp4","http://storage.googleapis.com/bloomsky-video/eaB1rJytnZSmm5y3_-4_2016-06-12.mp4","http://storage.googleapis.com/bloomsky-video/eaB1rJytnZSmm5y3_-4_2016-06-13.mp4"],"NumOfFavorites":0}'

		var jsonOfInput = JSON.parse(input);

		document.write("<P>Device ID is: " + jsonOfInput.DeviceID + "</P>");
		document.write("<P>Temp is: " + jsonOfInput.Data.Temperature + "</P>");
		document.write("<P>Luminance is: " + jsonOfInput.Data.Luminance + "</P>");
	      }
	    </script>
	  </head>
	  <body>
	  <h2>Press the button to start</h2>
	    <input type="button" onclick="parseMyData()" value="Parse"/>
	  </body>
	</html>

Serving Custom Error Pages From Apache

At work, we are in the process of retiring an old password management web site. We want to direct users to the new site, and I don’t particularly want to handle each possible entry point an individual may have bookmarked. It seemed a lot quicker and easier to just move everything out of the directory and throw up a custom 404 page.

I am certain that I’ve used just “ErrorDocument ### /file.xtn” in Apache configurations to serve custom error pages, but when I set this up in our staging environment … I got the generic 404. Three days of Googling and reading Apache documentation later, and I have a configuration that actually serves a custom page when error 404 is encountered:

        ErrorDocument 404 /customized-404.html
        <Files "customized-404.html">
        <If "-z %{ENV:REDIRECT_STATUS}">
            RedirectMatch 404 ^/customized-404.html$

Voila, a pretty page that doesn’t in any way indicate 404 / not found / etc but rather says “hey, this web site is being retired. please go over yonder to manage your password.”.

Tracking Electrical Usage With SmartThings and AeonLabs Home Energy Meters

When we started shopping for solar generation installations, how much electricity we can consume was a challenging question. We were replacing our HVAC system, so “look at the last 12 months of electric bills” wasn’t an approach that would yield valid data. What we needed was a way to see electrical consumption for the next two or three weeks once our new HVAC system was installed.

We purchased several AeonLabs Home Energy Meters (HEM) and have been using them to track our power consumption for almost six months now. The HEM’s are set up in SmartThings & have a SmartApp-HEMLogger “SmartApp” attached to them that posts data to a MySQL table on our server via a web form (myURL needs to be … well, your URL).

Install a quick MySQL server (does not need to be Internet accessible) and a web server / programming language of your choice combination (*does* need to be Internet accessible – we are using Apache and PHP).

Create the database and a table to hold your energy data:

mysql> describe EnergyMonitors;
+-----------+-------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+-------------------+----------------+
| keyID | int(11) | NO | PRI | NULL | auto_increment |
| monitorID | varchar(50) | YES | | NULL | |
| clampID | varchar(50) | YES | | NULL | |
| eventTime | timestamp | NO | | CURRENT_TIMESTAMP | |
| kwatts | double | YES | | NULL | |
| kwhours | double | YES | | NULL | |
+-----------+-------------+------+-----+-------------------+----------------+
6 rows in set (0.00 sec)

Create an ID within your database that has read/write permission to this table. I create another ID that has read-only access (pages displaying data use this ID, the page to post data uses the read/write ID).

We also track temperature — ideally, we’d be able to compare power consumption based on temperature *and* sunlight (we use a lot less power on a cold sunny day than I expect … just don’t know how much less) … but I’m not there yet. Currently, the weather database only holds temperature:

mysql> describe weather;
+--------------+-----------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------+------+-----+-------------------+-----------------------------+
| recordedTime | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| temperature | int(11) | YES | | NULL | |
| id | int(11) | NO | PRI | NULL | auto_increment |
+--------------+-----------+------+-----+-------------------+-----------------------------+
3 rows in set (0.01 sec)

Since we brought our Bloomsky online, I use the Bloomsky API to record temperature. Prior to that, I was parsing the XML from NWS’s closest reporting station’s – for Cleveland, that is  http://w1.weather.gov/xml/current_obs/KCLE.xml … there’s probably one near you. I grabbed both observation_time_rfc822 and temp_f in case observations were not updated regularly – the observation time was stored as the recordedTime. The temperature recording script is in cron as an hourly task.

You also need a small web page where SmartThings can post data — in PHP, I have

<?php
if(!$_POST["monitorID"] || !$_POST["clampID"] ){
echo "<html><body>\n";
echo "<form action=\"";
echo $_SERVER['PHP_SELF'];
echo "\" method=\"post\">\n";
echo "Monitor ID: <input type=\"text\" name=\"monitorID\"><br>\n";
echo "Clamp ID: <input type=\"text\" name=\"clampID\"><br>\n";
echo "KW Value: <input type=\"text\" name=\"kwatts\"><br>\n";
echo "KWH Value: <input type=\"text\" name=\"kWHours\"><br>\n";
echo "<input type=\"submit\">\n";
echo "</form>\n";
echo "</body></html>\n";
}
else{
$strMeter = $_POST["monitorID"];
$strClamp = $_POST["clampID"];
$strKWValue = $_POST["kwatts"];
$strKWHValue = $_POST["kWHours"];
print "<pre>Meter: $strMeter\nClamp: $strClamp\nKW: $strKWValue\nKWH: $strKWHValue\n</pre>\n";
$servername = "DatabaseHostname";
$username = "MySQLUID";
$password = 'MySQLPassword';
$dbname = "HomeAutomation";
$conn = new mysqli($servername, $username, $password, $dbname);
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
if($strKWHValue && $$strKWValue){
$sql = "INSERT INTO EnergyMonitors (monitorID, clampID, kwatts, kwhours) VALUES ('$strMeter', '$strClamp', '$strKWValue', '$strKWHValue')";
}
elseif($strKWHValue){
$sql = "INSERT INTO EnergyMonitors (monitorID, clampID, kwhours) VALUES ('$strMeter', '$strClamp', '$strKWHValue')";
}
else{
$sql = "INSERT INTO EnergyMonitors (monitorID, clampID, kwatts) VALUES ('$strMeter', '$strClamp', '$strKWValue')";
}
if ($conn->query($sql) === TRUE) {
echo "New record created successfully";
}
else {
echo "Error: " . $sql . "<br>" . $conn->error;
}

$conn->close();
}
?>

DatabaseHostname, MySQLUID and MySQLPassword need to be yours too. Since posted data is meant to come from a trusted source (a source where I’ve supplied the URL) and in a known good format, these are quick INSERT statements *not* the safest.

Check in your database that you are getting data, then let it run for a few hours. Once you have a little bit of history, you can start viewing your energy usage. Link it into Excel/Access using MyODBC for ad-hoc reporting, write your own code to do exactly what you want, use a generic charting package capable of reading MySQL data …

I am using pChart to display data – the chart I use most frequently is the stacked bar chart for kWH and a line chart for temperature. Here is the PHP code which generates this PNG: energyUsage-StackedBarChart-Flat – again, DatabaseHostname, MySQLUID, and MySQLPassword need to be yours.

StackedBarChart-KWHAndTemp

We no longer have our heat pump, air handler, and heat strips being monitored – but for periods where there is data from the other sources, we had several segments to our energy usage report (“other” is the report from the HEM on our mains MINUS all of the other reporting segments). You can yank all of the non-mains segments (or change them to be whatever sub-segments you are monitoring. The monitor ID comes from the HEM name in SmartThings – since those are user-configured, I have the names hard coded. You *could* hard code the Mains and then use “select distinct” to get a list of all the others and make the code more flexible.).

Short term charts (past 24 hours or so) can render out real-time, but longer term views take a long time to load. Since we’re looking more for trends and totals – being up to the second isn’t critical. I’ve got cron tasks that generate out PNG files of the charts.

Two enhancements for a nothing-doing rainy day – adding authentication to the HEM Logger (SmartThings posts from multiple netblocks, so there isn’t a good way to IP source restrict access to the post data page. Anyone with your URL could post rogue energy usage info into your database – or more likely try to hack your servers.) and add lumen to the weather data / graph displays.

Kerberos Authentication and LDAP Authorization In Apache

I’ve been authenticating users of Apache web sites against Active Directory using Kerberos for some time now. Installed krb5-workstation and mod_auth_kerb, configured /etc/krb5.conf for my specific domain, and added some config to the Directory section of the Apache config. Great if you just require valid-user (or require valid-user and then turn around and do some authorization within your web code using something like php_auth_user). Not so great, though, for restricting access to the site outside of web code. And I really didn’t want to code in an authorization function when my web server should be able to do that for me.

I FINALLY got kerberos authentication working in Apache with an LDAP authorization component. Turns out the  mod_auth_kerb version 5.1 that was available from the Yum repository is terribly buggy  – like not usable in this instance buggy. KrbLocalUserMapping did not consistently remove the realm component. I’d hit a site and it would know who I am, click a link and come across as me@REALM.TLD and get access denied errors, click refresh and get in because it knew I was me again. Or not. More than 50% failure rate.I built the 5.4 version from http://modauthkerb.sourceforge.net/ and haven’t had a problem since.

I’m authenticating to Active Directory using the Kerberos module then authorizing against a group housed in an external LDAP directory. You can totally point your LDAP config toward Active Directory & use AD groups instead:

AuthType Kerberos
AuthName “Kerberos AD Test”
KrbAuthoritative off
KrbMethodNegotiate on
KrbMethodK5Passwd on
KrbServiceName HTTP/this.isyour.url.tld@EXAMPLE.COM
KrbAuthRealms EXAMPLE.COM
KrbLocalUserMapping On
Krb5Keytab /path/to/keytabs/keytab.file

AuthBasicAuthoritative On
AuthBasicProvider ldap
AuthLDAPURL “ldaps://ldap.example.com/o=BaseDN?uid?sub?(&(cn=*))”
AuthLDAPBindDN “YOUR SERVICE ACCOUNT HERE”
AuthLDAPBindPassword “YOUR BIND PWD HERE”

AuthLDAPGroupAttribute uniqueMember
AuthLDAPGroupAttributeIsDN on
require ldap-group cn=Website Test,ou=groups,o=BaseDN

 

WooHoo! I hit the site from my domain-member computer, it knows I am LisaR. It then turns around and finds an LDAP user matching uid=LisaR and grabs the user’s fully qualified DN (because AuthLDAPGroupAttributesIsDN is ‘on’ here … if you are using just uids in your member list, that would be off). It then verifies that the fully qualified DN is a member of the Website Test group.

Now I’m trying to figure out how to let the user log in without supplying a realm (not everyone’s in the domain … and they need to be able to log in too. Works fine right now, provided they input their username as uid@REALM.TLD).

Same word, different meaning

I issue a lot of certificates from our internal company certificate authority – they’re free, and since I can publish trusted root signers out to the domain, they’re trusted to anyone who would be using the site. You can type pseudo-random values into your request and my CA will issue a certificate for you.
Today, though, I needed a certificate for a site that would be used by non-employees. People who are not subject to my domain GPO. People who do not trust my CA. So I did what everyone else does – got a real certificate 🙂
I generated my CSR (and actually typed in good data – my server is in Conway, AR, USA type location instead of Z or A). Went out to Verisign’s site … “The CSR contains an invalid state. Please click your browser’s Back button and enter a new CSR.”
Thought the CSR might have gotten corrupted somehow, so I tried again. Same result. Tried some different information – same result. Finally resorted to reading the instructions – locality names may not contain abbreviations. D’oh. State like organized political community not state like condition.