Category: Coding

Extracting Waste Stream Collection Dates for the Netherlands

Yeah … mostly saving this for the regex search with a start and end flag that spans newlines because I don’t really need to know the date they collect each waste stream in the Netherlands. Although it’s cool that they’ve got five different waste streams to collect.

import requests
import re

strBaseURL = '<some component of your address in the Netherlands>' 
iTimeout = 600 
strHeader = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'}

# Start and end flags for waste stream collection schedule content
START = '<ul id="ophaaldata" class="line">' 
END = '</ul>' 

page = requests.get(strBaseURL, timeout=iTimeout, headers=strHeader)
strContent = page.content
strContent = strContent.decode("utf-8")

result ='{}(.*?){}'.format(START, END), strContent, re.DOTALL)
strCollectionDateSource =

resultWasteStreamData = re.findall('<li>(.*?)</li>', strCollectionDateSource, re.DOTALL)
for strWasteStreamRecord in resultWasteStreamData:
    listWasteStreamRecord = strWasteStreamRecord.split("\n")
    strDate = listWasteStreamRecord[3]
    strWasteType = listWasteStreamRecord[4]
    print("On {}, they collect {}".format(strDate.strip().replace('<i class="date">','').replace('</i>',''), strWasteType.strip().replace('<i>','').replace('</i>','')))

Adding CSS To Header

I am currently working on a website that sources in a header and footer — not an uncommon thing to do as this ensures a consistent look across the site. The lead-in code starts head, closes head, starts body, and defines the common page elements (nav bar, etc). The footer then defines some more common page elements and closes body. This approach creates a problem when you want to add CSS. Now you could use style tags within the HTML, but I would rather not have the same style definition twenty times. Yeah, I’d make a single variable out of it and print the style-definition-variable twenty times … but I’d rather have my CSS sourced in from a style-sheet file.

Since I’m already using jQuery to dynamically append elements — add table rows as data is pulled back from the server — I wondered if you could append something to the header. Yes, you can!

* This function appends a CSS file to the document head
* @param {string} strFileName Path to CSS file
* @return n/a
* @example
* loadCSSStylesheetToHead('/path/to/file.css')
function loadCSSStylesheetToHead(strFileName){
var file = document.createElement("link");
file.setAttribute("rel", "stylesheet");
file.setAttribute("type", "text/css");
file.setAttribute("href", strFileName);

This allows me to after-the-fact add css from a style-sheet file into the document head.

Git: Using Soft Reset To Clean Up Un-pushed Commits

I missed a file when I was cleaning up debugging lines. I made the change and included it in a second commit, but I’d rather not have two commits for the same purpose. I hadn’t pushed my changes yet, so these commits only exist on my workstation … which means I can reset and bundle the changes into a single commit.

Find commit number that is one before the duplicate debug logging cleanup — this is the point to which you want to reset. In my case, it is the commit start with b443348c

Reset there with “–soft” — this doesn’t change anything on the file system (i.e. I don’t have to clean up those debug lines again) but puts the changes back into the staging area.

Now those files are staged again, so I can make a single commit for removing debug logging from my code.

Voila! I can push these changes and not clutter our history with my error.


Displaying An Image Tooltip

JQuery developers seem to have put a lot of effort into filtering HTML components out of tooltips … which, as someone who visits a website … is A Good Thing. But what’s a good security consideration can be a massive pain when building a website. I have a form which takes an internal ID number, and I have an image showing people how to find that internal ID number. I want a little question mark after the field name that pops up the image as a tooltip on mouseover events. And clears the image on mouseout.


// Show finding equipment ID image "tooltip" 
        	$('#FindingEID').css({ "display": "block" });
        	$('#FindingEID').css({ "display": "none" });
<div class="col-md-2 col-sm-2 col-lg-2 col-xs-2 text-left">
	<span><strong>Equipment ID(s): <a id="ShowEquipmentIDTip" href="#">(?)</a></strong></span>
	<div id="FindingEID" style="position: relative;top: 20;left: 60;width: 100%;height: 100%;z-index:99;display:none"><img src="/groomsGenerateCircuitReport/images/Tip-FindingEquipmentID.png" /></div>

Moving your mouse over the ShowEquipmentIDTip a element displays the div which contains my image “tooltip” and moving the mouse away sets the display back to none.

Shell Script: Path To Script

We occasionally have to re-home our shell scripts, which means updating any static path values used within scripts. It’s quick enough to build a sed script to convert /old/server/path to /new/server/path, but it’s still extra work.

The dirname command works to provide a dynamic path value, provided you use the fully qualified path to run the script … but it fails spectacularly whens someone runs ./ and you’re trying to use that path in, say, EXTRA_JAVA_OPTS. The “path” is just . — and Java doesn’t have any idea what to do with “-Xbootclasspath/a:./more/path/goes/here.jar”

Voila, realpath gives you the fully qualified file path for /new/server/path/, ./, or even bash … and the dirname of a realpath is the fully qualified path where resides:

DIRNAME=`dirname $(realpath "$0")`
echo ${DIRNAME}

Hopefully next time we’ve got to re-home our batch jobs, it will be a simple scp & sed the old crontab content to use the new paths.

Modifying Shared PHP Function

We needed to modify a shared function to include additional information … but didn’t want to coordinate changing all of the calls to the function as one change. Simplest way to accomplish that was to set a default value for the new parameter — either to NULL and just not do the new thing when the parameter is NULL or some value that indicates that we’re not yet gathering that data.


function testFunction($strOldParameter, $strNewParameter=NULL){
     echo "The old parameter is |$strOldParameter|\n";
          echo "The new parameter is |$strNewParameter|\n";

testFunction("first", "second");


Preventing erronious use of the master branch on development servers

One of the web servers at work uses a refspec in the “git pull” command to map the remote development branch to the local remote-tracking master branch. This is fairly confusing (and it looks like the dev server is using the master branch unless you dig into how the pull is performed), but I can see how this prevents someone from accidentally typing something like “git checkout master” and really messing up the development environment. I can also see a dozen ways someone can issue what is a completely reasonable git command 99% of the time and really mess up the development environment.

While it is simple enough to just checkout the development branch, doing so does open us up to the possibility that someone will erroneously  deliver the production code to the development server and halt all testing. While you cannot create shell aliases for multi-word commands (or, more accurately, alias expansion is performed for the first word of a simple command is checked to see if it has an alias … so you’ll never get the multi-word command), you can define a function to intercept git commands and avoid running unwanted commands:

function git() { 
     case $* in 
         "checkout master" ) command echo "This is a dev server, do not checkout the master branch!" ;; 
         "pull origin master" ) command echo "This is a dev server, do not pull the master branch" ;; 
         * ) command git "$@" ;; 

Or define the desired commands and avoid running any others:

function git(){
     if echo "$@" | grep -Eq '^checkout uat$'; then
          command git $@
     elif echo "$@" | grep -Eq '^pull .+ uat$'; then
          command git $@
          echo "The command $@ needs to be whitelisted before it can be run"

Either approach mitigates the risk of someone incorrectly using the master branch on the development server.

Handling PHP Execution Timeout

There’s no straight-forward way to handle execution timeout in PHP 5.x — it’s not like you can try/except or something. The execution time-limit is exceeded, the program terminates. Which, thinking from the perspective of the person who maintains the server, is a Good Thing … bugger up the ‘except’ component and now that becomes an infinite loop.

But I’m looking to throw a “pretty” error to the end user and have them try again with a data set that will take less time to process. Turns out, you can use a shutdown function to display something other than the generic PHP time limit exceeded page.


function runOnShutdown(){
     $arrayError = error_get_last();

     if( substr($arrayError['message'], 0, strlen("Maximum execution time of")) === "Maximum execution time of"   ){
          echo "<P>Maximum execution time";

function noOp($iInteger){
     for($z = 0; $z < $iInteger; $z++){
          $a = $iInteger * $iInteger;
     return $iInteger;

ini_set('display_errors', '0');
ini_set('max_execution_time', 2);

// for($i = 0; $i < 10; $i++){
for($i = 0; $i < 10000; $i++){
     $j = noOp($i);
     print "<P>$j</P>\n";
print "<P>Done</P>\n";


And the web output includes a customized message because the max execution time has been exceeded.


Reverting a Single File with Git

Git revert is great for resetting the entire project to a particular state – I went down a bad path, really don’t want to do this, and resetting to the state I was in this morning is exactly what I want to do. Sometimes, though … that’s not the case. I added a couple of debugging lines to a file that I don’t really need. Or I’ve gone down a bad path here but have good work in a few other files too. In those cases, you can revert a single file to the latest committed version. Run “git status” and “git diff” to confirm that it is an uncommited change.

To revert a single file to its latest committed state, use “git checkout – filename” – you can see the added line has disappeared.


Git Log

Git log can be used to get a quick summary of the differences between two branches. The three dots between the branch names indicates you want a “symmetric difference”. That is the set of commits that are in the branch on the left or the branch on the right but not in both branches.

The –left-right option prefixes each commit with an ‘<’ or ‘>’ indicating which “side” has the commit. The –oneline option prints the abbreviated commit ID and the first line of the commit message.

Showing the differences between your local uat branch and the remote uat branch:

D:\git\gittest>git log –left-right –oneline origin/uat…uat

> 961f53a (uat) Merge branch ‘ossa-123’ into uat

> 803096b (origin/ossa-123, ossa-123) Added additional files

> cf9c419 Added initial code to branch

The top line is the most recent commit, the bottom line is the oldest commit that does not exist in both branches. I can see that the uat branch in my local repo is not missing anything from the remote (there are no commits with “<” indicating changes in the remote that do not exist in my local copy) but I have local changes which have not yet been pushed: two code commits plus the merge commit which incorporated the code commits to my local repo’s uat branch. The head of the local and remote ossa-123 branch are at the commit just prior to the merge, so on my local repo that branch has been fully merged into UAT and I just need to push uat up to the remote.

Additional options to enhance output:

–cherry-pick will omit any changes that represent the same changes in both branches (or –cherry-mark to mark those commits with an “=” flag)

–graph uses an ASCII chart to depict branch relationships.

* The three dots mean something different in git diff than in git log. In git diff, mean “what are the differences between the right-hand branch and the common ancestor shared by both the right and left-hand branches”.

Two dots in git diff mean is the differences that are in the branch on the left or the branch on the right but not in both branches.

In git log, two dots displays only commits unique to the second branch. Since commits and differences are not exactly the same thing, two and three dots don’t exactly have the opposite meaning between diff and log. But the meaning is not logically consistent.