|
|||
|
|||
|
|
|||
SysInternals used to produce a suite of tools for working with Microsoft Windows systems — the company appears to have been acquired by Microsoft, and the tools continue to be developed. I used PSKill and PSExec to automate a lot of system administration tasks. ProcessMonitor is like truss/strace for Windows. Unlike the HFS standard, Windows files end up all over the place (plus info is stashed in the registry). Sometimes applications or services fall over for no reason. Process monitor reports out
When you open procmon, you can build filters to exclude uninteresting operations — there’s a default set of exclusions (no need to log out what procmon is doing!)
Adding exclusions for specific process names can eliminate a lot of I/O — I was looking to troubleshoot a problem on a Domain Controller that had nothing to do with AD specifically, so excluding activity by lsass.exe significantly reduced the amount of data being logged. If I’m using a browser to troubleshoot the problem, I’ll exclude the firefox.exe or chrome.exe binary too.
From the filter screen, click “OK” to begin grabbing data. The easiest thing I’ve found to do is stop capturing data when the program opens (use ctrl-a followed by ctrl-x to clear the already logged stuff). Stage whatever you want to log, use ctrl-e to start capturing. Perform the actions you want to log, return to procmon and use ctrl-e to stop again.
You’ll see reads (and writes) against the registry, including the specific keys. Network operations. File reads and writes. In the “Result” and “Detail” column, you can determine if the operation was successful. There are a lot of expected not found failures — I see these in truss/strace logs too, programs try a bunch of different things and one of them needs to work.
I’ve had programs using a specific, undocumented file for a critical operation — like the service would fail to start because the file didn’t exist. And seeing the path and file open failure allowed me to create that needed file and run my service. I’ve wanted to find out where a program stashes data, and procmon makes that easy to identify.
We’ve got to upgrade some Javascript modules at work — JQuery, Bootstrap, etc. Problem is that changes between the versions mean there’s a lot of rewriting required before we can update. And we pull in these modules using a shared header file. While we could stage all of the changes and update the entire website at once … that means we’re all dedicated to updating our components & are delaying the update until we’re finished.
That’s not ideal — and has the potential to break a lot of things at once. I plan, instead, of putting a default version in the shared header file. And some mechanism to source in a newer version by setting a variable in the individual tool’s PHP code before the header is pulled in. So each tool within the site has a $strJQueryRev, $strBootstrapRev, etc variable. Then the shared header file looks for that variable — loads a newer version when requested or loads the currently used older version when no version is indicated.
if($strJQueryRev == "3.5.1"){
echo "<script src=\"https://code.jquery.com/jquery-3.5.1.min.js\">\n";
}
elseif($strJQueryRev == "3.1.1"){
echo "<script src=\"https://code.jquery.com/jquery-3.1.1.min.js\">\n";
}
else{
echo "<script src=\"https://code.jquery.com/jquery-2.2.4.min.js\">\n"; # Old, in use, version is default
}
Or even
if(!$strRevisionNumber){$strRevisionNumber="2.2.4";}
echo "<script src=\"https://code.jquery.com/jquery-$strRevisionNumber.min.js\">
Each developer can add a version number to a single tool, test it, push it up through production using the newest modules. Move on to the next tool. The site still isn’t done until we’re all done, but we can slowly roll out the update as people are able to test their tools.
We’d been playing a silly game on the FireTV cube — Hovercraft Battle Arena. It’s what my former boss would call good, stupid fun. But you’ve got to wait to open these crates to get stuff — gems, money, and components to upgrade the hovercraft. We wanted to be able to play with maxed out hovercraft. So I poked around a little bit and found a cheat for insta-upgrades. Game components are stored as files. For Android installations, the files are under /Android/data/com.highscorehero.battlearena/files … but it’s a lot easier to write a batch to copy files in Windows. The Windows app data files are under %userprofile%\AppData\Local\Packages\ … then the trick is finding the right HighScoreHeroLLCHoverfractBattleArena folder.
Once you find the files, there’s no validation on the files. Now it’s binary stuff that would take a good bit of effort to manually edit (i.e. if I wanted to just give myself 10k gems, I’d have to figure out which exact bits to flip) but you can grab an entire file when the game is in a state you want. That is — when you have four crates ready to open, copy out the Crates* files. When your near the end of the season and at level twenty-something, copy the XP_Int.data file. When you’ve not claimed any of the season rewards, copy out the SeasonReward_Bool.data file. Drop those files into the live folder used by the game & launch the game. Now you can open four crates, claim the gems from each level, and close the game. Copy the season rewards and crate files back and do it again. And again. And again. Eventually, you’ll have all of the blueprints, upgrade parts, gems, and money that you would need. You can make a backup of the entire folder at this point. Use your money and gems to upgrade a hovercraft, then copy out the files for that craft. Snap the backed up files back so you’ve got all of the gems and money, then upgrade a different craft. Repeat until you’ve got upgraded files for all craft.
It does, unfortunately, kind of ruin the game. There’s no point — you’re not accumulating points to upgrade anymore. Anya and I still like it … but it’s absolutely not the same 🙂
It didn’t occur to me that Apache serves everything under a folder and the .git folder may well be under a folder (you can have your project up a level so there’s a single folder at the root of the project & that folder is DocumentRoot for the web site). Without knowing specific file names, you cannot get anything since directory browsing is disabled. But git has a well-known structure so browsing to /.git/index or really scary for someone who stuffs their password in the repo URL /.git/config is there and Apache happily serves it unless you’ve provided instructions otherwise.
A coworker brought up the intriguing idea of, instead of blocking the .git folder so things subordinate to .git are never served, having a specific list of known good extensions the web server was willing to serve. Which … ironically was one of the things I really didn’t like about IIS. Kind of like the extra frustration of driving behind someone who is going the speed limit. Frustrating because I want to go faster, extra frustrating because they aren’t actually wrong.
But configuring a list of good-to-serve extensions means you’ve got to get a handle on what extensions are on your server in the first place. This command provides a list of extensions and a count per extension (so you can easily identify one-offs that may not be needed):
find /path/to/search/ -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort | uniq -c
You can, of course, use –exclude and avoid adding the .git folders to your tar archive, but I discovered a really cool option that excludes the folders created by a whole host of version control systems:
--exclude-vcsWhich, as of version 1.32, means excluding CVS, RCS, SCCS, git, SVN, Arch, Bazaar, Mercurial, and Darcs as follows:
I was getting an odd error from my attempt to list accounts in Cyberark — “Object reference not set to an instance of an object”. Searching the Internet yielded a lot of issues that weren’t my problem (ampersands in account names in an older version, issues with SSL {and, seriously, someone says disable SSL on the connection they use to retrieve passwords!?! And not just random someone, but RAND?!?}). My issue turned out to be that I was copy/pasting code and used requests.post instead of requests.get — attempting to POST to a GET URL generates this error too.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): cyberark.example.com:443
DEBUG:urllib3.connectionpool:https://cyberark.example.com:443 “POST /PasswordVault/API/auth/Cyberark/Logon HTTP/1.1” 200 182
Before request, header is {‘Content-Type’: ‘application/json’, ‘Authorization’: ‘5TQz5WVjYm5tMjBh5C00M5YyLT50MjYt5Tc2Y5I2ZDI5…AwMDA5MDA7’}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): cyberark.example.com:443
DEBUG:urllib3.connectionpool:https://cyberark.example.com:443 “POST /PasswordVault/api/Accounts?search=sample_account&searchType=contains HTTP/1.1” 500 97
{“ErrorCode”:”CAWS00001E”,”ErrorMessage”:”Object reference not set to an instance of an object.”} 500 Internal Server Error
Two ways to add text to Microsoft Whiteboard sessions — ways that aren’t dragging your finger or mouse around in an attempt to draw legible text — are available. I’d like to be able to change the font in the text box — I get that their font choice is meant to evoke hand-written text, but it strikes me as non-professional.
This is a very strange problem — we had a web server upgraded recently. We use “sudo apachectl start” to bring up the server since the server is maintained by a dedicated Unix support team, and the site worked fine. Until … Sunday morning after the log rotation. Then Box Spout was unable to access the XML data for an Excel file to compress it. Lots of errors:
[Mon Sep 14 10:59:39.137728 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: ZipArchive::close(): Zlib error: stream error in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 199, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.137785 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: fopen(/tmp/xlsx5f5f934699dcd9.17695276.zip): failed to open stream: No such file or directory in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 213, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.137803 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: stream_copy_to_stream() expects parameter 1 to be resource, boolean given in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 214, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.137814 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: fclose() expects parameter 1 to be resource, boolean given in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 215, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.138360 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Fatal error: Uncaught exception ‘Box\\Spout\\Common\\Exception\\IOException’ with message ‘Cannot perform I/O operation outside of the base folder: /tmp’ in /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php:130\nStack trace:\n#0 /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php(82): Box\\Spout\\Common\\Helper\\FileSystemHelper->throwIfOperationNotInBaseFolder(‘/tmp/xlsx5f5f93…’)\n#1 /path/to/site/classes/vendor/box/spout/src/Spout/Writer/XLSX/Helper/FileSystemHelper.php(369): Box\\Spout\\Common\\Helper\\FileSystemHelper->deleteFile(‘/tmp/xlsx5f5f93…’)\n#2 /path/to/site/classes/vendor/box/spout/src/Spout/Writer/XLSX/Internal/Workbook.php(134): Box\\Spout\\Writer\\XLSX\\Helper\\FileSystemHelper->zipRootFolderAndCopyToStream(Resource id #26)\n#3 /path/to/site/ in /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php on line 130, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.139468 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: ZipArchive::close(): Zlib error: stream error in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 199, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.139504 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: fopen(/tmp/xlsx5f5f934699dcd9.17695276.zip): failed to open stream: No such file or directory in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 213, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.139515 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: stream_copy_to_stream() expects parameter 1 to be resource, boolean given in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 214, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.139533 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Warning: fclose() expects parameter 1 to be resource, boolean given in /path/to/site/classes/vendor/box/spout/src/Spout/Writer/Common/Helper/ZipHelper.php on line 215, referer: https://hostname.example.com/path/to/code.php
[Mon Sep 14 10:59:39.139599 2020] [:error] [pid 57117] [client 10.1.2.3:49276] PHP Fatal error: Uncaught exception ‘Box\\Spout\\Common\\Exception\\IOException’ with message ‘Cannot perform I/O operation outside of the base folder: /tmp’ in /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php:130\nStack trace:\n#0 /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php(82): Box\\Spout\\Common\\Helper\\FileSystemHelper->throwIfOperationNotInBaseFolder(‘/tmp/xlsx5f5f93…’)\n#1 /path/to/site/classes/vendor/box/spout/src/Spout/Writer/XLSX/Helper/FileSystemHelper.php(369): Box\\Spout\\Common\\Helper\\FileSystemHelper->deleteFile(‘/tmp/xlsx5f5f93…’)\n#2 /path/to/site/classes/vendor/box/spout/src/Spout/Writer/XLSX/Internal/Workbook.php(134): Box\\Spout\\Writer\\XLSX\\Helper\\FileSystemHelper->zipRootFolderAndCopyToStream(Resource id #26)\n#3 /path/to/site in /path/to/site/classes/vendor/box/spout/src/Spout/Common/Helper/FileSystemHelper.php on line 130, referer: https://hostname.example.com/path/to/code.php
The postupdate script is “systemctl reload httpd.service” — so not exactly the same thing we used to launch the service originally. But I’ve never seen differing behavior between apachectl and systemctl started HTTPD instances. Quick/dirty solution is to disable PrivateTmp, but I’m hoping to be able to isolate why exactly the postupdate script appears to break the service’s access to the private tmp space.
30 November 2020 addendum — In discussing the issue with RedHat, they suggested using either
/sbin/killall -HUP httpd
or
/bin/systemctl restart httpd.service > /dev/null 2>/dev/null || true
Doing this has allowed continual access to the Private Tmp space after log rotation. Woohoo! Not sure why the default configuration that came from the Apache httpd package didn’t work (i.e. it’s not like we built some funky weird log rotation script). But success is good enough for me.
I needed to post information into a SharePoint Online list. There’s an auth header required to post data, but the authentication expires every couple of minutes. Obviously, I could just get a new auth string each time … but that’s terribly inefficient. I could also use what I had and refresh it when it fails … inelegant but effective. I wanted, instead, to have a value cached for a duration slightly less than the server-side expiry on the auth string. This decorator allows me to use the cached auth header string for some period of time and actually run the function that grabs the auth header string before the string is invalidated.
import functools
import time
from datetime import datetime
def timed_cache(iMaxAge, iMaxSize=128, boolTyped=False):
#######################################
# LRU cache decorator with expiry time
#
# Args:
# iMaxAge: Seconds to live for cached results.
# iMaxSize: Maximum cache size (see functools.lru_cache).
# boolTyped: Cache on distinct input types (see functools.lru_cache).
#######################################
def _decorator(fn):
@functools.lru_cache(maxsize=iMaxSize, typed=boolTyped)
def _new(*args, __time_salt, **kwargs):
return fn(*args, **kwargs)
@functools.wraps(fn)
def _wrapped(*args, **kwargs):
return _new(*args, **kwargs, __time_salt=int(time.time() / iMaxAge))
return _wrapped
return _decorator
# Usage example -- 23 second cache expiry
@timed_cache(23)
def slow_function(iSleepTime: int):
datetimeStart = datetime.now()
time.sleep(iSleepTime)
return f"test started at {datetimeStart} and ended at at {datetime.now()}"
print(f"Start at {datetime.now()}")
for i in range(1, 50):
print(slow_function(5))
if i % 5 is 0:
print(f"sleeping 5 at {datetime.now()}")
time.sleep(5)
print(f"done sleeping at {datetime.now()}\n\n")
print(f"Ended at at {datetime.now()}")