Category: System Administration

Owntracks Stuck In “Connecting” To MQTT When Using WebSockets

Our home automation presence is maintained through an Android app, OwnTracks, which updates a Mosquitto server via a WebSockets reverse proxy. Mosquitto runs on a Fedora 25 server and was installed from the default RPM repository.

Recently, we stopped receiving location updates – both of our Android clients were stuck “Connecting” to the MQTT server. Nothing appeared in the Apache access or error logs, and capturing network traffic only got a small number of packets (TCP session overhead ‘stuff’). Even bypassing the reverse proxy and using the internal network to communicate directly to the Mosquitto server only created a couple of packets. Using a test client (http://www.hivemq.com/demos/websocket-client/), I saw strange connection failures — so I knew the problem was not specific to the OwnTracks client.

It seems there was a bug in libwebsockets v2.1.1 (and possibly others) — when we updated our Fedora installation, the new libwebsockets broke our MQTT over WebSockets. Currently, the Fedora repository still contains an impacted version of libwebsockets. To resolve the issue, I built the latest stable libwebsockets and built mosquitto against this updated library.

Process: The first step is to remove the dnf managed packages (rpm -e libwebsockets libwebsockets-devel mosquitto). Then build libwebsockets and mosquitto.

To Build LibWebSockets:

wget https://github.com/warmcat/libwebsockets/archive/master.zip
unzip master.zip
cd libwebsockets-master/
mkdir build
cd build
cmake ..
make
make install
cp libwebsockets.pc /usr/lib/
cp lws_config.h /usr/include/
cp ../lib/libwebsockets.h /usr/include/
cp ./lib/libwebsockets.so /usr/lib/

To Build Mosquitto:

wget https://github.com/eclipse/mosquitto/archive/master.zip
unzip master.zip
cd mosquitto-1.4.11
vi config.mk # Line 68, change to “WITH_WEBSOCKETS:=yes”
make
make install

Start the Mosquitto server and try again. Voila, presence works again!

Compiling Open ZWave On Fedora 25

Mostly writing this down for me, next time we need to run Open ZWave and try to build the latest version:

Download libmicrohttpd
Gunzip & untar it
cd libmicrohttpd
./configure
make
make install

Download and build the open-zwave library

mkdir /opt/ozw
cd /opt/ozw
git clone https://github.com/OpenZWave/open-zwave.git
cd open-zwave-master
make

Find error in build that says you don’t have libudev.h, install systemd-devel (dnf install systemd-devel) & try that make again.

Download open-zwave-control-panel
cd /opt/ozw
git clone https://github.com/OpenZWave/open-zwave-control-panel.git
cd open-zwave-control-panel-master

Open the Makefile and find the following line:
OPENZWAVE := ../

Change it to:
OPENZWAVE := ../open-zwave-master

Then find the section that says:
# for Linux uncomment out next three lines
LIBZWAVE := $(wildcard $(OPENZWAVE)/*.a)
#LIBUSB := -ludev
#LIBS := $(LIBZWAVE) $(GNUTLS) $(LIBMICROHTTPD) -pthread $(LIBUSB) -lresolv

# for Mac OS X comment out above 2 lines and uncomment next 5 lines
#ARCH := -arch i386 -arch x86_64
#CFLAGS += $(ARCH)
#LIBZWAVE := $(wildcard $(OPENZWAVE)/cpp/lib/mac/*.a)
LIBUSB := -framework IOKit -framework CoreFoundation
LIBS := $(LIBZWAVE) $(GNUTLS) $(LIBMICROHTTPD) -pthread $(LIBUSB) $(ARCH) -lresolv

And switch it around to be Linux … the Makefile becomes:
# for Linux uncomment out next three lines
LIBZWAVE := $(wildcard $(OPENZWAVE)/*.a)
LIBUSB := -ludev
LIBS := $(LIBZWAVE) $(GNUTLS) $(LIBMICROHTTPD) -pthread $(LIBUSB) -lresolv

# for Mac OS X comment out above 2 lines and uncomment next 5 lines
#ARCH := -arch i386 -arch x86_64
#CFLAGS += $(ARCH)
#LIBZWAVE := $(wildcard $(OPENZWAVE)/cpp/lib/mac/*.a)
#LIBUSB := -framework IOKit -framework CoreFoundation
#LIBS := $(LIBZWAVE) $(GNUTLS) $(LIBMICROHTTPD) -pthread $(LIBUSB) $(ARCH) -lresolv

ln -sd ../open-zwave/config
make

Then you can run it:
./ozwcp -p 8889

./ozwcp: error while loading shared libraries: libmicrohttpd.so.12: cannot open shared object file: No such file or directory

strace it (strace ./ozwcp -p 8889)

open(“/lib64/tls/x86_64/libmicrohttpd.so.12”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat(“/lib64/tls/x86_64”, 0x7ffefb50d660) = -1 ENOENT (No such file or directory)
open(“/lib64/tls/libmicrohttpd.so.12”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat(“/lib64/tls”, {st_mode=S_IFDIR|0555, st_size=4096, …}) = 0
open(“/lib64/x86_64/libmicrohttpd.so.12”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat(“/lib64/x86_64”, 0x7ffefb50d660) = -1 ENOENT (No such file or directory)
open(“/lib64/libmicrohttpd.so.12”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat(“/lib64”, {st_mode=S_IFDIR|0555, st_size=122880, …}) = 0
open(“/usr/lib64/tls/x86_64/libmicrohttpd.so.12”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat(“/usr/lib64/tls/x86_64”, 0x7ffefb50d660) = -1 ENOENT (No such file or directory)
open(“/usr/lib64/tls/libmicrohttpd.so.12”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat(“/usr/lib64/tls”, {st_mode=S_IFDIR|0555, st_size=4096, …}) = 0
open(“/usr/lib64/x86_64/libmicrohttpd.so.12”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat(“/usr/lib64/x86_64”, 0x7ffefb50d660) = -1 ENOENT (No such file or directory)
open(“/usr/lib64/libmicrohttpd.so.12”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat(“/usr/lib64”, {st_mode=S_IFDIR|0555, st_size=122880, …}) = 0

Huh … not looking in the right place. I’m sure there’s a right way to sort this, but we’re using Open ZWave for a couple of minutes to test some ZWave security stuff. Not worth the time:

ln -s /usr/local/lib/libmicrohttpd.so.12.41.0 /usr/lib64/libmicrohttpd.so.12

Try again (./ozwcp -p 8889). Voila, “2017-04-17 20:35:05.223 Always, OpenZwave Version 1.4.0 Starting Up”. Use your browser to hit http://<ipaddress>:8888 to access the Open ZWave Control Panel.

Uninformed Upgrades (PHP 5 => 7)

TL;DR: Check the list of what is being updated before you let an OS automatically update its programs.

We have a home automation / MythTV / ZoneMinder server with automatic updates disabled. In the process of updating OpenHAB to OpenHAB2, Scott suggested we update everything else while we’re at it. No big, did a quick “dnf update” … got a gig of packages downloaded, waiting for >1400 packages to install, and rebooted.

PHP could not talk to MySQL. At all. ZoneMinder just threw an error saying we didn’t have the PHP MySQL module installed (it worked half an hour ago, so it is INSTALLED). MythWeb completely failed to load – just a white screen. The quick web view of OpenHAB persistence history threw a class not found error.

I checked to see if the extensions were loaded (use the command “print_r(get_loaded_extensions());” in a PHP page) – huh, a LOT of my modules were missing. But there weren’t any useful errors anywhere indicating why.

I modified the php.ini file to show startup errors.

[root@fedora01 conf.modules.d]# grep display_startup_errors /etc/php.ini
; display_startup_errors
display_startup_errors = On

Oooooh, now there are errors! A lot of them. Not particularly useful, but at least a good clue that this isn’t going to go so well for me:

PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/pdo.so’ – /usr/lib64/php/modules/pdo.so: undefined symbol: zend_ce_exception in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mysqlnd.so’ – /usr/lib64/php/modules/mysqlnd.so: undefined symbol: zend_hash_str_del in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/bcmath.so’ – /usr/lib64/php/modules/bcmath.so: undefined symbol: _emalloc_16 in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/bz2.so’ – /usr/lib64/php/modules/bz2.so: undefined symbol: zend_fetch_resource2_ex in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/calendar.so’ – /usr/lib64/php/modules/calendar.so: undefined symbol: _emalloc_32 in Unknown on line 0
PHP Warning: PHP Startup: ctype: Unable to initialize module\nModule compiled with module API=20151012\nPHP compiled with module API=20131226\nThese options need to match\n in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/curl.so’ – /usr/lib64/php/modules/curl.so: undefined symbol: zend_list_close in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/dom.so’ – /usr/lib64/php/modules/dom.so: undefined symbol: zend_ce_exception in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/exif.so’ – /usr/lib64/php/modules/exif.so: undefined symbol: zend_hash_str_exists in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/fileinfo.so’ – /usr/lib64/php/modules/fileinfo.so: undefined symbol: zend_list_close in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/ftp.so’ – /usr/lib64/php/modules/ftp.so: undefined symbol: zend_fetch_resource2 in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/gd.so’ – /usr/lib64/php/modules/gd.so: undefined symbol: zend_list_close in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/gettext.so’ – /usr/lib64/php/modules/gettext.so: undefined symbol: zend_parse_arg_str_slow in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/iconv.so’ – /usr/lib64/php/modules/iconv.so: undefined symbol: _zval_get_string_func in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/json.so’ – /usr/lib64/php/modules/json.so: undefined symbol: _emalloc_56 in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mbstring.so’ – /usr/lib64/php/modules/mbstring.so: undefined symbol: zend_hash_str_del in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mysqlnd.so’ – /usr/lib64/php/modules/mysqlnd.so: undefined symbol: zend_hash_str_del in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/phar.so’ – /usr/lib64/php/modules/phar.so: undefined symbol: zend_sort in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/posix.so’ – /usr/lib64/php/modules/posix.so: undefined symbol: _zend_hash_str_update in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/shmop.so’ – /usr/lib64/php/modules/shmop.so: undefined symbol: zend_list_close in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/simplexml.so’ – /usr/lib64/php/modules/simplexml.so: undefined symbol: zend_ce_exception in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/sockets.so’ – /usr/lib64/php/modules/sockets.so: undefined symbol: zend_hash_str_del in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/sqlite3.so’ – /usr/lib64/php/modules/sqlite3.so: undefined symbol: zend_ce_exception in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/sysvmsg.so’ – /usr/lib64/php/modules/sysvmsg.so: undefined symbol: _emalloc_64 in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/sysvsem.so’ – /usr/lib64/php/modules/sysvsem.so: undefined symbol: _emalloc_24 in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/sysvshm.so’ – /usr/lib64/php/modules/sysvshm.so: undefined symbol: zend_list_close in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/tidy.so’ – /usr/lib64/php/modules/tidy.so: undefined symbol: _zend_hash_str_update in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/tokenizer.so’ – /usr/lib64/php/modules/tokenizer.so: undefined symbol: _emalloc_large in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/xml.so’ – /usr/lib64/php/modules/xml.so: undefined symbol: _zend_hash_str_add in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/xmlwriter.so’ – /usr/lib64/php/modules/xmlwriter.so: undefined symbol: _emalloc_16 in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/xsl.so’ – /usr/lib64/php/modules/xsl.so: undefined symbol: dom_node_class_entry in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mysql.so’ – /usr/lib64/php/modules/mysql.so: undefined symbol: mysqlnd_connect in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/mysqli.so’ – /usr/lib64/php/modules/mysqli.so: undefined symbol: zend_ce_exception in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/pdo_mysql.so’ – /usr/lib64/php/modules/pdo_mysql.so: undefined symbol: mysqlnd_allocator in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/pdo_sqlite.so’ – /usr/lib64/php/modules/pdo_sqlite.so: undefined symbol: php_pdo_unregister_driver in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/wddx.so’ – /usr/lib64/php/modules/wddx.so: undefined symbol: zend_list_close in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/xmlreader.so’ – /usr/lib64/php/modules/xmlreader.so: undefined symbol: dom_node_class_entry in Unknown on line 0
PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib64/php/modules/json.so’ – /usr/lib64/php/modules/json.so: undefined symbol: _emalloc_56 in Unknown on line 0

Turns out DNF installed PHP 7, but didn’t do anything to remove the PHP 5 modules from my Apache configuration:

[root@fedora01 tmp]# cd /etc/httpd/modules
[root@fedora01 modules]# grep php *
Binary file libphp5.so matches
Binary file libphp5-zts.so matches
Binary file libphp7.so matches
Binary file libphp7-zts.so matches

[root@fedora01 modules]# mkdir /tmp/oldphp
[root@fedora01 modules]# mv libphp5* /tmp/oldphp

And remove them from the conf.modules.d too (if you just remove the module files but try to load them in the conf.modules.d … Apache will just fail to load. You could remove them from conf.modules.d … but I don’t want a lot of no-longer-used files sitting there to confuse me in a year or two!)

[root@fedora01 modules]# cd /etc/httpd/conf.modules.d/
[root@fedora01 conf.modules.d]# grep php *
10-php.conf: LoadModule php5_module modules/libphp5.so
10-php.conf: LoadModule php5_module modules/libphp5-zts.so
15-php.conf:# Cannot load both php5 and php7 modules
15-php.conf:<IfModule !mod_php5.c>
15-php.conf: LoadModule php7_module modules/libphp7.so
15-php.conf:<IfModule !mod_php5.c>
15-php.conf: LoadModule php7_module modules/libphp7-zts.so

[root@fedora01 conf.modules.d]# mv 10-php.conf /tmp/oldphp/

Then restart Apache without PHP 5:

root@fedora01 conf.modules.d]# service httpd start
Redirecting to /bin/systemctl start httpd.service

Voila, perfectly functioning web sites. And, yeah, I should probably check the list of “what will be updated” when I update a server. Would save HOURS of reading through strace output to find out old versions were still hanging about.

 

LAPS For Local Computer Administrator Passwords

Overview

LAPS is Microsoft’s solution to a long-existing problem within a corporation using Windows computers: when you image computers, all of the local administrator passwords are the same. Now some organizations implemented a process to routinely change that password, but someone who is able to compromise the local administrator password on one box basically owns all of the other imaged workstations until the next password change.

Because your computer’s local administrator password is the same as everyone else’s, IT support cannot just give you a local password to access your box when it is malfunctioning. This means remote employees with incorrect system settings end up driving into an office just to allow an IT person to log into the box.

With LAPS, there is no longer one ring to rule them all – LAPS allows us to maintain unique local administrator passwords on domain member computers. A user can be provided their local administrator password without allowing access to all of the other domain-member PCs (or a compromised password one one box lets the attacker own only that box). A compromised box is still a problem, but access to other boxes within the domain would only be possible by retrieving other credentials stored on the device.

Considerations

Security: The end user is prevented from accessing the password or interacting with the process. The computer account manages the password, not the user (per section 4 LAPS_TechnicalSpecification.docx from https://www.microsoft.com/en-us/download/details.aspx?id=46899).

Within the directory, read access is insufficient (per https://blogs.msdn.microsoft.com/laps/2015/06/01/laps-and-password-storage-in-clear-text-in-ad/) to view the attribute values. In my proposed deployment, users (even those who will be retrieving the password legitimately) will use a web interface, so a single service acct will have read access to the confidential ms-Mcs-AdmPwd attribute and write access to ms-Mcs-AdminPwdExpirationTime. There are already powershell scripts published to search an improperly secured directory and dump a list of computer names & local administrator passwords. You should run Find-AdmPwdExtendedrights -identity :<OU FQDN> to determine who has the ability to read the password values to avoid this really embarrassing oversight.

Should anyone have access to read the ms-Mcs-AdmPwd value beyond the service account? If the web interface goes down for some reason, is obtaining the local administrator password sufficiently important that, for example, help desk management should be able to see the password through the MS provided client? Depends on the use cases, but I’m guessing yes (if for no other reason than the top level AD admins will have access and will probably get rung up to find the password if the site goes down).

In the AD permissions, watch who has write permission to ms-Mcs-AdminPwdExpirationTime as write access allows someone to bump out the expiry date for the local admin password. Are we paranoid enough to run a filter for expiry > GPO interval? Or does setting “not not allow password expiration time longer than required by policy” to Enabled sufficently mitigate the issue? To me, it does … but the answer really depends on how confidential the data on these computers happens to be.

With read access to ms-Mcs-AdmPwdExpirationTime, you can ascertain which computers are using LAPS to manage the local administrator password (a future value is set in the attribute) and which are not (a null or past value). Is that a significant enough security risk to worry about mitigating? An attacker may try to limit their attacks to computers that do not use LAPS to manage the local admin password. They can also ascertain how long the current password will be valid.

How do you gain access to the box if the local admin password stored in AD does not work (for whatever reason)? I don’t think you’re worse off than you would be today – someone might give you the local desktop password, someone might make you drive into the office … but bears considering if we’ve created a scenario where someone might have a bigger problem than under the current setup.

Does this interact at all with workplace join computers? My guess is no, but haven’t found anything specific about how workplace joined computers interact with corporate GPOs.

Server Side

Potential AD load – depends on expiry interval. Not huge, but non-zero.

Schema extension needs to be loaded. Remove extended rights from attribute for everyone who has it. Add computer self rights. Add control access for web service acct – some individuals too as backup in case web server is down??

Does a report on almost expired passwords and notify someone have value?

Client Side

Someone else figures this out, not my deal-e-o. Set GPO for test machines, make sure value populates, test logon to machine with password from AD. Provide mechanism to force update of local admin password on specific machine (i.e. if I ring in and get the local admin password today, it should get changed to a new password in some short delta time).

Admin Interface

Web interface, provide computer name & get password. Log who made request & what computer name. If more than X requests made per user in a (delta time), send e-mail alert to admin user just in case it is suspicious activity. If more than Y requests made per user in a (longer delta time), send e-mail alert to admin user manager.

Additionally we need a function to clear the password expiry (force the machine to set a new password) to be used after local password is given to an end user.

User Interface

Can we map user to computer name and give the user a process to recover password without calling HD? Or have the manager log in & be able to pull local administrator for their directs? Or some other way to go about actually reducing call volume.

Future Considerations

Excluding ms-Mcs-AdmPwd  from repl to RODC – really no point to it being there.

Do we get this hooked up for acquired company domains too, or do they wait until they get in the WIN domain?

Does this facilitate new machine deployment to remote users? If you get a newly imaged machine & know its name, get the local admin password, log in, VPN in … can you do a run-as to get your creds cached? Or do a change user and still have the VPN session running so you can change to a domain user account?

LAPS For Servers: Should this be done on servers too? Web site could restrict who could view desktops v/s who could view servers … but it would save time/effort when someone leaves the group/company there too. Could even have non-TSG folks who would be able to get access to specific boxes – no idea if that’s something Michael would want, but same idea as the desktop side where now I wouldn’t give someone the password ‘cause it’s the password for thousands of other computers … may be people they wouldn’t want having local admin on any WIN box they maintain … but having local admin on the four boxes that run their app … maybe that’s a bonus. If it is deployed to servers, make sure they don’t put it on DCs (unless you want to use LAPS to manage the domain administrator password … which is an interesting consideration but has so many potential problems I don’t want to think about it right now especially since you’d have to find which DC updated the password most recently).

LAPS For VDI: Should this be done on VDI workstations? Even though it’s a easier to set the password on VDI the base VDI images than each individual workstation, it’s still manual effort & provides an attack vector for all of the *other* VDI sessions. Persistent sessions are OK without any thought because functionally no different than workstations. Non-persistent with new name each time are OK too – although I suspect you end up with a BUNCH of machine objects in AD that need to be cleaned up as new machine names come online. Maybe VDI sorts this … but the LAPS ‘stuff’ is functionally no different than bringing a whole bunch of new workstations online all the time.

Non-persistent sessions with same computer name … since the password update interval probably won’t have elapsed, the in-image password will be used. Can implement an on-boot script that clears AdmPwdExpirationTime to force change. Or a script to clear value on system shutdown (but that would need to handle non-clean shutdowns). That would require some testing.

 

Testing Process

We can have a full proof of concept type test by loading schema into test active directory (verify no adverse impact is seen) and having a workstation joined to the test domain. We could provide a quick web site where you input a computer name & get back a password (basically lacking the security-related controls where # of requests generate some action). This would allow testing of the password on the local machine. Would also allow testing of force-updating the local admin password.

Once we determine that this is worth the effort, web site would need to be flushed out (DB created for audit tracking). Schema and rights would need to be set up in AD. Then it’s pretty much on the desktop / GPO side. I’d recommend setting the GPO for a small number of test workstations first … but that’s what they do for pretty much any GPO change so not exactly ground breaking.

Viewing Active Directory Object Metadata

Objects in active directory have a modification timestamp attribute, whenChanged, that reflects the time of the last change to the object. This is useful if you want to confirm a change had not been made after a specific time (e.g. the user began having problems at 2PM yesterday, but their object was last changed November of last year … an account change is not likely to be the cause).

There is additional stored metadata which provides a modification timestamp (and source domain controller for the modification event) for each individual attribute on an object. This can be a lot more useful (e.g. a user’s home directory is incorrect, but the object modification timestamp reflects the fact they changed their password yesterday). To view the metadata, use repadmin /showobjmeta DC-Hostname “objectFQDN”

I redirect the output to a file; it’s a lot easier to search a text file for the attribute name than scroll through all of the attributes in a DOS window.

repadmin /showobjmeta dc.domain.gTLD "cn=user account,ou=pathToObject,dc=domain,dc=gTLD" > myaccount.txt

57 entries.
Loc.USN Originating DSA                       Org.USN   Org.Time/Date       Ver   Attribute
======= ===============                       ========= =============       ===   =========
20822   92d3c1e5-d4ed-41c7-989f-62a1712b1084  20822     2014-06-08 22:20:57 1     cn
...
4659114 Default-First-Site-Name\DC            4659114   2016-12-29 20:56:21 10    unicodePwd
3299408 Default-First-Site-Name\DC            3299408   2016-01-16 17:03:05 13    lockoutTime
4978129 Default-First-Site-Name\DC            4978129   2017-02-18 21:50:13 90    lastLogonTimestamp
4988421 Default-First-Site-Name\DC            4988421   2017-02-22 10:31:06 54333 msDS-LastSuccessfulInteractiveLogonTime
4977488 Default-First-Site-Name\DC            4977488   2017-02-18 16:21:12 223   msDS-LastFailedInteractiveLogonTime
4977488 Default-First-Site-Name\DC            4977488   2017-02-18 16:21:12 223   msDS-FailedInteractiveLogonCount
4977489 Default-First-Site-Name\DC            4977489   2017-02-18 16:21:18 165   msDS-FailedInteractiveLogonCountAtLastSuccessfulLogon

The originating DSA may be an odd GUID value (the domain controller on which this change was initiated has since been decommissioned) or it may be an AD site and domain controller name.

The originating timestamp indicates when the attribute’s value was last changed. The version indicates the number of revisions on the attribute value – which itself can provide interesting information like the number of times an account has been locked out or the number of times a user has changed their password.

This information can be useful when an account change does correspond with a user experiencing problems. You can identify the specific attributes that were updated and research those specific values.

It’s also useful to track down who changed a specific attribute value. The combination of originating domain controller and attribute modification time can make searching for the event log record corresponding to a specific change a lot easier — you know which server to search and can filter the log down to records spanning a few seconds.

 

Serial Port Sniffer

We use a Wink hub to communicate with our ZigBee devices – scripts on the OpenHAB server make web calls over to the Wink hub to set bulb levels. Works great on outbound communication to the bulbs, but it is not real-time bi-directional (i.e. if a bulb level is changed elsewhere, OpenHAB would need to poll and get the new value). Doesn’t matter for the GE Link bulbs because there isn’t another way they get set beyond dropping and returning power (which turns the bulb on at 100%), but we cannot use the Wink hub to communicate with interactive devices — unlock the door manually and OpenHAB has no idea the light should be turned on until the next polling cycle. And polling is a lot of extra overhead – check every device every minute 24×7. And it’s slow – hit the polling cycle wrong and it takes a minute from unlocking the garage door before the light turns on.

Had the idea of monitoring data that moves across the serial interfaces and use a script to communicate real-time inbound changes over to OpenHAB. Watching the serial interface, we get lots of cryptic traffic from socat:

socat -x /dev/SerialPort,raw,echo=0,crnl PTY,link=/dev/ttyV1,raw,echo=0,crnl

Custom Password Filter Update (unable to log on after changing password with custom filter in place)

I had written and tested a custom Active Directory password filter – my test included verifying the password actually worked. The automated testing was to select a UID from a pool, select a test category (good password, re-used password, password from dictionary, password that doesn’t meet character requirements, password containing surname, password containing givenName), set the password on the user id. Record the result from the password set, then attempt to use that password and record the result from the bind attempt. Each test category has an expected result, and any operation where the password set or bind didn’t match the expected results were highlighted. I also included a high precision timer to record the time to complete the password set operation (wanted to verify we weren’t adversely impacting the user experience). Published results, documented the installation and configuration of my password filter, and was done.

Until the chap who was installing it in production rang me to say he couldn’t actually log in using the password he set on the account. Which was odd – I set one and then did an LDAP bind and verified the password. But he couldn’t use the same password to log into a workstation in the test domain. Huh?? I actually knew people who wanted *some* users to be able to log in anywhere and others to be restricted to LDAP-only logons (i.e. web portal stuff) and ended up using the userWorkstations attribute to allow logon to DCs only.

We opened a case with Microsoft and it turns out that their Password Filter Programming Considerations didn’t actually mean “Erase all memory used to store passwords by calling the SecureZeroMemory function before freeing memory.” What they meant was “If you have created copies of the password anywhere within your code, make sure you erase memory used to store those copies by calling SecureZeroMemory …”

Which makes SO much more sense … as the comments in the code I used as our base says, why wouldn’t MS handle wiping the memory? Does it not get cleaned well if you don’t have a custom password filter?? Remarked out the call to SecureZeroMemory and you could use the password on NTLM authentications as well as kerberos!

// MS documentation suggests doing this. I honestly don’t know why LSA
// doesn’t just do this for you after we return. But, I’ll do what the
// docs say…
// LJR – 2016-12-15 Per MS, they actually mean to wipe any COPIES you make
// SecureZeroMemory(Password->Buffer, Password->Length);

 

I’ve updated my version of the filter and opened an issue on the source GitHub project … but if anyone else is working a custom password filter, following MS’s published programming considerations, and finds themselves unable to use the password they set … see if you are zapping your copies of the password or the PUNICODE_STRING that comes in.

Active Directory: Custom Password Filtering

At work, we’ve never used the “normal” way of changing Windows passwords. Historically, this is because computers were not members of the domain … so you couldn’t use Ctrl-Alt-Del to change your domain password. Now that computers are members of the domain, changing Active Directory passwords using an external method creates a lot of account lockouts. The Windows workstation is logged in using the old credentials, the password gets changed without it knowing (although you can use ctrl-alt-del, lock the workstation unlock with the new password and update the local workstation creds), and the workstation continues using the old credentials and locks the account.

This is incredibly disruptive to business, and quite a burden on the help desk … so we are going to hook the AD-initiated password changes and feed them into the Identity Management platform. Except … the password policies don’t match. But AD doesn’t know the policy on the other end … so the AD password gets changed and then the new password fails to be committed into the IDM system. And then the user gets locked out of something else because they keep trying to use their new password (and it isn’t like a user knows which directory is the back-end authentication source for a web app to use password n in AD and n-1 in DSEE).

long time ago, back when I knew some military IT folks who were migrating to Windows 2000 and needed to implement Rainbow series compliant passwords in AD – which was possible using a custom password filter. This meant a custom coded DLL that accepted or rejected the proposed password based on custom-coded rules. Never got into the code behind it – I just knew they would grab the DLL & how to register it on the domain controller.

This functionality was exactly what we needed — and Microsoft still has a provision to use a custom password filter. Now all we needed was, well, a custom password filter. The password rules prohibit the use of your user ID, your name, and a small set of words that are globally applied to all users. Microsoft’s passfilt.dll takes care of the first two — although with subtle differences from the IDM system’s rules. So my requirement became a custom password filter that prohibits passwords containing case insensitive substrings from a list of words.

I based my project on OpenPasswordFilter on GitHub — the source code prohibits exact string matches. Close, but not quite 🙂 I modified the program to check the proposed password for case insensitive substrings. I also changed the application binding to localhost from all IP address since there’s no need for the program to be accessed from outside the box. For troubleshooting purposes, I removed the requirement that the binary be run as a service and instead allowed it to be run from a command prompt or as a service.  I’m still adding some more robust error handling, but we’re ready to test! I’ve asked them to baseline changing passwords without the custom filter, using a custom filter that has the banned word list hard coded into the binary, and using a custom filter that sources its banned words list from a text file. Hopefully we’ll find there isn’t a significant increase in the time it takes a user to change their password.

My updated code is available at http://lisa.rushworth.us/OpenPasswordFilter-Edited.zip

 

Really Wacky Exchange (ActiveSync) Error

My husband changed his Active Directory password. Routine enough – we’ve got 15k accounts at the office and require a password change every 90 days. That’s 150-200 people changing their password every day. They get themselves locked out a lot (mobile devices, cached workstation credentials, and a host of other unique places people manage to store their creds), but it’s trivial to unlock an individual user.

*Except* — after the account was unlocked, his Windows 10 mail client updated properly and was interacting with the Exchange server. Android, however, still wouldn’t accept his new password. If he typed the wrong thing, it would say invalid password. But whenever he typed the right thing, he got an error indicating the phone and tablet were unable to communicate with the server. Which was bogus — I could see the communication coming across the reverse proxy server. With 200 codes — although you can have a very successful HTTP call deliver an application error message. But it wasn’t like he couldn’t COMMUNICATE with the server. He turned sync off on the phones to avoid getting locked out again, and in the process of troubleshooting ended up deleting all of his accounts hosted on our Exchange 2013 server.

I looked through all of the event logs, Exchange logs … nothing interesting. In desperation, I enabled the individual user ActiveSync logging:

Set-CASMailbox mailNickName -ActiveSyncDebugLogging:$true

Had him attempt to add the mailbox profile again, and dropped the log myself:

Get-ActiveSyncDeviceStatistics -Mailbox mailNickName -GetMailboxLog:$true -NotificationEmailAddress mysmtp@mydomain.ccTLD

Bingo! An exception in the provisioning (Microsoft-Server-ActiveSync?Cmd=Provision) call — I see the phone information come across, the mobile device gets partially added to his account (no OS, phone number, carrier type information … but if you go into OWA and remove the mobile device, an Android device gets added). Error:

Command_WorkerThread_Exception :
— Exception start —
Exception type: System.IO.FileLoadException
Exception message: Could not load file or assembly ‘Microsoft.Exchange.Configuration.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Exception level: 0
Exception stack trace: at Microsoft.Exchange.AirSync.DeviceInformationSetting.ProcessSet(XmlNode setNode)
at Microsoft.Exchange.AirSync.DeviceInformationSetting.Execute()
at Microsoft.Exchange.AirSync.ProvisionCommand.Microsoft.Exchange.AirSync.IProvisionCommandHost.ProcessDeviceInformationSettings(XmlNode inboundDeviceInformationNode, XmlNode provisionResponseNode)
at Microsoft.Exchange.AirSync.ProvisionCommandPhaseOne.Process(XmlNode provisionResponseNode)
at Microsoft.Exchange.AirSync.ProvisionCommand.ExecuteCommand()
at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows…
Exception type: System.IO.FileLoadException
Exception message: The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Exception level: 1
Exception stack trace:
— Exception end —

Now that is an error I’ve never seen before. As a programmer, I know what it means … you’ve basically got some components that don’t match another. But … huh? He changed his password. Connected to the Exchange server directly (instead of remotely viewing logs & files) and saw Windows Update had dropped files and a reboot was pending. Which … some files replaced, others staged for replacement pending a reboot. *That* is some components not matching others. Rebooted our box, and voila … registration goes through, mailbox sync started.

I don’t know how many people allow auto-updates with a manual reboot on a production enterprise server (we manually patch and reboot during a scheduled maintenance window) where this could happen … but evidently Windows Update can get your Exchange server into a state where already configured clients are able to send and receive mail. But clients are unable to update passwords, and new clients cannot be configured.

USN Rollback

I had to recover my domain controller from the Hyper-V image backup. There’s some protection build into AD which prevents just randomly reverting a server. When you’ve got a larger domain, the built-in protection after unsupported restoration procedures serves a purpose. Pausing netlogon avoids having users log on against bad data. Disabling replication avoids propagating bad information out to the remainder of the network. The solution is simple – demote the DC, promote it again, and the DC returns to service. But when you have a single domain controller in a single domain in a single forest … well, there’s no other data around. What the recovered DC has is as good as it’s going to get (i.e. a change from 2AM is lost when I revert to my 10PM backup). And taking the entire domain down and building it overkill. You can, instead, basically tell AD to go with it. From the MS documentation:

To restore a previous version of a virtual domain controller VHD without system state data backup

  1. Using the previous VHD, start the virtual domain controller in DSRM, as described in the previous section. Do not allow the domain controller to start in normal mode. If you miss the Windows Boot Manager screen and the domain controller begins to start in normal mode, turn off the virtual machine to prevent it from completing startup. See the previous section for detailed instructions for entering DSRM.
  2. Open Registry Editor. To open Registry Editor, click Start, click Run, type regedit, and then click OK. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Yes. In Registry Editor, expand the following path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters. Look for a value named DSA Previous Restore Count. If the value is there, make a note of the setting. If the value is not there, the setting is equal to the default, which is zero. Do not add a value if you do not see one there.
  3. Right-click the Parameters key, click New, and then click DWORD (32-bit) Value.
  4. Type the new name Database restored from backup, and then press ENTER.
  5. Double-click the value that you just created to open the Edit DWORD (32-bit) Value dialog box, and then type 1 in the Value data box. The Database restored from backup entry option is available on domain controllers that are running Windows 2000 Server with Service Pack 4 (SP4), Windows Server 2003 with the updates that are included in article 875495 (http://go.microsoft.com/fwlink/?LinkId=137182) in the Microsoft Knowledge Base installed, and Windows Server 2008.
  6. Restart the domain controller in normal mode.
  7. When the domain controller restarts, open Event Viewer. To open Event Viewer, click Start, click Control Panel, double-click Administrative Tools, and then double-click Event Viewer.
  8. Expand Application and Services Logs, and then click the Directory Services log. Ensure that events appear in the details pane.
  9. Right-click the Directory Services log, and then click Find. In Find what, type 1109, and then click Find Next.
  10. You should see at least an Event ID 1109 entry. If you do not see this entry, proceed to the next step. Otherwise, double-click the entry, and then review the text confirming that the update was made to the InvocationID:

 

  • Active Directory has been restored from backup media, or has been configured to host an application partition. 
    The invocationID attribute for this directory server has been changed. 
    The highest update sequence number at the time the backup was created is <time>
    
    InvocationID attribute (old value):<Previous InvocationID value>
    InvocationID attribute (new value):<New InvocationID value>
    Update sequence number:<USN>
    
    The InvocationID is changed when a directory server is restored from backup media or is configured to host a writeable application directory partition.
    
  • Close Event Viewer.
  • Use Registry Editor to verify that the value in DSA Previous Restore Count is equal to the previous value plus one. If this is not the correct value and you cannot find an entry for Event ID 1109 in Event Viewer, verify that the domain controller’s service packs are current. You cannot try this procedure again on the same VHD. You can try again on a copy of the VHD or a different VHD that has not been started in normal mode by starting over at step 1.
  • Close Registry Editor.

 

After following the instructions from Microsoft, I still had a problem — my DC has replication turned off & netlogon comes up paused. In regedit, locate HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters and delete the “Dsa Not Writable” key (value: dword:00000004). In a command prompt, run the following:

 

repadmin /options dchostname.example.com -DISABLE_OUTBOUND_REPL
repadmin /options dchostname.example.com -DISABLE_INBOUND_REPL

Reboot the DC. When it starts, netlogon should be running and replication.