Category: Technology

Viewing Active Directory Object Metadata

Objects in active directory have a modification timestamp attribute, whenChanged, that reflects the time of the last change to the object. This is useful if you want to confirm a change had not been made after a specific time (e.g. the user began having problems at 2PM yesterday, but their object was last changed November of last year … an account change is not likely to be the cause).

There is additional stored metadata which provides a modification timestamp (and source domain controller for the modification event) for each individual attribute on an object. This can be a lot more useful (e.g. a user’s home directory is incorrect, but the object modification timestamp reflects the fact they changed their password yesterday). To view the metadata, use repadmin /showobjmeta DC-Hostname “objectFQDN”

I redirect the output to a file; it’s a lot easier to search a text file for the attribute name than scroll through all of the attributes in a DOS window.

repadmin /showobjmeta dc.domain.gTLD "cn=user account,ou=pathToObject,dc=domain,dc=gTLD" > myaccount.txt

57 entries.
Loc.USN Originating DSA                       Org.USN   Org.Time/Date       Ver   Attribute
======= ===============                       ========= =============       ===   =========
20822   92d3c1e5-d4ed-41c7-989f-62a1712b1084  20822     2014-06-08 22:20:57 1     cn
...
4659114 Default-First-Site-Name\DC            4659114   2016-12-29 20:56:21 10    unicodePwd
3299408 Default-First-Site-Name\DC            3299408   2016-01-16 17:03:05 13    lockoutTime
4978129 Default-First-Site-Name\DC            4978129   2017-02-18 21:50:13 90    lastLogonTimestamp
4988421 Default-First-Site-Name\DC            4988421   2017-02-22 10:31:06 54333 msDS-LastSuccessfulInteractiveLogonTime
4977488 Default-First-Site-Name\DC            4977488   2017-02-18 16:21:12 223   msDS-LastFailedInteractiveLogonTime
4977488 Default-First-Site-Name\DC            4977488   2017-02-18 16:21:12 223   msDS-FailedInteractiveLogonCount
4977489 Default-First-Site-Name\DC            4977489   2017-02-18 16:21:18 165   msDS-FailedInteractiveLogonCountAtLastSuccessfulLogon

The originating DSA may be an odd GUID value (the domain controller on which this change was initiated has since been decommissioned) or it may be an AD site and domain controller name.

The originating timestamp indicates when the attribute’s value was last changed. The version indicates the number of revisions on the attribute value – which itself can provide interesting information like the number of times an account has been locked out or the number of times a user has changed their password.

This information can be useful when an account change does correspond with a user experiencing problems. You can identify the specific attributes that were updated and research those specific values.

It’s also useful to track down who changed a specific attribute value. The combination of originating domain controller and attribute modification time can make searching for the event log record corresponding to a specific change a lot easier — you know which server to search and can filter the log down to records spanning a few seconds.

 

Serial Port Sniffer

We use a Wink hub to communicate with our ZigBee devices – scripts on the OpenHAB server make web calls over to the Wink hub to set bulb levels. Works great on outbound communication to the bulbs, but it is not real-time bi-directional (i.e. if a bulb level is changed elsewhere, OpenHAB would need to poll and get the new value). Doesn’t matter for the GE Link bulbs because there isn’t another way they get set beyond dropping and returning power (which turns the bulb on at 100%), but we cannot use the Wink hub to communicate with interactive devices — unlock the door manually and OpenHAB has no idea the light should be turned on until the next polling cycle. And polling is a lot of extra overhead – check every device every minute 24×7. And it’s slow – hit the polling cycle wrong and it takes a minute from unlocking the garage door before the light turns on.

Had the idea of monitoring data that moves across the serial interfaces and use a script to communicate real-time inbound changes over to OpenHAB. Watching the serial interface, we get lots of cryptic traffic from socat:

socat -x /dev/SerialPort,raw,echo=0,crnl PTY,link=/dev/ttyV1,raw,echo=0,crnl

Amazon Prime Household and Security (or lack thereof)

There are a lot of odd security lapses in Amazon’s implementation of Prime Household. Anything that uses Alexa creates shared cards that are visible in the Alexa app. There’s some implicit trust between family members, but even married people may not want to share EVERYTHING. “Echo, add this present for my wife to the list” shows up as a card in your wife’s Alexa app.

We share an Amazon Prime Household with Scott’s dad. We both have Alexa-enabled devices (FireTVs and Dots). Our Dots interact with our home automation system through a Phillips bridge emulator. Scott’s dad has some smart devices, but his FireTV could not find any smart devices when it would search. It would, however, turn the ‘outside lights’ on and off. Except the lights didn’t turn on or off in his house.

Our outside lights started turning on and off one night. We have a lot of home automation, but nothing in the logs indicated why this was occurring. I’m not sure if Scott called him or he called Scott, but we were lucky not to spend a day trying to track down some crazy issue. Evidently your Alexa can switch between profiles on the Amazon Prime Household just by saying “switch accounts”. You can ask it “which account is this?”.

The thing I find odd — our HA Bridge is not publicly addressable. Evidently any device on your account can have another device on your account initiate communication. Otherwise there’s no way a device that is connected to our profile but NOT on our network would be able to communicate with the HA Bridge.

Web-Accessible History From OpenHAB MySQL Persistence Database

My husband has wanted a quick/easy way to see the data stored in OpenHAB’s MySQL persistence database, so I put together a very quick (and ugly) PHP page that provides a list of all Items. If you click on an item, you can page through the item’s records. The index.php from the page is available here. You need a web server (I am using Apache on Fedora), PHP (I am using 5.6) and MySQLi (php-mysqlnd package).

This is a bit of paranoia on my part, but even on a page that is ONLY available internally … I don’t like to use an account with read/write access to display data. I create a new user and assign read access:

CREATE USER 'YourUserName'@'localhost' IDENTIFIED BY 'P#ssw0rdH3r3';
GRANT SELECT ON openhabdb.* to 'YourUserName'@'localhost';
FLUSH PRIVILEGES;

Then use *that* user in the php code. This example has a web server running on the database server – and you connect to the MySQL server via localhost. If your web server is located on a different host, you’ll need to create and grant ‘YourUserName’@ the web server hostname.

Custom Password Filter Update (unable to log on after changing password with custom filter in place)

I had written and tested a custom Active Directory password filter – my test included verifying the password actually worked. The automated testing was to select a UID from a pool, select a test category (good password, re-used password, password from dictionary, password that doesn’t meet character requirements, password containing surname, password containing givenName), set the password on the user id. Record the result from the password set, then attempt to use that password and record the result from the bind attempt. Each test category has an expected result, and any operation where the password set or bind didn’t match the expected results were highlighted. I also included a high precision timer to record the time to complete the password set operation (wanted to verify we weren’t adversely impacting the user experience). Published results, documented the installation and configuration of my password filter, and was done.

Until the chap who was installing it in production rang me to say he couldn’t actually log in using the password he set on the account. Which was odd – I set one and then did an LDAP bind and verified the password. But he couldn’t use the same password to log into a workstation in the test domain. Huh?? I actually knew people who wanted *some* users to be able to log in anywhere and others to be restricted to LDAP-only logons (i.e. web portal stuff) and ended up using the userWorkstations attribute to allow logon to DCs only.

We opened a case with Microsoft and it turns out that their Password Filter Programming Considerations didn’t actually mean “Erase all memory used to store passwords by calling the SecureZeroMemory function before freeing memory.” What they meant was “If you have created copies of the password anywhere within your code, make sure you erase memory used to store those copies by calling SecureZeroMemory …”

Which makes SO much more sense … as the comments in the code I used as our base says, why wouldn’t MS handle wiping the memory? Does it not get cleaned well if you don’t have a custom password filter?? Remarked out the call to SecureZeroMemory and you could use the password on NTLM authentications as well as kerberos!

// MS documentation suggests doing this. I honestly don’t know why LSA
// doesn’t just do this for you after we return. But, I’ll do what the
// docs say…
// LJR – 2016-12-15 Per MS, they actually mean to wipe any COPIES you make
// SecureZeroMemory(Password->Buffer, Password->Length);

 

I’ve updated my version of the filter and opened an issue on the source GitHub project … but if anyone else is working a custom password filter, following MS’s published programming considerations, and finds themselves unable to use the password they set … see if you are zapping your copies of the password or the PUNICODE_STRING that comes in.

Active Directory: Custom Password Filtering

At work, we’ve never used the “normal” way of changing Windows passwords. Historically, this is because computers were not members of the domain … so you couldn’t use Ctrl-Alt-Del to change your domain password. Now that computers are members of the domain, changing Active Directory passwords using an external method creates a lot of account lockouts. The Windows workstation is logged in using the old credentials, the password gets changed without it knowing (although you can use ctrl-alt-del, lock the workstation unlock with the new password and update the local workstation creds), and the workstation continues using the old credentials and locks the account.

This is incredibly disruptive to business, and quite a burden on the help desk … so we are going to hook the AD-initiated password changes and feed them into the Identity Management platform. Except … the password policies don’t match. But AD doesn’t know the policy on the other end … so the AD password gets changed and then the new password fails to be committed into the IDM system. And then the user gets locked out of something else because they keep trying to use their new password (and it isn’t like a user knows which directory is the back-end authentication source for a web app to use password n in AD and n-1 in DSEE).

A long time ago, back when I knew some military IT folks who were migrating to Windows 2000 and needed to implement Rainbow series compliant passwords in AD – which was possible using a custom password filter. This meant a custom coded DLL that accepted or rejected the proposed password based on custom-coded rules. Never got into the code behind it – I just knew they would grab the DLL & how to register it on the domain controller.

This functionality was exactly what we needed — and Microsoft still has a provision to use a custom password filter. Now all we needed was, well, a custom password filter. The password rules prohibit the use of your user ID, your name, and a small set of words that are globally applied to all users. Microsoft’s passfilt.dll takes care of the first two — although with subtle differences from the IDM system’s rules. So my requirement became a custom password filter that prohibits passwords containing case insensitive substrings from a list of words.

I based my project on OpenPasswordFilter on GitHub — the source code prohibits exact string matches. Close, but not quite 🙂 I modified the program to check the proposed password for case insensitive substrings. I also changed the application binding to localhost from all IP address since there’s no need for the program to be accessed from outside the box. For troubleshooting purposes, I removed the requirement that the binary be run as a service and instead allowed it to be run from a command prompt or as a service.  I’m still adding some more robust error handling, but we’re ready to test! I’ve asked them to baseline changing passwords without the custom filter, using a custom filter that has the banned word list hard coded into the binary, and using a custom filter that sources its banned words list from a text file. Hopefully we’ll find there isn’t a significant increase in the time it takes a user to change their password.

My updated code is available at http://lisa.rushworth.us/OpenPasswordFilter-Edited.zip

 

Change

There are a lot of areas where the trend oscillates between two states. “Cloud computing” is somewhat new and somewhat trendy, but the “computers are expensive, bandwidth is cheap … centralize everything” and “computers are cheap, bandwidth is expensive … distribute everything” states haven’t changed. A VAX with its terminals or a Citrix farm with its thin clients. Better graphics today, but conceptually the same thing. The difference is marketing — I don’t believe they pushed VT100’s for the home market. SaaS has personal targets as well as commercial.

With these changes, there were winners and losers. Mainframes lost market-share as companies deployed desktop PCs. And now desktop PC vendors are losing market-share as “cloud” services become prevalent. Sucks on an individual level — for, say, the people IBM laid off as their mainframe business contracted — but it wasn’t the driving force behind a political movement.

Manufacturing moved overseas. Energy production moved overseas. Some manufacturing (electronics, for instance) are harder to bring back — we simply lack the knowledge and equipment to pick up manufacturing electronics. Beyond that, though, it is hard to compete with someone who can continually undercut you on cost. You can slap import tariffs on everything you see, but the Chinese government can force employees to work for less. And who wants to start paying MORE for the same stuff? That’s the other side of import duties that people fail to talk about — sure we can jack up the price of ‘stuff’ that comes in from overseas so domestically produced items are competitive … but unless you’re getting a serious raise to go with it, that means items become less affordable. Apart from political change, some manufacturing is apt to move back to personal production (i.e. I’ll 3D print any cheap plastic junk we used to ship in from overseas). And that will negatively impact some of the BRIC economies. The move to personal production may benefit American companies — they design the products, license out the print file, and you make it (or use the 3D printer at your public library to print it, or go to the 3D document centre at Staples).

Until about 3AM today, I thought we’d be seeing a similar apolitical shift in energy markets. The renewable energy tax credits got extended in a Congress where even the trivial faced unimaginable opposition. Personal energy production makes electric vehicles a lot more enticing — paying 20$ a week to the petrol station or 20$ a week to the electric company … not much immediate, tangible benefit. But using the electricity that I’m producing v/s paying 20$ a week to the petrol station – that’s a whole different story.

That shift had worrying geo-political implications — the Middle East isn’t stable, but the area is useful and there’s incentive on both sides to maintain some semblance of order. As demand for oil shifts, incentives change. Odd, to me, that someone who wants to levy tariffs to make American products competitive doesn’t agree with leaving tax incentives in place to promote domestically manufactured clean energy solutions. “Clean” coal, sure … but don’t give ’em a tax incentive to buy solar panels manufactured out on the West coast.

 

 

Really Wacky Exchange (ActiveSync) Error

My husband changed his Active Directory password. Routine enough – we’ve got 15k accounts at the office and require a password change every 90 days. That’s 150-200 people changing their password every day. They get themselves locked out a lot (mobile devices, cached workstation credentials, and a host of other unique places people manage to store their creds), but it’s trivial to unlock an individual user.

*Except* — after the account was unlocked, his Windows 10 mail client updated properly and was interacting with the Exchange server. Android, however, still wouldn’t accept his new password. If he typed the wrong thing, it would say invalid password. But whenever he typed the right thing, he got an error indicating the phone and tablet were unable to communicate with the server. Which was bogus — I could see the communication coming across the reverse proxy server. With 200 codes — although you can have a very successful HTTP call deliver an application error message. But it wasn’t like he couldn’t COMMUNICATE with the server. He turned sync off on the phones to avoid getting locked out again, and in the process of troubleshooting ended up deleting all of his accounts hosted on our Exchange 2013 server.

I looked through all of the event logs, Exchange logs … nothing interesting. In desperation, I enabled the individual user ActiveSync logging:

Set-CASMailbox mailNickName -ActiveSyncDebugLogging:$true

Had him attempt to add the mailbox profile again, and dropped the log myself:

Get-ActiveSyncDeviceStatistics -Mailbox mailNickName -GetMailboxLog:$true -NotificationEmailAddress mysmtp@mydomain.ccTLD

Bingo! An exception in the provisioning (Microsoft-Server-ActiveSync?Cmd=Provision) call — I see the phone information come across, the mobile device gets partially added to his account (no OS, phone number, carrier type information … but if you go into OWA and remove the mobile device, an Android device gets added). Error:

Command_WorkerThread_Exception :
— Exception start —
Exception type: System.IO.FileLoadException
Exception message: Could not load file or assembly ‘Microsoft.Exchange.Configuration.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Exception level: 0
Exception stack trace: at Microsoft.Exchange.AirSync.DeviceInformationSetting.ProcessSet(XmlNode setNode)
at Microsoft.Exchange.AirSync.DeviceInformationSetting.Execute()
at Microsoft.Exchange.AirSync.ProvisionCommand.Microsoft.Exchange.AirSync.IProvisionCommandHost.ProcessDeviceInformationSettings(XmlNode inboundDeviceInformationNode, XmlNode provisionResponseNode)
at Microsoft.Exchange.AirSync.ProvisionCommandPhaseOne.Process(XmlNode provisionResponseNode)
at Microsoft.Exchange.AirSync.ProvisionCommand.ExecuteCommand()
at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows…
Exception type: System.IO.FileLoadException
Exception message: The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
Exception level: 1
Exception stack trace:
— Exception end —

Now that is an error I’ve never seen before. As a programmer, I know what it means … you’ve basically got some components that don’t match another. But … huh? He changed his password. Connected to the Exchange server directly (instead of remotely viewing logs & files) and saw Windows Update had dropped files and a reboot was pending. Which … some files replaced, others staged for replacement pending a reboot. *That* is some components not matching others. Rebooted our box, and voila … registration goes through, mailbox sync started.

I don’t know how many people allow auto-updates with a manual reboot on a production enterprise server (we manually patch and reboot during a scheduled maintenance window) where this could happen … but evidently Windows Update can get your Exchange server into a state where already configured clients are able to send and receive mail. But clients are unable to update passwords, and new clients cannot be configured.

Zoneminder Setup

I just installed ZoneMinder tonight. I don’t know if I missed a section in the documentation or something’s just missing — there’s doc for getting the build onto your box (or building your own from source). I didn’t want to install the package and all of its dependencies from their repo, so I manually installed the prereqs from the standard Fedora repositories. Installed the zoneminder rpm and switched from the ‘installation guide’ to the ‘user guide’. Which starts out on a web site. Umm … what web site? There’s definitely something missing here. Their package drops a config file in /etc/httpd/conf.d … *but* it presupposes that it is a dedicated server (which, in fairness, is the recommended configuration).

I edited the zoneminder.conf file and threw the whole thing in a VirtualHost tag, added an SSL cert for the hostname I’m using, and restarted Apache. OK, that’s better … I get *something*. Unfortunately ‘something’ is a massive MySQL error.

Some searching (rpm -ql zoneminder, then search through the files the package installed for something that looks good) yielded a config file at /etc/zm/zm.conf. Went in there, defined a database, user, and password. Created said database & user in mysql & assigned rights. Umm, stranger database error that leads me to believe the database hasn’t been initialized. Oops. LMGTFY. Found a SQL file at /usr/share/zoneminder/db/zm_create.sql and imported *that* … wasn’t sure if it was as simple as changing the zm database name in the SQL file to what I wanted or not, so I just used their default database name. Went back into MySQL and assigned the user I’d created earlier rights to the zm database and dropped *my* database. (Note, if you don’t use PHP date ‘stuff’, you may need to define the time zone in your php.ini file … I’d already set a default for other purposes).

Refresh and voila, I’ve got a system. You’ve got to click ‘start’ at the top for it to, well, start pulling in video feeds. I set up a VERY basic link to our WansView camera — ffmpg type, source is the full videostream.cgi URL, and remote method is “RTP/Unicast”. I can now see the video from our camera via the web site. WooHoo!

Now we need to get the motion detection set up & play with all of the options 🙂

USN Rollback

I had to recover my domain controller from the Hyper-V image backup. There’s some protection build into AD which prevents just randomly reverting a server. When you’ve got a larger domain, the built-in protection after unsupported restoration procedures serves a purpose. Pausing netlogon avoids having users log on against bad data. Disabling replication avoids propagating bad information out to the remainder of the network. The solution is simple – demote the DC, promote it again, and the DC returns to service. But when you have a single domain controller in a single domain in a single forest … well, there’s no other data around. What the recovered DC has is as good as it’s going to get (i.e. a change from 2AM is lost when I revert to my 10PM backup). And taking the entire domain down and building it overkill. You can, instead, basically tell AD to go with it. From the MS documentation:

To restore a previous version of a virtual domain controller VHD without system state data backup

  1. Using the previous VHD, start the virtual domain controller in DSRM, as described in the previous section. Do not allow the domain controller to start in normal mode. If you miss the Windows Boot Manager screen and the domain controller begins to start in normal mode, turn off the virtual machine to prevent it from completing startup. See the previous section for detailed instructions for entering DSRM.
  2. Open Registry Editor. To open Registry Editor, click Start, click Run, type regedit, and then click OK. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Yes. In Registry Editor, expand the following path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters. Look for a value named DSA Previous Restore Count. If the value is there, make a note of the setting. If the value is not there, the setting is equal to the default, which is zero. Do not add a value if you do not see one there.
  3. Right-click the Parameters key, click New, and then click DWORD (32-bit) Value.
  4. Type the new name Database restored from backup, and then press ENTER.
  5. Double-click the value that you just created to open the Edit DWORD (32-bit) Value dialog box, and then type 1 in the Value data box. The Database restored from backup entry option is available on domain controllers that are running Windows 2000 Server with Service Pack 4 (SP4), Windows Server 2003 with the updates that are included in article 875495 (http://go.microsoft.com/fwlink/?LinkId=137182) in the Microsoft Knowledge Base installed, and Windows Server 2008.
  6. Restart the domain controller in normal mode.
  7. When the domain controller restarts, open Event Viewer. To open Event Viewer, click Start, click Control Panel, double-click Administrative Tools, and then double-click Event Viewer.
  8. Expand Application and Services Logs, and then click the Directory Services log. Ensure that events appear in the details pane.
  9. Right-click the Directory Services log, and then click Find. In Find what, type 1109, and then click Find Next.
  10. You should see at least an Event ID 1109 entry. If you do not see this entry, proceed to the next step. Otherwise, double-click the entry, and then review the text confirming that the update was made to the InvocationID:

 

  • Active Directory has been restored from backup media, or has been configured to host an application partition. 
The invocationID attribute for this directory server has been changed. 
The highest update sequence number at the time the backup was created is <time>

InvocationID attribute (old value):<Previous InvocationID value>
InvocationID attribute (new value):<New InvocationID value>
Update sequence number:<USN>

The InvocationID is changed when a directory server is restored from backup media or is configured to host a writeable application directory partition.
  • Close Event Viewer.
  • Use Registry Editor to verify that the value in DSA Previous Restore Count is equal to the previous value plus one. If this is not the correct value and you cannot find an entry for Event ID 1109 in Event Viewer, verify that the domain controller’s service packs are current. You cannot try this procedure again on the same VHD. You can try again on a copy of the VHD or a different VHD that has not been started in normal mode by starting over at step 1.
  • Close Registry Editor.

 

After following the instructions from Microsoft, I still had a problem — my DC has replication turned off & netlogon comes up paused. In regedit, locate HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters and delete the “Dsa Not Writable” key (value: dword:00000004). In a command prompt, run the following:

 

repadmin /options dchostname.example.com -DISABLE_OUTBOUND_REPL
repadmin /options dchostname.example.com -DISABLE_INBOUND_REPL

Reboot the DC. When it starts, netlogon should be running and replication.