{"id":3490,"date":"2018-10-10T16:06:18","date_gmt":"2018-10-10T21:06:18","guid":{"rendered":"http:\/\/lisa.rushworth.us\/?p=3490"},"modified":"2020-04-29T22:54:14","modified_gmt":"2020-04-30T03:54:14","slug":"debugging-an-active-directory-custom-password-filter","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=3490","title":{"rendered":"Debugging An Active Directory Custom Password Filter"},"content":{"rendered":"

A few years ago, I implemented a custom password filter<\/a> in Active Directory. At some point, it began accepting passwords that should be rejected. The updated code is available at https:\/\/github.com\/ljr55555\/OpenPasswordFilter<\/a> and the following is the approach I used to isolate the cause of the failure.<\/p>\n

 <\/p>\n

Technique #1<\/em><\/strong> — Netcap on the loopback There are utilities that allow you to capture network traffic across the loopback interface. This is helpful in isolating problems in the service binary or inter-process communication. I used RawCap<\/a> because it’s free for commercial use. There are other approaches<\/a> too – or consult the search engine of your choice.<\/p>\n

\"\"<\/a><\/p>\n

The capture file can be opened in Wireshark. The communication is done in clear text (which is why I bound the service to localhost), so you\u2019ll see the password:<\/p>\n

\"\"<\/a><\/p>\n

And response<\/p>\n

\"\"<\/a><\/p>\n

To ensure process integrity, the full communication is for the client to send \u201ctest\\n\u201d then<\/em> \u201cPasswordToTest\\n\u201d, after which the server sends back either true or false.<\/p>\n

Technique #2<\/em><\/strong> — Debuggers Attaching a debugger to lsass.exe is not fun. Use a remote debugger — until you tell the debugger to proceed, the OS is pretty much useless. And if the OS is waiting on you to click something running locally, you are quite out of luck. A remote debugger allows you to use a functional operating system to tell the debugger to proceed, at which time the system being debugged returns to service.<\/p>\n

Install the SDK debugging utilities on your domain controller and another box. Which<\/em> SDK debugging tool? That\u2019s going to depend on your OS. For Windows 10 and Windows Server 2012 R2, the Windows 10 SDK (Debugging Tools For Windows 10) work. https:\/\/docs.microsoft.com\/en-us\/windows-hardware\/drivers\/debugger\/debugger-download-tools<\/a> or Google it.<\/p>\n

On the domain controller, find the PID of LSASS and write it down (472 in my example). Check the IP address of the domain controller (10.104.164.110 in my example).<\/p>\n

From the domain controller, run:<\/p>\n

dbgsrv.exe -t tcp:port=11235,password=s0m3passw0rd<\/p>\n

\"\"<\/a><\/p>\n

Where port=11235 can be any un-used port and password=s0m3passw0rd can be whatever string you want \u2026 you\u2019ve just got to use the same values when you connect from the client. Hit enter and you\u2019ve got a debugging server. It won\u2019t look like it did anything, but you\u2019ll see the port bound on netstat<\/p>\n

\"\"<\/a><\/p>\n

And the binary running in taskman<\/p>\n

\"\"<\/a><\/p>\n

From the other box, run the following command (substituting the correct server IP, port, password, and process ID):<\/p>\n

windbg.exe -y “srv:c:\\symbols_pub*http:\/\/msdl.microsoft.com\/downloads\/symbols<\/a>” -premote tcp:server=10.104.164.110,port=11235,password=s0m3passw0rd -p 472<\/p>\n

This attaches your WinDBG to the debugging server & includes an internet-hosted symbol path. Don\u2019t worry when it says \u201cDebugee not connected\u201d at the bottom \u2013 that just means the connection has not completed. If it didn\u2019t connect at all (firewall, bad port number, bad password), you\u2019d get a pop-up error indicating that the initial connection failed.<\/p>\n

\"\"<\/a><\/p>\n

Wait for it … this may take a long time to load up, during which time your DC is vegged. But eventually, you’ll be connected. Don\u2019t try to use the DC yet \u2013 it will just seem hung, and trying <\/em>to get things working just make it worse. Once the debugger is connected, send ‘g’ to the debugger to commence \u2013 and now the DC is working again.<\/p>\n

Down at the bottom of the command window, there\u2019s a status (0:035> below) followed by a field where you enter commands. Type the letter g<\/strong> in there & hit enter.<\/p>\n

\"\"<\/a><\/p>\n

The status will then say \u201cDebuggee is running \u2026\u201d and you\u2019re server is again responsive to user requests.<\/p>\n

\"\"<\/a><\/p>\n

When you reach a failing test, pause the debugger with a break command (Debug=>Break, or Ctrl-Break) which will veg out the DC again. You can view the call stack, memory, etc.<\/p>\n

To search the address space for an ASCII string use:<\/p>\n

!for_each_module s -[1]a ${@#Base} L?${@#Size}\u00a0 \"bobbob\"<\/pre>\n
\"\"<\/a><\/p>\n
Where \u201cbobbob\u201d is the password I had tested.<\/p>\n
Alternately, run the \u201cpsychodebug\u201d build where LARGEADDRESSAWARE is set to NO and you can search just<\/em> the low 2-gig memory space (32-bit process memory space):<\/p>\n
s -a 0 L?80000000 \"bobbob\"<\/pre>\n
* The true\/false server response is an ASCII string, not a Boolean. *<\/p>\n
Once you have found what you are looking for, \u201cgo\u201d the debugger (F5, Debug=>Go, or \u00a0\u2018g\u2019) to restore the server to an operational state. Break again when you want to look at something.<\/p>\n
To disconnect, break and send \u201cqd\u201d to the debugger (quit and detach). If you do not detach with qd, the process being debugged terminates. Having lsass.exe terminate really freaks out the server, and it will go into an auto-recovery \u201cI\u2019m going to reboot in one minute\u201d mode. It\u2019ll come back, but detaching without terminating the process is a lot nicer.<\/p>\n
Technique #3<\/em><\/strong> \u2013 Compile a verbose version. I added a number of event log writes within the DLL (obviously, it’s not a good idea in production to log out candidate passwords in clear text!). While using the debugger will get you there eventually, half an hour worth of searching for each event (the timing is tricky so the failed event is still in memory when you break the debugger) \u2026 having each iteration write what it was doing to the event log was FAAAAAR simpler.<\/p>\n
And since I\u2019m running this on a dev DC where the passwords coming across are all generated from a load sim script \u2026 not exactly super-secret stuff hitting the event log.<\/p>\n
Right now, I\u2019ve got an incredibly verbose DLL on APP556 under d:\\tempcsg\\ljr\\2\\debugbuild\\psychodebug\\ \u2026 all of the commented out event log writes from https:\/\/github.com\/ljr55555\/OpenPasswordFilter<\/a> aren\u2019t<\/em> commented out.<\/p>\n
Stop the OpenPasswordFilter service, put the verbose DLL and executables in place, and reboot. Change some passwords, then look in the event viewer.<\/p>\n
ERROR events are actual problems that would show up either way. INFORMATION events are extras. I haven\u2019t bothered to learn how to properly register event sources in Windows yet \ud83d\ude42 You can find the error content at the bottom of the “this isn’t registered” complaint:<\/p>\n
\"\"<\/a><\/p>\n
You will see events for the following steps:<\/p>\n
DLL starting CreateSocket<\/p>\n
About to test password 123paetec123-Dictionary-1-2<\/p>\n
Finished sendall function to test password123paetec123-Dictionary-1-2<\/p>\n
Got t on test of paetec123-Dictionary-1-2<\/p>\n
The final line will either say \u201cGot t\u201d for true or \u201cGot f\u201d for false.<\/p>\n
Technique #4<\/em><\/strong> \u2013 Running the code through<\/em> the debugger. Whilst there\u2019s no good way to get the \u201cNotification Package\u201d hook to run the DLL through the debugger, you can install Visual Studio on a dev domain controller and execute the service binary through the debugger. This allows you to set breakpoints and watch variable values as the program executes \u2013 which makes it a whole lot easier than using WinDBG to debug the production code.<\/p>\n
Grab a copy of the source code \u2013 we\u2019re going to be making some changes that should not be promoted to production, so I work on a temporary copy of the project and delete the copy once testing has completed.<\/p>\n
Open the project in Visual Studio. Right-click OPFService in the \u201cSolution Explorer\u201d and select \u201cProperties\u201d<\/p>\n
\"\"<\/a><\/p>\n
Change the build configuration to \u201cDebug\u201d<\/p>\n
\"\"<\/a><\/p>\n
Un-check \u201cOptimize code\u201d \u2013 code optimization is good for production run, but it will wipe out variable values when you want to see them.<\/p>\n
\"\"<\/a><\/p>\n
Set a breakpoint on execution \u2013 on the OPFDictionary.cs file, the loop checking to see if the proposed word is contained in the banned word list is a good breakpoint. The return statements are another good breakpoint as it pauses program execution right<\/em> before a password test iteration has completed.<\/p>\n
\"\"<\/a><\/p>\n
Build the solution (Build=>Build Solution). Stop the Windows OpenPasswordFilter service.<\/p>\n
Launch the service binary through the debugger (Debug=>Start Debugging).<\/p>\n
Because the program is being run interactively instead of through a service, you\u2019ll get a command window that says \u201cPress any key to stop the program\u201d. Minimize this.<\/p>\n
\"\"<\/a><\/p>\n
From a new<\/em> command prompt, telnet to localhost on port 5995 (the telnet client is not installed by default, so you may need to use \u201cTurn Windows features on or off\u201d and enable the telnet client first).<\/p>\n
\"\"<\/a><\/p>\n
Once the connection is established, use CTRL and ] to get into the telnet command prompt. Type set localecho<\/strong> \u2026 now you\u2019ll be able to see what you are typing.<\/p>\n
\"\"<\/a><\/p>\n
Hit enter again and you\u2019ll return to the blank window that is your telnet client. Type test<\/strong> and hit enter. Then type a candidate password and hit enter.<\/p>\n
\"\"<\/a><\/p>\n
Program execution will pause at the breakpoint you\u2019ve set. Return to Visual Studio. Select Debug =>Window=>Locals to open a view of the variable values<\/p>\n
\"\"<\/a><\/p>\n
View the locals at the breakpoint, then hit F5 if you want to continue.<\/p>\n
\"\"<\/a><\/p>\n
If you\u2019re set breakpoints on either of the return statements, program execution will also pause before the return \u2026 which gives you an opportunity to see which return is being used & compare the variable values again.<\/p>\n
\"\"<\/a><\/p>\n
In this case, I submitted a password that was<\/em> in the banned word list, so the program rightly evaluated line 56 to true and returns true.<\/p>\n
\"\"<\/a><\/p>\n\n\n
\n    
\n        \n        \n        \n        \n        \n        \n        \n        \n        \n        \"\"\n    <\/div>\n<\/form>\n\n\n","protected":false},"excerpt":{"rendered":"
A few years ago, I implemented a custom password filter in Active Directory. At some point, it began accepting passwords that should be rejected. The updated code is available at https:\/\/github.com\/ljr55555\/OpenPasswordFilter and the following is the approach I used to isolate the cause of the failure.   Technique #1 — Netcap on the loopback There …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[68,366,365,672,32],"class_list":["post-3490","post","type-post","status-publish","format-standard","hentry","category-system-administration","tag-active-directory","tag-ad-password-filter","tag-custom-password-filter","tag-password-filter","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/3490","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3490"}],"version-history":[{"count":2,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/3490\/revisions"}],"predecessor-version":[{"id":6350,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/3490\/revisions\/6350"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3490"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3490"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3490"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}