Category: Technology

Tracking Electrical Usage With SmartThings and AeonLabs Home Energy Meters

When we started shopping for solar generation installations, how much electricity we can consume was a challenging question. We were replacing our HVAC system, so “look at the last 12 months of electric bills” wasn’t an approach that would yield valid data. What we needed was a way to see electrical consumption for the next two or three weeks once our new HVAC system was installed.

We purchased several AeonLabs Home Energy Meters (HEM) and have been using them to track our power consumption for almost six months now. The HEM’s are set up in SmartThings & have a SmartApp-HEMLogger “SmartApp” attached to them that posts data to a MySQL table on our server via a web form (myURL needs to be … well, your URL).

Install a quick MySQL server (does not need to be Internet accessible) and a web server / programming language of your choice combination (*does* need to be Internet accessible – we are using Apache and PHP).

Create the database and a table to hold your energy data:

mysql> describe EnergyMonitors;
+-----------+-------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+-------------------+----------------+
| keyID | int(11) | NO | PRI | NULL | auto_increment |
| monitorID | varchar(50) | YES | | NULL | |
| clampID | varchar(50) | YES | | NULL | |
| eventTime | timestamp | NO | | CURRENT_TIMESTAMP | |
| kwatts | double | YES | | NULL | |
| kwhours | double | YES | | NULL | |
+-----------+-------------+------+-----+-------------------+----------------+
6 rows in set (0.00 sec)

Create an ID within your database that has read/write permission to this table. I create another ID that has read-only access (pages displaying data use this ID, the page to post data uses the read/write ID).

We also track temperature — ideally, we’d be able to compare power consumption based on temperature *and* sunlight (we use a lot less power on a cold sunny day than I expect … just don’t know how much less) … but I’m not there yet. Currently, the weather database only holds temperature:

mysql> describe weather;
+--------------+-----------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------+------+-----+-------------------+-----------------------------+
| recordedTime | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| temperature | int(11) | YES | | NULL | |
| id | int(11) | NO | PRI | NULL | auto_increment |
+--------------+-----------+------+-----+-------------------+-----------------------------+
3 rows in set (0.01 sec)

Since we brought our Bloomsky online, I use the Bloomsky API to record temperature. Prior to that, I was parsing the XML from NWS’s closest reporting station’s – for Cleveland, that is  http://w1.weather.gov/xml/current_obs/KCLE.xml … there’s probably one near you. I grabbed both observation_time_rfc822 and temp_f in case observations were not updated regularly – the observation time was stored as the recordedTime. The temperature recording script is in cron as an hourly task.

You also need a small web page where SmartThings can post data — in PHP, I have

<?php
if(!$_POST["monitorID"] || !$_POST["clampID"] ){
echo "<html><body>\n";
echo "<form action=\"";
echo $_SERVER['PHP_SELF'];
echo "\" method=\"post\">\n";
echo "Monitor ID: <input type=\"text\" name=\"monitorID\"><br>\n";
echo "Clamp ID: <input type=\"text\" name=\"clampID\"><br>\n";
echo "KW Value: <input type=\"text\" name=\"kwatts\"><br>\n";
echo "KWH Value: <input type=\"text\" name=\"kWHours\"><br>\n";
echo "<input type=\"submit\">\n";
echo "</form>\n";
echo "</body></html>\n";
}
else{
$strMeter = $_POST["monitorID"];
$strClamp = $_POST["clampID"];
$strKWValue = $_POST["kwatts"];
$strKWHValue = $_POST["kWHours"];
print "<pre>Meter: $strMeter\nClamp: $strClamp\nKW: $strKWValue\nKWH: $strKWHValue\n</pre>\n";
$servername = "DatabaseHostname";
$username = "MySQLUID";
$password = 'MySQLPassword';
$dbname = "HomeAutomation";
$conn = new mysqli($servername, $username, $password, $dbname);
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
if($strKWHValue && $$strKWValue){
$sql = "INSERT INTO EnergyMonitors (monitorID, clampID, kwatts, kwhours) VALUES ('$strMeter', '$strClamp', '$strKWValue', '$strKWHValue')";
}
elseif($strKWHValue){
$sql = "INSERT INTO EnergyMonitors (monitorID, clampID, kwhours) VALUES ('$strMeter', '$strClamp', '$strKWHValue')";
}
else{
$sql = "INSERT INTO EnergyMonitors (monitorID, clampID, kwatts) VALUES ('$strMeter', '$strClamp', '$strKWValue')";
}
if ($conn->query($sql) === TRUE) {
echo "New record created successfully";
}
else {
echo "Error: " . $sql . "<br>" . $conn->error;
}

$conn->close();
}
?>

DatabaseHostname, MySQLUID and MySQLPassword need to be yours too. Since posted data is meant to come from a trusted source (a source where I’ve supplied the URL) and in a known good format, these are quick INSERT statements *not* the safest.

Check in your database that you are getting data, then let it run for a few hours. Once you have a little bit of history, you can start viewing your energy usage. Link it into Excel/Access using MyODBC for ad-hoc reporting, write your own code to do exactly what you want, use a generic charting package capable of reading MySQL data …

I am using pChart to display data – the chart I use most frequently is the stacked bar chart for kWH and a line chart for temperature. Here is the PHP code which generates this PNG: energyUsage-StackedBarChart-Flat – again, DatabaseHostname, MySQLUID, and MySQLPassword need to be yours.

StackedBarChart-KWHAndTemp

We no longer have our heat pump, air handler, and heat strips being monitored – but for periods where there is data from the other sources, we had several segments to our energy usage report (“other” is the report from the HEM on our mains MINUS all of the other reporting segments). You can yank all of the non-mains segments (or change them to be whatever sub-segments you are monitoring. The monitor ID comes from the HEM name in SmartThings – since those are user-configured, I have the names hard coded. You *could* hard code the Mains and then use “select distinct” to get a list of all the others and make the code more flexible.).

Short term charts (past 24 hours or so) can render out real-time, but longer term views take a long time to load. Since we’re looking more for trends and totals – being up to the second isn’t critical. I’ve got cron tasks that generate out PNG files of the charts.

Two enhancements for a nothing-doing rainy day – adding authentication to the HEM Logger (SmartThings posts from multiple netblocks, so there isn’t a good way to IP source restrict access to the post data page. Anyone with your URL could post rogue energy usage info into your database – or more likely try to hack your servers.) and add lumen to the weather data / graph displays.

Kerberos Authentication and LDAP Authorization In Apache

I’ve been authenticating users of Apache web sites against Active Directory using Kerberos for some time now. Installed krb5-workstation and mod_auth_kerb, configured /etc/krb5.conf for my specific domain, and added some config to the Directory section of the Apache config. Great if you just require valid-user (or require valid-user and then turn around and do some authorization within your web code using something like php_auth_user). Not so great, though, for restricting access to the site outside of web code. And I really didn’t want to code in an authorization function when my web server should be able to do that for me.

I FINALLY got kerberos authentication working in Apache with an LDAP authorization component. Turns out the  mod_auth_kerb version 5.1 that was available from the Yum repository is terribly buggy  – like not usable in this instance buggy. KrbLocalUserMapping did not consistently remove the realm component. I’d hit a site and it would know who I am, click a link and come across as me@REALM.TLD and get access denied errors, click refresh and get in because it knew I was me again. Or not. More than 50% failure rate.I built the 5.4 version from http://modauthkerb.sourceforge.net/ and haven’t had a problem since.

I’m authenticating to Active Directory using the Kerberos module then authorizing against a group housed in an external LDAP directory. You can totally point your LDAP config toward Active Directory & use AD groups instead:

AuthType Kerberos
AuthName “Kerberos AD Test”
KrbAuthoritative off
KrbMethodNegotiate on
KrbMethodK5Passwd on
KrbServiceName HTTP/this.isyour.url.tld@EXAMPLE.COM
KrbAuthRealms EXAMPLE.COM
KrbLocalUserMapping On
Krb5Keytab /path/to/keytabs/keytab.file

AuthBasicAuthoritative On
AuthBasicProvider ldap
AuthLDAPURL “ldaps://ldap.example.com/o=BaseDN?uid?sub?(&(cn=*))”
AuthLDAPBindDN “YOUR SERVICE ACCOUNT HERE”
AuthLDAPBindPassword “YOUR BIND PWD HERE”

AuthLDAPGroupAttribute uniqueMember
AuthLDAPGroupAttributeIsDN on
require ldap-group cn=Website Test,ou=groups,o=BaseDN

 

WooHoo! I hit the site from my domain-member computer, it knows I am LisaR. It then turns around and finds an LDAP user matching uid=LisaR and grabs the user’s fully qualified DN (because AuthLDAPGroupAttributesIsDN is ‘on’ here … if you are using just uids in your member list, that would be off). It then verifies that the fully qualified DN is a member of the Website Test group.

Now I’m trying to figure out how to let the user log in without supplying a realm (not everyone’s in the domain … and they need to be able to log in too. Works fine right now, provided they input their username as uid@REALM.TLD).

Same word, different meaning

I issue a lot of certificates from our internal company certificate authority – they’re free, and since I can publish trusted root signers out to the domain, they’re trusted to anyone who would be using the site. You can type pseudo-random values into your request and my CA will issue a certificate for you.
Today, though, I needed a certificate for a site that would be used by non-employees. People who are not subject to my domain GPO. People who do not trust my CA. So I did what everyone else does – got a real certificate 🙂
I generated my CSR (and actually typed in good data – my server is in Conway, AR, USA type location instead of Z or A). Went out to Verisign’s site … “The CSR contains an invalid state. Please click your browser’s Back button and enter a new CSR.”
Thought the CSR might have gotten corrupted somehow, so I tried again. Same result. Tried some different information – same result. Finally resorted to reading the instructions – locality names may not contain abbreviations. D’oh. State like organized political community not state like condition.

Missing The Point

A security researcher used a modified cat6 cable and default creds on airline seat electronic boxes to compromise flight control systems on an aircraft. That’s really bad, and the FBI is investigating the crime. But why is it that no one seems to care that (1) SEB’s ride on the same network as flight control systems, (2) there’s a default password no one has bothered to change, and (3) no one on the aircraft was in any way bothered by some dude digging around under the seat and messing with cables?

Seriously – in the system design meetings for a million dollar aircraft, someone thought it would be a good idea to save, what, a grand by having a single open network for all electronic components on the aircraft?!

And I sincerely hope the WiFi networks they’re starting to put on the aircraft are on an isolated network that has nothing to do with any of the flight control equipment. It’s one thing to notice a guy plugging into some box under his seat … a guy using his computer mid-flight, nothing to see there.

Linux Utility: xxd

When there’s something different between two files but you just cannot see it — xxd is a command line hex dump utility. You can even pair it with diff to see … oh, one has \r\n and the other just has \n

[lisa@FVP05 ljr]# diff -u <(xxd unix.txt) <(xxd dos.txt)
--- /dev/fd/63  2025-01-15 16:39:58.016083028 -0500
+++ /dev/fd/62  2025-01-15 16:39:58.018083031 -0500
@@ -1 +1 @@
-00000000: 4865 6c6c 6f21 0a                        Hello!.
+00000000: 4865 6c6c 6f21 0d0a                      Hello!..

Response Policy Zone (RPZ)

Years ago, Paul Vixie developed a component of the BIND DNS server that allowed server owners to easily override specific hostnames. We had done something similar for particularly bad hostnames — if your workstations use your DNS servers, you just have to declare yourself the name server for a domain that has the same name as the hostname you want to block (i.e. I become the NS record for forbidden.google.com and my clients are able to resolve all other records within the google.com zone, but when they resolve forbidden.google.com … they get whatever I provide). I usually did this to route traffic over a B2B VPN – provided the private IP address instead of the public IP provided by the domain owner’s name servers. But for a few really bad malware variants, I overrode their hostname. Problem was the technique wasn’t exactly easy. Every single host required a new DNS zone be created, configured on your DNS servers, and (at least in BIND) the service restarted.

Response Policy Zone was pushed as a functionality that would allow service providers (ISPs). That’s not a use case I forsee (it’s a lot of manual work), but it has become an important component of our company’s network security. Hosting an RPZ domain allows us to easily add new overrides for B2B VPN connected hosts. But it also means we can override hostnames that appear in phishing e-mail campaigns, malware hosts, infected web sites … basically anything we don’t want employees accessing.

Stopping clients from accessing infected sites is a great thing; but for hostnames that are indicative of a compromised box (i.e. there’s a difference between an employee clicking on a link within their e-mail that links them to a specific host and someone having malware on their box that automatically contacts a specific host), we set the IP address for the hostname to a honeypot.

The honeypot is bound to all unused ports on the host (there aren’t a lot of used ports on it), logs all contact to a database, then basically hangs the connection. We have a scheduled job that looks at the contact log and opens a ticket to the desktop support team to investigate the compromised host.

Baked French Toast Casserole with Maple Syrup

Ingredients

  • 1 loaf French bread (~16 ounces)
  • 8 large eggs
  • 1.5 cups heavy whipping cream
  • 1.5 cups milk
  • 2 tablespoons maple syrup
  • 1 teaspoon vanilla extract
  • 1/4 teaspoon ground cinnamon
  • 1/4 teaspoon ground nutmeg
  • Dash salt

Directions

Slice bread into 20 slices, 1-inch each. Arrange slices in a generously buttered 9 by 13-inch baking dish in 2 rows, overlapping the slices. In a large bowl, combine the eggs, half-and-half, milk, sugar, vanilla, cinnamon, nutmeg and salt and beat with a rotary beater or whisk until blended but not too bubbly. Pour mixture over the bread slices, making sure all are covered evenly with the milk-egg mixture. Spoon some of the mixture in between the slices. Cover with foil and refrigerate overnight.

The next day, preheat oven to 350 degrees F.

Bake for 40 minutes, until puffed and lightly golden. Serve with maple syrup, fresh fruit, and fresh whipped cream.

 

Anatomy of an LDAP Filter

LDAP filters are searches. Equality tests are supported for any attribute — attribute=value. Most attributes are case insensitive (but check your schema definition to verify!), and you can perform both exact matches (attribute=value) and sub-string matches (attribute=value*). While you will see * used in substring searches, LDAP filters do not support regex pattern matching. Some attributes support greater than and less than comparisons as well. A filter like (modifyTimestamp>=20140811000000Z) finds any object modified since 11 August 2014. This is useful when you are processing directory records and don’t want to look at any that haven’t changed since the last time your batch ran.

Tests are combined using Boolean operators. The three operators — AND (&), OR (|), and NOT (|) can be grouped and nested within parenthesis to from complex queries.

Your directory may support extensible matching — Active Directory does not. You may be able to find objects in the OUName OU using “ou:dn:=OUName”.

Your directory may support approximate matching — find close matches using “givenName=~Tim” where Tim and Timmy are returned.

To better understand an LDAP filter, decompose it into its sub-components. An example filter is:

(&(|(uid=e0*)(uid=n9*))(!(homeDirectory=*))(|(memberOf=cn=group1,ou=groups,o=example)(memberOf=cn=group2,ou=groups,o=example)))

This filter becomes

(&
     (|(uid=e0*)(uid=n9*))
     (!(homeDirectory=*))
     (|(memberOf=cn=group1,ou=groups,o=example)(memberOf=cn=group2,ou=groups,o=example))
)

The three sub-groups are AND’d — for an object to match the filter, it must meet all three sets of criterion.

Then decompose the first of the three sub-groups.

(|
     (uid=e0*)
     (uid=n9*)
)

This group uses an OR operator — the value of the uid attribute needs starts with e0 OR the value of the uid attribute starts with n9. You don’t need to use the same attribute in with the OR operator. I could use (|(st=OH)(title=Engineer)) to find all records where st is OH or title is Engineer.

The second sub-group has a single sub-component. The filter (homeDirectory=*) means “the value of homeDirectory is not NULL”. There is a NOT operator around the comparison — so we have the value of homeDirectory is NULL.

Decomposing the third sub-group:

(|
     (memberOf=cn=group1,ou=groups,o=example)
     (memberOf=cn=group2,ou=groups,o=example)
)

This group uses an OR operator — the value of memberOf is cn=group1,ou=groups,o=example OR the value of memberOf is cn=group2,ou=groups,o=example

The complete filter, then, finds records where

the value of the uid attribute needs starts with e0 OR the value of the uid attribute starts with n9

AND

the value of homeDirectory is NULL

AND

the value of memberOf is cn=group1,ou=groups,o=example OR the value of memberOf is cn=group2,ou=groups,o=example

 

Bulk LDAP Operations

LDIF – Directory Import and Export using LDIFDE.EXE and LDAPMODIFY

 

You can obtain ldifde.exe from any existing domain controller – copy \\dcname\c$\winnt\system32\ldifde.exe  to your SYSTEM32 folder. The ldapmodify command is part of the openldap-clients package on Linux. Windows builds of the openldap clients are available. The data being imported is essentially the same, just the command line to invoke the program differs.

Using LDIF files to update LDAP data is facilitated if you know the directory schema attributes, especially those associated to the user object class.  Active Directory schema is well documented on MSDN – base Active Directory schema can be found at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/adschema/adschema/active_directory_schema_site.asp and the extensions made by Exchange are documented at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wss/wss/wss_ldf_AD_Schema_intro.asp

LDIFDE is a command line program which runs with currently logged on user’s credentials – this means that your ID can write changes to AD using LDIFDE.  Please do not play with this program in the production Active Directory domain but rather test writing to a test domain.

 

LDIF Export

Exporting directory information is fairly straight forward:

 

ldifde –f filename.txt –d “ou=base,DC=windstream,DC=com” –p subtree –r “(&(attribute=value)(otherattribute=othervalue))” –s domaincontroller.windstream.com –l “attribute1, attribute2, attribute3 …”

 

-f File to contain exported data
-d Search base
-p Search scope
-r RFC-2254 compliant filter
-s Domain controller from which to obtain data
-l Attributes to be returned (eliding this command will return values for all attributes)

 

This will create a file named ljlexport.txt of all e####### users with email addresses whose accounts are located under WINDSTREAM.COM\WINDSTREAM\IT.  The file will contain, for each user, their logon ID (sAMAccountName), email address (mail), account status (userAccountControl), display name, and telephone number.

ldifde –f ljlexport.txt –d “ou=IT,ou=windstream,DC=windstream,DC=com” –r “(&(sAMAccountName=e*)(mail=*))” –s neohtwnnt836.windstream.com –l “sAMAccountName, mail, userAccountControl, displayName, telephoneNumber”

 

-r specifies the search filter and can become a rather complex query depending on what you are looking for — & is an AND filter, | is an OR filter.  ! can be used to find unmatched values and * works as a wildcard

“(&(mail=*)(!sAMAccountName=n99*))”        find all mail enabled accounts which are not N99’s for instance.

“(&(sAMAccountName=e0*)(!(employeeID=*)))”      Find all employee accounts with no employee ID specified

“(&(mail=*)(|(sAMAccountName=n99*)(sAMAccountName=g99*)))”            mail enabled accounts which are either n99’s or g99’s

“(&(objectClass=user)(objectCategory=person))”        Real user accounts, objectClass=user alone will return a lot of things you don’t believe are users J

“(&(objectClass=user)(objectCategory=person)(telephoneNumber=813*))”     Real user accounts with phone numbers in the 813 area code

“(&(objectClass=user)(objectCategory=person)(msExchHomeServerName=*SCARLITNT841))”       Real user with mailboxes on SCARLITNT841

 

-d specifies the search base (subtree search by default) – you can use “DC=windstream,DC=com” to get the entire directory or something like “ou=Central,ou=windstream,DC=windstream,DC=com” to just get users within the Central OU.

 

LDIF Import

Importing Directory Information is not so straight forward and again do not play with this program in the production Active Directory domain.  You need to create an ldif import file to make changes to objects.  A sample file content:

dn: cn=Landers\, Lisa,ou=GPOTest,ou=IT,ou=windstream,dc=windstreamtest,dc=com
changetype: modify
add: proxyAddresses
proxyAddresses: smtp:lisa@newtestdomain.windstream.com
-
replace: telephoneNumber
telephoneNumber: 501-905-4305
-
delete: mobile
mobile: 501-607-3750
-
delete: facsimileTelephoneNumber
-
 
dn: cn=Ahrend\, Sam,ou=IT,ou=windstream,dc=windstreamtest,dc=com
changetype: modify
replace: mDBUseDefaults
mDBUseDefaults: FALSE
-
replace: mDBStorageQuota
mDBStorageQuota: 190000
-
replace: mDBOverQuotaLimit
mDBOverQuotaLimit: 200000
-

Provided you have an import file, the syntax of the command is ldifde –i –v –k –y –f filename.txt  

 

-i LDIFDE import operation
-v Produce verbose output
-k Ignore constraint violations (and entry exists errors on add)
-y Lazy commit
-f File name to be imported

 

 

Changetype Add

A changetype of add is used when the entire object does not currently exist – this ldap operation creates a new object with the attributes specified in the stanza.

 

dn: cn=TestingGroup 10001,ou=testing,ou=it,DC=windstream,DC=com
changetype: add
cn: TestingGroup 10001
distinguishedName: cn= TestingGroup 10001,ou=testing,ou=it,DC=windstream,DC=com
name: TestingGroup10001
sAMAccountName: TestingGroup10001
objectClass: group
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=windstream,DC=com
groupType: -2147483646
managedBy: cn=Landers\, Lisa,OU=GPOTest,ou=IT,OU=WINDSTREAM,DC=windstream,DC=com
member: cn=Landers\, Lisa,OU=GPOTest,ou=IT,OU=WINDSTREAM,DC=windstream,DC=com
member: cn=Ahrend\, Sam, ou=IT,OU=WINDSTREAM,DC=windstream,DC=com
legacyExchangeDN: /o=WINDSTREAMEXCH/ou=First Administrative Group/cn=Recipients/cn=TestingGroup10001
mailNickname: TestingGroup10001
reportToOriginator: TRUE

 

This example will create an e-mail enabled global security group named “TestingGroup 10001” under windstream.com – it – testing.  Both Sam and I will be listed as members of  the group and I will be listed as the group owner.  Add operations can be chained together with just a blank line between them should you need to add multiple objects in batch.

 

Any mandatory attributes for the schema classes need to be included (handled by ldifde works too) or the add operation will fail.  Attributes not valid for the object classes will cause the operation to fail as well.  If an object already exists, no change will be made even if some of the attributes specified differ from the values within AD.

 

Changetype Delete

Delete is used to delete the entire object – so be extra careful here.  The syntax is quite simple – the DN of the object to be removed and a line that says “changetype: delete”.  Again, multiple operations can be chained together with just a blank line.

 

dn: cn=TestingGroup 10002,ou=testing,ou=it,DC=windstream,DC=com
changetype: delete
 
dn: cn=TestingGroup 10003,ou=testing,ou=it,DC=windstream,DC=com
changetype: delete
 
dn: cn=TestingGroup 10004,ou=testing,ou=it,DC=windstream,DC=com
changetype: delete
 
dn: cn=TestingGroup 10005,ou=testing,ou=it,DC=windstream,DC=com
changetype: delete
 

Changetype Modify

Modify is used to change attributes on an existing object.  Modify can be used to add, replace, or delete an attribute.  The example above has two different stanza’s (separated by a blank line).  Within each stanza several operations are made:

 

First to add another email address to the secondary email addresses.  For a multi-value attribute (member, proxyAddresses …) changetype: modify\nadd: attribute adds another value to the attribute.  For single-valued attributes modify/add will fail if a value is present.

dn: cn=Landers\, Lisa,ou=GPOTest,ou=IT,ou=windstream,DC=windstream,DC=com
changetype: modify
add: proxyAddresses
proxyAddresses: smtp:lisa@newtestdomain.windstream.com
-

The next operation replaces the telephone number with the value specified – this will overwrite the existing value.  Be careful not to replace multi-value attributes

replace: telephoneNumber
telephoneNumber: 501-905-4305
-

The next operation deletes the mobile phone number with the value specified – if the value does not match, a change is not made.  This can be used as a failsafe, in this case only delete my mobile telephone number if the value is what I expect it to be, or to remove entries from multi-value attributes.  Delete the member of the group which is the specific member listed without changing the other group members, for instance.

delete: mobile
mobile: 501-607-3750
-

The next operation deletes the fax number – regardless of content the value is removed.

delete: facsimileTelephoneNumber
-

A blank line separates the two stanzas and a new object is specified.  Again the modify/replace option is used which will change the attributes to the values specified.

dn: cn=Ahrend\, Sam,ou=IT,ou=windstream,DC=windstream,DC=com
changetype: modify
replace: mDBUseDefaults
mDBUseDefaults: FALSE
-
replace: mDBStorageQuota
mDBStorageQuota: 190000
-
replace: mDBOverQuotaLimit
mDBOverQuotaLimit: 200000
-
 
dn: cn=Landers\, Lisa,ou=IT,ou=windstream,DC=windstream,DC=com
changetype: modify
replace: mDBUseDefaults
mDBUseDefaults: TRUE
-

Changetype ModDN
ModDN changes the object’s distinguishedName.  This is interesting as it can be used to move users – this example would move my account into the Central OU under ACI

dn: CN=Landers\, Lisa,OU=GPOTest,OU=IT,OU=WINDSTREAM,,DC=windstream,DC=com
changetype: moddn
newrdn: CN= Landers \, Lisa,OU=Central,OU=WINDSTREAM,DC=windstream,DC=com
deleteoldrdn: 1

 

ModDN can also be used to rename the object’s display in administrative listings:

dn: CN=Landers\, Lisa,OU=GPOTest,OU=IT,OU=WINDSTREAM,,DC=windstream,DC=com
changetype: moddn
newrdn: CN= Landers\, Jane, OU=GPOTest,OU=IT,OU=WINDSTREAM,,DC=windstream,DC=com
deleteoldrdn: 1

You would of course want to modify/replace at least givenName and displayName on the object dn to avoid confusion – otherwise my middle name would appear in active directory users and computers but my first name in Outlook.  I would modify the attributes first – if you modify the DN first, you need to remember to use the new DN for subsequent attribute value changes.

dn: CN=Landers\, Lisa,OU=GPOTest,OU=IT,OU=WINDSTREAM,,DC=windstream,DC=com
changetype: modify
replace: givenName
givenName: Jane
-
replace: displayName
displayName: Landers, Jane
-
 
dn: CN=Landers\, Lisa,OU=GPOTest,OU=IT,OU=WINDSTREAM,,DC=windstream,DC=com
changetype: moddn
newrdn: CN= Landers\, Jane, OU=GPOTest,OU=IT,OU=WINDSTREAM,,DC=windstream,DC=com
deleteoldrdn: 1
-
 

Microsoft Directories – NT and Windows 2000/2003/2008/2012

Windows NT provided a limited repository for user id’s and passwords.  NT domain credentials had the advantage of providing single-sign-on access to other Microsoft resources such as file shares and Exchange.  Exchange itself housed a secondary directory, used for the “global address list” type details for Exchange accounts.  Address, phone number, manager, email addresses … basically anything other than the user’s ID and password were stored within the Exchange directory.  The Exchange directory then linked each account into an NT4 domain user account for logon credentials.

With Windows 2000, Microsoft integrated the two directories into Active Directory.  This allowed a more robust set of user details to be provided – and moved the LDAP compliant directory off the Exchange server onto the domain controllers.  Major changes were introduced in Active Directory – an increased maximum object count (from 40,000 to ten million in a single domain with a billions of objects in an AD forest), multi-master architecture, and attribute level replication being some of the key changes.

Data Store

Active Directory data is stored in ntds.dit.  ESE (extensible storage engine) is used to access the data within the database.  In addition to ntds.dit, there are several peripheral database files – edb.log is the current in-use transaction log file.  EDB#####.log may be present if the edb.log file has been filled.  EDB.CHK is the checkpoint file – this keeps track of which transactions have been committed to ntds.dit and a crash of the system will cause the transaction logs to be replayed from the pointer referenced in the chk file.  Res1.log and res2.log, ten meg in total, are placeholder files just in case should the server run out of disk space the files are removed to allow continued operation.

Within NTDS.DIT there are two main tables:

  • The link table – metadata for calculating linked values
  • The data table – actual domain data

There are four other tables about which no additional information will be provided here-in

  • System Table – metadata for the DSA-defined tables and indices
  • HiddenTable – DSA metadata
  • SDPropTable – Transiently stores Security Descriptor propagation, records are removed from table as propagation completes
  • MSysDefrag1 – ESE database table, not specific to AD

For linked attributes, the backlinked attribute is not modified directly but rather determined when it is queried.  As an example – Active Directory generates a reporting structure.  An object has a manager, but the “reports” listing is calculated based on object’s managers.  The linkID of a forward link attribute is always even and it’s associated backlinked attribute is always the forward linkID plus one (consequently also always odd).  A full list of forward/back link pairs can be generated by looking at the linkID values.

The data table contains three different naming contexts – the schema, the configuration, and the domain data.  These correspond to the three partitions shown in REPLMON – “cn=schema,cn=configuration,dc=windstream,dc=com”, “cn=configuration,dc=windstream,dc=com”, and “dc=windstream,dc=com”.  The term partition in Active Directory is used to indicate a naming context – in no way related to Novell’s use of the term to indicate a replication boundry.

The schema and configuration partitions are replicated to all domain controllers in a forest – since we only have one tree in the forest rendering the point moot since all the domain controllers in the domain are also all the domain controllers in the forest.  The domain partition is replicated to all domain controllers in the domain.

 

Active Directory – Schema

Microsoft’s documentation on unmodified schema classes and attributes can be found at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/adschema/adschema/active_directory_schema.asp   The modifications Exchange makes to the AD schema can be found at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wss/wss/wss_ldf_AD_Schema_intro.asp

The schema management MMC is not automatically available on a Windows machine.  To enable the snap-in, run regsvr32 c:\winnt\system32\schmmgmt.dll – then “Active Directory Schema” will be an option when adding snap-ins to MMC

Active Directory’s schema is normally in a read-only mode and no user has rights to modify the schema.  Prior to enacting a schema change, then, you must enable schema writes and add your account to the “Schema Admins” group.  To enable schema writes, right click on the “Active Directory Schema” item in the MMC and select “Operation Master”.  Then check the box next to “The Schema may be modified on this domain controller”

When creating new schema classes or attributes, ensure you use the correct OID for our organisation.  Preferably, too, create auxiliary classes and associate the aux class with a structural class.  This prevents any vendor changes to the structural class from impacting your schema attributes.

In AD, schema changes cannot be deleted (well, it can but the process is unsupported).  An attribute can be deactivated, but it remains in the schema definition.

Active Directory – Configuration

The AD Configuration partition holds, as the name implies, configuration for the domain and some services within the domain.

  • Display Specifiers: Under the DisplaySpecifiers CN you will see multiple three digit hex number combinations.  These are codes for different languages – 409 being English.  http://www.microsoft.com/globaldev/reference/win2k/setup/lcid.mspx lists the codes used within the Windows internationalisation features.  Under each regional container you will find the actual display specifier for structural schema objects.  On, for instance, the user-Display object, is defined what appears when you right-click a user object in Active Directory Users and Computers.  Another attribute defines the pages which appear when you create a user and the order in which those pages appear.  The createDialog attribute is of particular interest – we modify this to automatically create the display name as lastname, firstname MI if you manually create a user within AD.  This is done by defining the createDialog value as “%<sn>, %<givenName> %<initials>”
  • Extended Rights:  On the controlRightsAccess object, appliesTo defines structural schema objects to which the controlRightsAccess object applies.  The controlRightsAccess objects themselves have several functions.
    • When validAccesses is set to 8, this is to validate writes – or check the attribute value beyond the schema definition.  Implementation is not widespread.
    • When validAccesses is 256, then the object defines an actual extended right – something not part of the normal ACL’s.  Recieve-as and Send-As, for instance, are a special operations for Exchange which can be found in the ExtendedRights container.
    • Other validAccesses codes define ACL groups which can be assigned through the “Delegate Control” function.  and validAccesses indicates what rights the ACL group permits – 16 for read, 32 for write, and the sum of 48 for read/write access.  The membership object in ExtendedRights, with appliesTo bf967aba-0de6-11d0-a285-00aa003049e2 and validAccess of 48 means this access group allows whomever is granted it to both read and write to user objects (bf967aba-0de6-11d0-a285-00aa003049e2 is the guid of the user schema object).  On the schema object “member”, then, the rightsGUID is entered as the attributeSecurityGUID.

An example of the rights grouping is the “Personal-Information” object, rightsGUID 77B5B886-944A-11d1-AEBD-0000F80367C1.  You will find the corresponding octet string, 0x86 0xb8 0xb5 0x77 0x4a 0x94 0xd1 0x11 0xae 0xbd 0x00 0x00 0xf8 0x03 0x67 0xc1, applied to several schema attributes – telephoneNumber, facsimileTelephoneNumber, streetAddress, telexNumber, and so on:

Thus using the “Delegation Of Control” wizard, it is possible to select “Read and write Personal Information” as a permission set rather than specifying each individual attribute you want editable. Note, too, in the ACL editor the listing of “Personal Information” is retained

ForestUpdates

Under ForestUpdates you will see an “Operations” CN.  Operations holds a listing of updates made to the forest (e.g. exchange /forestprep).  This allows the system to check that the requisite forest updates are in place prior to installation without requiring the changes to be re-run.

LostandFoundConfiguration

This is basically the same thing “LostAndFound” in the domain naming context is, but within the configuration partition.  All things being equal, it should be empty.  Should an object be created within the Configuration partition at the same time it’s parent is deleted, the object is moved to “LostAndFoundConfig” for holding.

Partitions

Contains crossref objects to all partitions within the forest – again not as interesting here as it could be with just one tree and domain.

Physical Locations

This is intended for use with Directory Enabled Networking.  The DEN concept is maintained by DMTF (http://www.dmtf.org/standards/wbem/den/) and is not at present implemented at Windstream

Services

Forest-wide application settings – objects within this container correspond directly to the “Services” listed within the “Active Directory Sites And Services” snapin.  One of the services listed is Microsoft Exchange.  Should a server fail, running setup /disasterrecovery will recover most of the Exchange settings for the server from within this container.

Sites

The “Sites” of “Active Directory Sites and Services”.  IP subnets and their associated sites are defined in this container, as well as the replication partnerships between domain controllers.

WellKnown Security Principals

What I call the “virtual credentials” – system security credentials like Everyone and Self are defined herein.

Active Directory – Domain Data

Objects specific to just one domain within the forest – the obvious users, computers, printers, file shares, groups, and contacts.  Less obvious items too are stored within the domain data.  If Windows DNS zones are configured as “Active Directory Integrated”, the DNS entries will appear under “cn=MicrosoftDNS,cn=System,dc=…”.  File replication service (FRS) shares (including the domain SYSVOL), some information on Group Polices, Oracle database connections … any of the structural schema objects … can also be found within this partition.

An object named Infrastructure is in the root of the domain naming context, this object holds the NTDS settings for the domain infrastructure operations master.

Flexible Single Master Operations (FSMO) Roles

FSMO roles are assigned for functions which cannot practically be performed by any domain controller – functionality which cannot subscribe to the multi-master principal.

There are two forest-wide FSMO roles, the Domain Naming Master and the Schema Master.

  • The Schema Master is the server on which writes can be made to the schema.  All domain controllers will have a read-only copy of the schema, but only the schema master can write changes.
  • The Domain Naming Master is used when a new domain is created within a forest – it verifies the new domain has a unique name.

Three additional FSMO roles exist in each domain within a forest.  The Infrastructure Master, RID Master, and PDC Emulator.

  • The Infrastructure Master, in a multi-domain environment, handles cleanup of phantom objects created as members are added to groups via a trust.  The cleanup process is detailed by Microsoft at http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q248047    As we have a single domain, this is somewhat immaterial.  Should we begin implementing other domains, the Infrastructure Master will need to be moved to a non-global catalogue (GC) server.  The GC functionality precludes the phantom objects from being created (and hence from being purged).
  • The RID master allocates blocks of relative ID’s, RID’s, to the domain controllers within the domain to ensure unique GUID’s.  Should the RID master be offline for a short interval, new objects can still be created until the already-allocated RID block has been exhausted.
  • The PDC emulator is multi-function.  Were the domain to be in mixed-mode and therefore support NT4 BDC’s, the PDC emulator is required by the NT4 domain controllers for backwards compatibility.  Our domain is in native-mode and cannot have NT4 BDC’s.  This does not preclude NT4 member servers, just domain controllers.  The PDC Emulator server is authoritative for the user’s password.  Any failed logons are re-checked against the PDC emulator.  In the NT4 environment this was because a BDC was a read-only directory copy to which password changes could not be made.  If you changed your password and attempted to authenticate prior to the domain replicating the change completely, you could receive an invalid password error using your correct new password.  To prevent this issue, a password failure on the BDC was re-checked with the PDC before the logon attempt was failed.  This is how we can allow CSO password changes into AD without requiring the user to wait for domain synchronisation.  The DirXML AD driver is installed to the PDC emulator server to allow immediate use of the user’s new CSO password.  Group Policy Objects are created and edited on the PDC Emulator’s SYSVOL share.  The PDC emulator is also the time source for the domain.  Our PDC emulator is configured to use time.windstream.com as its time source with a time sync period of eight hours.

Normally you can move the FSMO roles between domain controllers using MMC’s.  For the three per-domain roles the change is made in Active Directory Users and Computers.  The Domain Naming Operation Master is changed from Active Directory Domains and Trusts”; the Schema master is changed within Active Directory Schema Manager.

Within Active Directory Users and Computers, right click the domain and select “Connect To Domain Controller” – select the domain controller which will receive the new role.  Then right click the domain and select “Operation Masters”.  You just click the “Change” button to move the role.

In the event of a catastrophic server failure complete with no system state backups you can forcibly transfer the FSMO roles from a non-operational source.  We have done this once in production, the ICM domain, but there is additional peripheral cleanup required to remove the failed domain controller from operation.  http://support.microsoft.com/kb/255504/ contains instructions for seizing FSMO roles.  Microsoft mostly documents the domain cleanup process at http://support.microsoft.com/kb/216498/   If you want to try it for the experience, build two servers, create a fake domain with the two of them, turn one off and seize all the roles onto the remaining machine.  This is effectively what happened in the ICM domain, they had three domain controllers but the first which held all roles was destroyed.  Be careful in production as the post-seizure cleanup is not fully documented.  DNS entries will still exist in BIND.  It is possible for your domain controller machine password to be out of sync with the domain.  I’m sure there are other situations which could arise as well which we didn’t happen across.

Domain Registration – WINS

The WINS entries of your domain should only be used by ‘legacy’ clients, NT4 workstations and servers.  If you configure your domain controllers TCP/IP properties to use your WINS servers, the registration for the domain will be created automatically.  Alternately you can create an LMHOST file for import into a foreign WINS server.  The only reason we do this is to establish a trust with an NT4 domain.  There are two records needed – and for our domain the text is included here-in.

10.33.8.25       SCARLITNT631         #DOM:ALLTEL          #PRE
10.33.8.25       "ALLTEL         \0x1b"  #PRE

If you are attempting to create the LMHOST file for an alternate domain, you can just change the values except the “ALLTEL         \0x1b” entry.  There is a quotation mark, sixteen characters only (insert holy hand grenade like joke here) followed by the \0x1b then the closing quotation mark.  If your domain name is BOB you cannot replace ALLTEL with BOB, you need to replace it with BOB and three trailing space characters.

Domain Registration – DNS

There are four “underscore zones” – new DNS zones used to store the SRV records relevant to the domain. Active Directory works fine with BIND DNS servers – you need to allow dynamic updates from the domain controller IP addresses. Since I do not allow dynamic updates on the root zone, I manually add the domain controller A records.

  • _sites.domain.tld.   Service records advertise servers providing global catalogue, Kerberos, and LDAP services within each site.  The sites are differentiated within the record name – _service._tcp.SITENAME._sites.domain.tld.    The following lines are the _sites records for the TWNUserAuth site
$ORIGIN _tcp.TWNUserAuth._sites.alltel.com.
_gc                               SRV     0 100   3268    neohtwnnt630.alltel.com.
_kerberos                         SRV     0 100   88      neohtwnnt630.alltel.com.
_ldap                             SRV     0 100   389     neohtwnnt630.alltel.com.
_gc                               SRV     0 100   3268    neohtwnnt631.alltel.com.
_kerberos                         SRV     0 100   88      neohtwnnt631.alltel.com.
_ldap                            SRV      0 100   389     neohtwnnt631.alltel.com.
  • _tcp.domain.tld. Service records advertise all domain controllers within the domain providing global catalogue, Kerberos, LDAP, and kpasswd services.  The following lines are the _tcp records for the NEOHTWNNT630 server
$ORIGIN _tcp.alltel.com.
_gc                           SRV     0 100   3268    neohtwnnt630.alltel.com.
_kerberos                     SRV     0 100   88      neohtwnnt630.alltel.com.
_kpasswd                      SRV     0 100   464     neohtwnnt630.alltel.com.
_ldap                         SRV     0 100   389     neohtwnnt630.alltel.com.
  • _udp.domain.tld. Used for UDP kerberos connections to get tickets and change passwords.  Service records in this zone advertise the UDP Kerberos and kpasswd services for the domain.  The following lines are the _udp records for the NEOHTWNNT630 server
$ORIGIN _udp.alltel.com.
_kerberos                   SRV     0 100   88       neohtwnnt630.alltel.com.
_kpasswd                    SRV     0 100   464      neohtwnnt630.alltel.com.
  • _msdcs.domain.tld.  Kerberos, ldap, and global catalogue records by site and not.  In addition each domain controller’s GUID used for replication is registered here.  Again the example provides the service records for NEOHTWNNT630
$ORIGIN _msdcs.alltel.com.
47c1965e-87e8-4445-8552-fd20892c08c2    CNAME          neohtwnnt630.alltel.com.
_ldap._tcp.e0f0a709-9edf-483b-96e6-55c0dd55c1a6.domains           SRV 0 100 389             neohtwnnt630.alltel.com.
gc                      A       10.10.90.217
$ORIGIN _tcp.dc._msdcs.alltel.com.
_kerberos                                 SRV     0 100   88        neohtwnnt630.alltel.com.
_ldap                                     SRV     0 100   389       neohtwnnt630.alltel.com.
$ORIGIN _tcp.TWNUserAuth._sites.dc._msdcs.alltel.com.
_kerberos                                 SRV     0 100   88        neohtwnnt630.alltel.com.
_ldap                                     SRV     0 100   389       neohtwnnt630.alltel.com.
$ORIGIN gc._msdcs.alltel.com.
_ldap._tcp                                SRV     0 100   3268    neohtwnnt630.alltel.com.
$ORIGIN _sites.gc._msdcs.alltel.com.
_ldap._tcp.TWNUserAuth                    SRV     0 100   3268    neohtwnnt630.alltel.com.

The PDC emulator is also advertised here

$ORIGIN _msdcs.alltel.com.
_ldap._tcp.pdc                         SRV     0 100   389      scarlitnt631.alltel.com.

 

Client Authentication

A client which has already authenticated to the domain will have a registry entry which retains the client’s site.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters]
"DynamicSiteName"="LITUserAuth" 

When a client attempts to authenticate to Active Directory, the service records for the kerberos service are used to determine an appropriate authentication source.  In the case of the PC above, this would be a query for _kerberos._tcp.LITUserAuth._sites.dc._msdcs.alltel.com. service records is made.  An LDAP connection is initiated over udp/389 to every domain controller returned by the DNS query.  Each connection is initiated in 1/10th intervals second.  The receiving servers compare the client’s IP address to the subnet configuration to verify the client is reaching the correct site for it’s current subnet.  The first LDAP response received is then used as the kerberosauthentication server.  If the client’s site is incorrect a referral is returned for the correct site – which then prompt the client to re-query DNS for the correct new site.