Category: Technology

DMARC and DKIM

Microsoft’s latest security newsletter included the fact that more than 90% of Fortune 500 companies have not fully implemented DMARC. Wow — that’s something I do at home! Worse still, the Fortune 500 company for which I work is in that 90% … a fact I hope to rectify this week. SPF is just some DNS entries that indicate the source IPs that are expected to be sending email from your domain. Lots of SPF record generators online.

DKIM is a little more involved, but it’s a lot easier now that packages for DKIM are available on Linux distro repositories. You still *can* build it from source, but it’s easier to install the OpenDKIM package.

Once the package is installed, generate the key(s) to be used with your domain(s).

cd /etc/opendkim/keys/
openssl genrsa -out dkim.private 2048
openssl rsa -in dkim.private -out dkim.public -pubout -outform PEM
# secure private key file
chown opendkim:opendkim dkim.private
chmod go-r dkim.private

Decide on the selector you are using — I use ‘mail’ as my selector. At work, I use ‘2017Q3Key’ — this allows us to change to a new key without in-transit mail being impacted. Old mail was sent with the 2017Q2 selector and *that* public key is in DNS. New mail comes across with 2017Q3 and uses the new DNS record to verify. I do *not* share these keys – anyone else sending mail from our domain needs to generate their own key (or I make one for them), use their own unique selector, and I will create the DNS records for their selector. When marketing engages a third party to send e-mails on our behalf, we have a 2017VendorName selector too.

Edit /etc/opendkim.conf. The socket line is not necessary – I just tend away from default ports as a habit. Since it’s bound to localhost, not such a big deal.

Mode sv
Socket   inet:8895@localhost
Selector mail
KeyFile /etc/opendkim/keys/dkim.private
KeyTable /etc/opendkim/KeyTable
SigningTable refile:/etc/opendkim/SigningTable
InternalHosts refile:/etc/opendkim/TrustedHosts

There’s a config option to “SendReports” — it’s a boolean that indicates if you want your system to send failure reports when the sender indicates they want such reports and provide a reporting address. Especially for testing purposes, I recommend indicating your domain wants reports — it is helpful in case you’ve got something configured not quite right and are failing delivery on some messages. As such, configure my installation to send reports. It’s additional overhead in cases where verification fails; I don’t see all that many failures, and it isn’t a lot of extra load. Since I know my installation will send detailed failure information, I can use my domain when testing new implementations.

Once you have the base configuration set, edit /etc/opendkim/SigningTable and add your domain(s) and the appropriate selector

*@rushworth.us mail._domainkey.rushworth.us
*@lisa.rushworth.us mail._domainkey.lisa.rushworth.us
*@scott.rushworth.us mail._domainkey.scott.rushworth.us
*@anya.rushworth.us mail._domainkey.anya.rushworth.us

Edit /etc/opendkim/KeyTable and map each selector from the SigningTable to a key file

mail._domainkey.rushworth.us rushworth.us:default:/etc/opendkim/keys/dkim.private
mail._domainkey.lisa.rushworth.us lisa.rushworth.us:default:/etc/opendkim/keys/lisa.dkim.private
mail._domainkey.scott.rushworth.us scott.rushworth.us:default:/etc/opendkim/keys/scott.dkim.private
mail._domainkey.anya.rushworth.us anya.rushworth.us:default:/etc/opendkim/keys/anya.dkim.private

Edit /etc/opendkim/TrustedHosts and add the internal IPs that relay your domain’s mail through the server (IP addresses or subnets)

Create DNS TXT records – the part after p= is the content of the public key file for that selector. When you are first setting up DKIM, use t=y (yes, we are just testing this). Once you confirm everything is functional, you can change to y=n (nope, really pay attention to our DKIM signature and policy). The policy is an individual preference. I use ‘all’ (all mail from my domain will be signed) and “o=-” (again all mail from my domain will be signed). You can use “o=~” (some mail from my domain is signed, some isn’t … who knows) and “dkim=unknown” (again, some is signed). You can use “dkim=discardable” (don’t just consider the message as more likely to be spam if it is not signed … you can outright drop the message). As a business, I don’t use this *just in case*. Something crazy happens – the dkim service falls over, your key gets mangled – and receiving parties can start dropping your messages. Using “dkim=all” means they are more apt to quarantine them as spam, but someone can go and get the messages. And hopefully notice something odd is happening.

mail._domainkey.domain.tld  TXT k=rsa;t=y;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzTnpc7tHfyH1zgT3Jx/JHmGSz8WCy1jvzu5QsYvDBmimKEHRY4Kz4mya5bOYsDQuJ/sz+BJo6xDwsUXCuyEkykIlgqP+7E9oK2EcW0dZms87SGmNEnNBN5iTe0pdzk1lXx2js3QdOWswO+cmA9F1Z8OzSR+2u79huugPFBHl79zFvOEHbigrmeHEfo0KHWpeNomf/xKx+wyYr1n3R5gS+28CeC3abSyKgmaYYRLoZsjrCLbEM0m2YPJRKd1ZGOObBMa4PZWj7pT07ISEjoNnXQ27BtcL/QjKKeLkbJ0UGEOSdPEJKuEpAUvYU9lA5hbtzrqiwdlPxWYocDVPrcqAHwIDAQAB
_adsp._domiankey.domain.tld TXT dkim=all
_domainkey.rushworth.us TXT t=y;o=-;r=dkim@lisa.rushworth.us
_ssp._domainkey.rushworth.us TXT t=y;dkim=all

 

Edit /etc/mail/sendmail.mc (using the port defined in /etc/opendkim.conf

INPUT_MAIL_FILTER(`dkim-filter’, `S=inet:8895@localhost’)

Make your sendmail.mc to sendmail.cf and verify that you’ve got the dkim-filter line

Xdkim-filter, S=inet:8895@localhost

Start opendkim, then restart sendmail. Now test it — inbound mail should have *their* DKIM signatures verified, outbound mail should be signed with the appropriate key.

Once you have verified your DKIM is functioning properly — well, first of all you can update your DNS records to remove testing mode. Then create your DMARC record:

_dmarc.rushworth.us     v=DMARC1; p=quarantine; sp=quarantine; rua=mailto:dmarc-rua@lisa.rushworth.us!10m; ruf=mailto:dmarc-ruf@lisa.rushworth.us!10m; rf=afrf; pct=100; ri=172800
 Again, you don’t need to use quarantine — ‘reject’ would recommend mail be dropped or ‘none’ recommends no action (good for testing). The rua (aggregate reporting email address) and ruf (address to recieve failing samples for analysis) should be in your domain.
You could add either/both “adkim=s” or “aspf=s” to indicate your DKIM or SPF adhere to strict standards. I use relaxed (default, do not need to specify it in the TXT record).
If you want the reports delivered to an address outside of your domain, that domain needs to publish a DNS record authorizing receipt of the reports:
     rushworth.us._report._dmarc.lisa.rushworth.us     v=DMARC1

Sendmail VirtUserTable

Some mail systems support sub-addressing (i.e. user+ignoredstring@example.com), but Exchange is not one of them. Even if/when it gets supported, it’s really easy to figure out the real e-mail address in that sub-address. Instead, we use sendmail’s virtusertable to map entire subdomains (i.e. @lisa.example.com) over to our primary e-mail addresses. If an address becomes compromised, we can blacklist the particular something@subdomain.rushworth.us address in the access table).

Virtual Domain Aliases

These aliases allow changes to be made to intended recipient addresses.  There are two files required for an address to be aliased.  An entry for “VIRTUSER_DOMAIN_FILE” will exist in the sendmail.mc specifying the file listing the domains to be included for aliasing.  For us, this is /etc/mail/virtuser-domains.  This is a text file containing the name of each domain to be virtualized for aliasing, one domain per line.  Please note, the domains included herein need only be the recipient domains, not the domains to which aliases are mapped.  E.G. our virtuser-domains file contains just:

example.com

And yet we can alias test.addy@example.com to someotheraddy@example.net … it is only the source address that needs to be defined in virtuser-domains.

Aliases for the virtual domains are contained in /etc/mail/virtusertable.  The left-hand entry is the recipient address and the right-hand entry is what that recipient will be translated to.  Left-hand entries can be an email address (testaddy@example.com) or a domain (@lisa.example.com)

Right-hand entries can be an alternate address.  If the address should remain the same, an exclamation point can be used:

myfakeaddress@example.com        external.email@example.net
myaddress@example.com            !

The right-hand entry can also be an action, like error which will return an error code

compromised.address@lisa.example.com            error:nouser User unknown

 

To commit changes to the virtusertable:

makemap hash /etc/mail/virtusertable.db < /etc/mail/virtusertable

 

Testing Virtual Aliases:

You can test the results of the virtual address space aliasing using sendmail –bt.  From within the new prompt (a greater than sign on a blank line) type3,0 followed by the address you would like to test.  E.G.:

[uid@NEOHTWNLX821 ~]# sendmail -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 3,0 llanders@example.com
canonify           input: llanders @ example . com
Canonify2          input: llanders < @ example . com >
Canonify2        returns: llanders < @ example . com . >
canonify         returns: llanders < @ example . com . >
parse              input: llanders < @ example . com . >
Parse0             input: llanders < @ example . com . >
Parse0           returns: llanders < @ example . com . >
ParseLocal         input: llanders < @ example . com . >
ParseLocal       returns: llanders < @ example . com . >
Parse1             input: llanders < @ example . com . >
Recurse            input: llanders @ example . net
canonify           input: llanders @ example . net
Canonify2          input: llanders < @ example . net >
Canonify2        returns: llanders < @ example . net . >
canonify         returns: llanders < @ example . net . >
parse              input: llanders < @ example . net . >
Parse0             input: llanders < @ example . net . >
Parse0           returns: llanders < @ example . net . >
ParseLocal         input: llanders < @ example . net . >
ParseLocal       returns: llanders < @ example . net . >
Parse1             input: llanders < @ example . net . >
Mailertable        input: < example . net > llanders < @ example . net . >
Mailertable        input: example . < com > llanders < @ example . net . >
Mailertable      returns: llanders < @ example . net . >
Mailertable      returns: llanders < @ example . net . >
MailerToTriple     input: < > llanders < @ example . net . >
MailerToTriple   returns: llanders < @ example . net . >
Parse1           returns: $# esmtp $@ example . net . $: llanders < @ example . net . >
parse            returns: $# esmtp $@ example . net . $: llanders < @ example . net . >
Recurse          returns: $# esmtp $@ example . net . $: llanders < @ example . net . >
Parse1           returns: $# esmtp $@ example . net . $: llanders < @ example . net . >
parse            returns: $# esmtp $@ example . net . $: llanders < @ example . net . >

Use ctrl-d to exit the test.

Sendmail Mailertable

Mailertable (/etc/mail/mailertable)

Routing information for external delivery. Functionally, these are like the SMTP Connectors within Exchange. The mailertable entries can override everything including smarthost definitions. This is required for internal mail routing – our sendmail servers should not transmit email for @windstream.com to the MX records but rather the destination we intend. We also use mailertable entries to force B2B communication over internal secured channels.

If a server is unable to deliver mail to a specific domain (e.g. one of our public IP addresses gets blacklisted), a mailertable entry can be used to direct all mail destined for the domain through one of our servers still able to make delivery.

The file contains two columns, domains and actions. Domains can be ends-with substring matches:
.anythingfromthisdomain.com

Will match @thishost.anythingfromthisdomain.com as well as @thathost.anythingfromthisdomain.com. Domains can also be a full match of the right-hand side of the email address:
justthisemaildomain.com

Which will match @justthisemaildomain.com. The most “accurate” match will win, not just the first match in line. So if your file contains the following:
.mysampledomain.com relay:[10.10.10.10]
thishost.mysampledomain.com relay:[20.20.20.20]

Mail destined for thishost.mysampledomain.com will be sent to 20.20.20.20

Actions contain both a mailer and a host. The mailer can redirect messages to local users:
.egforlocaldelivery.com local:username

Or it can force an error response:
Baddomain.com error:5.3.0:Unknown User

Our use of the mailertable, though, is to redirect mail destined for the domain:
windstream.com relay:[twnexchinbound.windstream.com]
newacquisition.com relay:[theirinternalhost.theirdomain.com]

In these cases, the square brackets around the destination override the MX record. To reroute a domain’s delivery destination, then, it is imperative that the host be enclosed in square brackets.

To commit changes to the file, either use “make” from within /etc/mail to commit all changes or the following command to commit just the changes to mailertable:
makemap hash /etc/mail/mailertable < /etc/mail/mailertable

Sendmail Configuration

Sendmail Configuration Files – sendmail.cf and sendmail.mc

 

Sendmail configuration files are located by default in /etc/mail/.  PureMessage uses /opt/pmx4/sendmail/etc/mail/

 

The main configuration file is sendmail.cf.  This is a rather cryptic file which we will not configure directly.  If you want to know the syntax for sendmail.cf, read the doc at http://www.sendmail.org or get the O’Reily book.  This information is specific to the MC file from which a macro builds the CF file..

 

sendmail.mc contains instructions to allow the M4 macro processor to build sendmail.cf.  Very important, before you can use a macro to create a sendmail.cf file, you need to have the macro installed.  This is the sendmail package sendmail-cf.  To ascertain if the package has been installed on RedHat:

 

[root@LJLLX001 mail]# rpm -qa | grep sendmail

sendmail-8.13.1-2

sendmail-cf-8.13.1-2

 

Both sendmail and sendmail-cf packages should appear in the results.  If you do not have the CF package, install it.

 

The text “dnl” within sendmail.mc denotes a comment – like a tic in VisualBasic or a hash in perl.  Many lines end with dnl, or dnl with some type of commentary.  Lines beginning with dnl are not processed.

 

Common instructions within a sendmail.mc file:

 

include(`/usr/share/sendmail-cf/m4/cf.m4′)dnl

This line refers the m4 utility to the correct “translation” to build the sendmail.cf file.  Important that the line is at the top of the mc file, but nothing to do with sendmail configuration specifically

 

VERSIONID(`setup for Red Hat Linux’)dnl

This line is not required, and we have ‘junk’ in it frequently.  It records the version of sendmail in the cf file for administrative reference.

 

OSTYPE(`linux’)dnl

More instructions for m4, different OS’s have different locations for sendmail files and the OS defined here identifies which parameters to use.  This line again needs to be at the top of the mc file

 

define(`confDEF_USER_ID’,“8:12”)dnl

Defines which user and group sendmail will run as – do NOT pick root here.  User id 8 (mail) and group id 12 (mail) from /etc/passwd and /etc/groups respectively.

 

define(`confTO_CONNECT’, `1m’)dnl

Time limit for SMTP connection timeout, set to one minute normally.  This is how long your server will wait for an initial connect() to complete.

 

define(`confTRY_NULL_MX_LIST’,true)dnl

Email is normally routed by MX records.  This instruction means the ‘domain’ can also be a host name with no MX defined.  E.G.  sending email to @windstream.com will return the MX records, as they exist.  Attempting to email @neohtwnlx810.windstream.com will return no MX records, but LX810 will be contacted directly to attempt delivery.  This is a most useful instruction for return delivery to system mailers.

 

define(`confDONT_PROBE_INTERFACES’,true)dnl

The sendmail class w lists the host and IP addresses for which sendmail accepts and takes local delivery.  This class can be automatically populated, or using this directive not automatically populated.  We configure this information manually in other files.

 

You can use a sendmail command line to determine what is set to various system variables:

[root@LJLLX001 ~]# sendmail -d0.1 -bv

Version 8.13.1

Compiled with: DNSMAP HESIOD HES_GETMAILHOST LDAPMAP LOG MAP_REGEX

MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6

NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS

USERDB USE_LDAP_INIT

============ SYSTEM IDENTITY (after readcf) ============

(short domain name) $w             =      LJLLX001

(canonical domain name) $j        =      LJLLX001.vibiant.dnsalias.com

(subdomain name) $m                 =      vibiant.dnsalias.com

(node name) $k                             =      LJLLX001.vibiant.com

========================================================

 

define(`PROCMAIL_MAILER_PATH’,`/usr/bin/procmail’)dnl

Exactly what it says – the location of procmail

 

define(`ALIAS_FILE’, `/etc/aliases’)dnl

Location of the file for local delivery aliases – not something we use often as there are few local delivery accounts.  In the ISP, this file can be used to give someone additional addresses which deliver to the same mailbox.  This file can also be used to direct delivery of a local account to a program – in PureMessage for example, /opt/pmx4/sendmail/etc/mail/aliases directs the pmx-auto-approve address to the application which releases user messages.

 

define(`confBIND_OPTS’, `WorkAroundBrokenAAAA’)dnl

This is a resolver option, it instructs sendmail to ignore SERVFAIL errors during an IPv6 lookup.  We had a few domains for which we could not deliver mail without this directive.

 

define(`SMART_HOST’, `[192.168.1.53]’)

A smart host can be used instead of direct mail delivery.  For a server which is not meant to deliver mail to the internet (neohtwnlx824 for instance) the smart_host directive sends all mail to the defined destination.  The destination can be a hostname or an IP address.  Note, the mailertable will override the smarthost.

 

define(`STATUS_FILE’, `/var/log/mail/statistics’)dnl

Retains statistical information on server – use the command mailstats to output the statistics, the file created here is not text

 

define(`UUCP_MAILER_MAX’, `2000000′)dnl

Maximum size for messages relayed by UUCP mailers

 

define(`confPRIVACY_FLAGS’, `authwarnings,novrfy,noexpn,restrictqrun’)dnl

Disables unwanted commands – usually for security reasons.  EXPN expands groups into component members, for instance, so NOVRFY is used to disable the command.  Some of these are more important if local delivery is handled by the sendmail server.

 

define(`confAUTH_OPTIONS’, `A’)dnl

What kinds of authentication are supported by the server.  Useful if you are requiring authentication to relay mail, we do not do this.  Some UNIX hosts get confused if AUTH is an option made available, and you need to remark this line out of the mc file.

 

define(`confTO_QUEUEWARN’, `6d’)dnl

If you ever see an email from a destination mail server saying it is still trying to deliver your message and just wanted to let you know – that is what this interval defines.  To truly adhere to RFC specifications, a sendmail server should continue to attempt delivery for at least four to five days.  As a “nice” feature, the server can send periodic notifications to the sender that delivery has been delayed.  This standard comes from a time when circuits were smaller and quite lossy.  It could reasonably take days to establish a connection to the destination and transmit a message.

We are rogue and just return mail as undeliverable after a shorter period.  No reason to notify users, but to ensure that a notification is not sent, we put the warning interval at something higher than the expiration interval.

 

define(`confTO_QUEUERETURN’, `12h’)dnl

Related to the QUEUEWARN interval – this is the period after which the sendmail server considers the message undeliverable and returns it to the sender.  By default, this is five days so we make sure to define something more reasonable.  Otherwise there would be no way to identify “high” mail queue counts for alerting.

 

define(`confQUEUE_LA’, `16′)dnl

Load average at which queue only functionality is engaged

 

define(`confREFUSE_LA’, `48′)dnl

Load average at which SMTP connections are refused

 

define(`confDELAY_LA’, `30′)dnl

Load average at which sendmail will delay one second on SMTP commands

 

define(`confMIN_QUEUE_AGE’, `5m’)dnl

Minimum time a message has to sit in the queue before it is retried

 

define(`confTO_HOSTSTATUS’, `2m’)dnl

If a host has been denoted as unavailable, the status will be cached for this duration.  After the interval expires, connection to the host will be retried

 

define(`confMAX_DAEMON_CHILDREN’, 2000)

Maximum number of children processes permitted.  Sendmail will reject subsequent connections once this number has been reached.  Very important to have something defined on the DMZ servers.  Default is infinite and it is possible for a server to become unresponsive and need to be rebooted with out of memory errors when too many processes are spawned.

 

define(`confTO_IDENT’, `0′)dnl

Timeout for responses to IDENT

 

FEATURE(`no_default_msa’,`dnl’)dnl

The default MSA options are not used, but rather explicitly defined in the DAEMON_OPTIONS directive

 

FEATURE(`smrsh’,`/usr/sbin/smrsh’)dnl

Shell used for command line mailing programs, not really pertinent in our case

 

FEATURE(`mailertable’,`hash -o /etc/mail/mailertable.db’)dnl

This file will be discussed in more detail later, this directive specifies the use of a mailertable and the location of the file.

 

VIRTUSER_DOMAIN_FILE(/etc/mail/virtuser-domains)dnl

This file will be discussed in more detail later, this directive specifies the location of the file containing virtualised domains

 

FEATURE(`virtusertable’,`hash -o /etc/mail/virtusertable.db’)dnl

This file will be discussed in more detail later, this directive specifies the use of virtual user mapping and the location of the file containing said mappings

 

FEATURE(always_add_domain)dnl

Appends the local host domain to even locally delivered mail.

 

FEATURE(use_cw_file)dnl

Alternate host names are in /etc/mail/local-host-names – machine aliases

 

FEATURE(use_ct_file)dnl

Users who can set alternate envelope from addresses without generating a warning message.  File is /etc/mail/trusted-users

 

FEATURE(local_procmail,`’,`procmail -t -Y -a $h -d $u’)dnl

Specifies program to use as the local mailer, and command options

 

FEATURE(`access_db’,`hash -T<TMPF> -o /etc/mail/access.db’)dnl

This file will be discussed in more detail later, this directive specifies the use of an access restriction table and the location of the file.

 

EXPOSED_USER(`root’)dnl

 

 

DAEMON_OPTIONS(`Port=smtp, Name=MTA’)dnl

This is where the settings for the MSA are defined.  Port=smtp uses the default port of 25, or an alternate port can be used.  Addr=# can be included to bind sendmail to a specific address (including 127.0.0.1 for localhost access only).

 

INPUT_MAIL_FILTER(`vamilter’,`S=inet:3333@localhost,F=R,T=S:10m;R:10m;E:10m’)

Defines a “milter” – mail filter.  The port and destination of the milter must be included with S=.  S=inet is a IPv4 socket, S=inet6 is an IPv6 socket, and S=local is a Unix-domain socket (/var/run/)

F= defines an action to take on failure, R (reject), T (tempfail), or if no option is included just pass the message through sendmail and ignore the milter

T= defines timeouts for sendmail’s communication with the milter:

C         Connect timeout

S          Sending timeout (sendmail transmission of data to milter)

R          Reading timeout (for reply from milter)

E          Overall timeout (between sending end of message and final ack)

 

MASQUERADE_AS(`vibiant.dnsaliascom’)dnl

FEATURE(`masquerade_envelope’)dnl

FEATURE(`allmasquerade’)dnl

MASQUERADE_DOMAIN(`arlitljl.com’)dnl

MASQUERADE_DOMAIN(`homedomain.local’)dnl

This group of directives are all interrelated.  Masquerading is basically replacement – MASQUERADE_AS is the domain which will be used in place of the domains identified in MASQUERADE_DOMAIN lines.  In this case, both @arlitljl.com and @homedomain.local will be overwritten with @vibiant.dnsalias.com.  The directive FEATURE(masquerade_entire_domain) could be included to replace any subdomain of the masquerade domains (e.g. @secured.arlitljl.com, @public.arlitljl.com, and @restricted.arlitljl.com in addition to @arlitljl.com)

Masquerade envelope applies the masquerade to the envelope information and allmasquerade applies the masquerade to everything in the envelope, including cc:, from: and to: — this directive is important when we mask an acquired company’s email domain with our own.

 

FEATURE(`accept_unresolvable_domains’)dnl

Allows the use of domains in the MAIL FROM command to be invalid network and sender domains.  Since some people do not manage to configure their mail servers properly, we are less restrictive here to avoid complaints.

 

LOCAL_DOMAIN(`localhost.localdomain’)dnl

Domain(s) for which the server will accept local delivery – since our servers do not really deliver mail the domain should include the localdomain to prevent accidental misdirection of mail

 

MAILER(smtp)dnl

MAILER(procmail)dnl

Defines mailers to be used in addition to local – these should be the last lines of the mc file

 

 

 

When you make changes to the sendmail.mc file, you will need to run the macro processor to update the CF file.  You can see the results by running:

m4 sendmail.mc | less

 

The text which will be used in sendmail.cf will be displayed on the screen.  To actually commit the changes, use:

m4 sendmail.mc > sendmail.cf

or just type

make

 

Make will update all of the files in /etc/mail, so ensure you like all the changes you have made, not just the changes to sendmail.mc

Sendmail

 

Sendmail is an OpenSource SMTP mail transfer agent implemented on many different Unix platforms. The original version of sendmail, written in the early 1980’s, was written by Eric Allman at Berkeley.  The release code base of sendmail is version 8.  The packages and source can be found at http://www.sendmail.org.

 

Sendmail in its current iteration is configured by many individual files.  All of the configuration options available within the product are well documented at http://www.sendmail.org/doc and http://www.sendmail.org/m4/readme.html.

Future Releases:

There is not a code base 9, but rather SendmailX which has now become MeTA1 (http://www.meta1.org/).  MeTA1 does not include a local delivery agent or mail submission program – it is intended as a conduit for email only.  It will use a single configuration file with a radically different syntax.  Currently, the summer of 2007, the code is in a pre-alpha release.  So, it will be a while.

 

Practical Information:

We back the sendmail configuration files up nightly to NEOHTWNLX810 (/home/NDSSupport/Backups/).  You can restore the files from /etc/mail (or /opt/pmx4/sendmail/etc/mail as appropriate) to a rebuilt server and return the server’s complete configuration.

 

Mail Queues:

Sendmail stores unsent relayed messages in /var/spool/mqueue.  Unsent locally submitted messages will first be in /var/spool/clientmqueue.  Within the mqueue folder, each message has two separate files, one for the header information and a second for the message data.  To count the number of messages queued for delivery, then, you need to divide the number of files within /var/spool/mqueue in half:

echo `ls -al /var/spool/mqueue | wc -l` / 2 | bc

 

New Email Domain Configuration:

We have all of the resources required to establish a new email domain.  (Registration may well be required for a new DNS zone).  To establish a new publicly functional email domain from an existing DNS zone:

  • Create MX records within the DNS zone.  The 10 weight record should point to neohtwnlx821.windstream.com. and the 20 weight record should point to neohtwnlx823.windstream.com.  It is important in these MX records to include the period trailing the hostname
  • On NEOHTWNLX821 and NEOHTWNLX823, edit /etc/mail/access to include the new domain with RELAY
  • On NEOHTWNLX821 and NEOHTWNLX823, edit /etc/mail/mailertable to direct mail to the appropriate destination (unix host, Exchange server, etc)
    • The destination must be configured to accept email from LX821/LX823
  • If internal mail routing needs to be established, an SMTP connector needs to be added to the Exchange organization.  Additionally, mailertable entries should be created on at minimum LX825, LX830, LX833, and LX828
  • If mail should be delivered to mailboxes in the Exchange organization, the new domain should be added to the “Additional Mail Domains” recipient policy.  In this case, the SMTP connector would not be created with Exchange.

 

Sendmail Troubleshooting:

To display information about queued messages:

sendmail –bp

Or to obtain analysis of the domains and addresses within the mail queue, use the perl scripts located in /root/bin:

frombydomain.pl                       Ascending count of sender domains

frombyemail.pl                          Ascending count of sender email addresses

tobydomain.pl                           Ascending count of recipient domains

tobyemail.pl                              Ascending count of recipient email addresses

 

To retry the queues messages with output to the terminal:

sendmail –v –q –C/etc/mail/sendmail.cf &

 

To retry a specific recipient domain’s queue:

sendmail –v –qRthedomain.com –C/etc/mail/sendmail.cf &

Or a specific sender domain’s queue:

sendmail –v –qSthedomain.com –C/etc/mail/sendmail.cf &

 

To retry a specific message ID:

sendmail –v –qImsgidgoeshere –C/etc/mail/sendmail.cf &

 

Add “-d8.11” to the queue retry commands to output debug level diagnostic information to the terminal.  E.G.

sendmail –v -qIl6UJtCE3021014 –C/etc/mail/sendmail.cf –d 8.11

 

Linux Authentication Over Key Exchange

On Linux, you can log in without logging in (essential for non-interactive processes that run commands on remote hosts, but also nice accessing hosts when you get paged at 2AM to look into an issue). The first thing you need is a key. You can use the openssh installation on a server to generate the key:

ssh-keygen -t rsa -b 2048

You’ll get an id_rsa and id_rsa.pub. Your private key (keep it somewhere safe) is in id_rsa; your public key is in id_rsa.pub.

Alternately you can run puttygen.exe (www.chiark.greenend.org.uk/~sgtatham/putty/download.html) for a GUI key generator. Click the “Generate” button & then move the mouse around over the blank area of the PuttyGen window – your coordinates are used as random data for the key seed.

Once the key is generated, click “save public key” and store it somewhere safe. Click “save private key” and store it somewhere safe. Copy the public key at the top of the window. You don’t have to – you can drop the newline characters from the saved public key file, but this saves time.

Either way, you’ve got a base 64 encoded public and private key.

** Key recovery isn’t a big deal – you can always generate a new public/private key pair and set it up. Time consuming if your public key is all over the place, but it isn’t a data loss kind if thing.

*** Anyone who gets your private key can log in as you anywhere you set up this key exchange. You can add a passphrase to your key for additional security.

 

Go to whatever box you want to log into using the key exchange. ** I have a key exchange set up from my Windows boxes (laptop, terminal server) to myid@jumphost. I then have a different key used from myid@jumphost to all of our other boxes. This allows me to change my on laptop key (i.e. the one more likely to get lost) out more frequently without having to get a new public key on dozens of hosts.

Once you are on the box you want as the ID you want (you can do a key exchange to any id for which you know the password – so you can log into serviceaccount@hostname or otherserviceaccount@otherhostname and do this, or you can be logged in as yourid@hostname). Run “cd ~/.ssh” – if it says no such file, run “ssh localhost” – it will ask you if you want to store the server public key – say yes, that creates the .ssh folder with proper permissions. Ctrl-c and cd ~/.ssh again. Now determine if there is an authorized_keys, authorized_keys2, or both. Vim the one you find – if there aren’t any, try “vi authorized_keys” first (authorized_keys2 on RedHat/Fedora, long story) – go into edit mode and paste in the public key line we copied earlier. Save the file. If you get an error like “The server refused our key”, you can “mv authorized_keys authorized_keys2” (or “mv authorized_keys2 authorized_keys” if you started with keys2).

In putty, load in your configuration for whatever host we just pasted the public key into. Under Connection -> Data, find the “Auto-login username” section. Put in whatever ID you used when you added the public key (my use case is me e0082643 … but if you were using ldapAdmin@hostname, you would put ldapAdmin in here)

Then under Connection ->SSH->Auth, find the “private key file for authentication” section and put in your private key location. Go back to the Session section and save the configuration changes.

Now connect & you shouldn’t need to supply a password (or you only need to supply your key passphrase).

** OpenSSH automatically uses the id_dsa or id_rsa (private keys) from ~/.ssh/ when you attempt to authenticate to other hosts. If the destination id@host has your public key in its ~/.ssh/authorized_keys (or ~/.ssh/authorized_keys2), then you’ll get magic key based authentication too. Caveat: on the source Linux host, your private key cannot be group or other readable. Run “chmod go-rw ~/.ssh/id_rsa” to ensure it is sufficiently private, otherwise auth will fail due to permissive access.

** Once you have a key exchange in place, it is fairly easy to update your key. Create a new one but do not yet replace your old one. You can make a shell script that updates all remote hosts with your new public key – per host, run:

ssh user@remoteHost “echo \”`cat ~/.ssh/new_id_rsa.pub`\” >> ~/.ssh/authorized_keys”

Once the new public key info has been pushed out, test it using “ssh -i new_id_rsa user@remoteHost” and verify the key authentication works. Once confirmed, rename your old id_rsa and id_rsa.pub files to something else. Then rename your new_id_rsa to id_rsa and new_id_rsa.pub to id_rsa.pub

Role Based Provisioning

I had done a good bit of data mining research to build out an role based provisioning analysis engine. A decade ago, at University. I had a friend in high-school who had just completed her PhD project on using technology to enhance education in K-12 education. She performed paid consulting services to implement a technology approach package in school systems. Except for the one which employed her — even though she offered her package and guidance for free as part of her employment. I remember thinking that seemed a bit insulting. Here all sorts of people are handing over taxpayer/tuition money for your expertise, but the people who employ you won’t even take it for free.

Well, my company was never much for role based provisioning. Even algorithms I’d built as part of my own research projects. Ohhh, they were all for it in theory. Get the data mining in place, figure out what everyone has, build the templates. Now who signs off on all customer service reps getting access to the billing system and the entire corporate finance group getting access to the financial record system? Anyone? Hello?

Because, in the real world, some finance flunky is going to embezzle some money. And some customer service kid is going to credit back his friends accounts. At which point who said so-and-so could access such-and-such. And no one wants their name associated with that decision. Which makes sense — the individual manager hired the person. Trusted the person. Is responsible for ensuring that trust was warranted.

I am proposing a new approach to role based provisioning. We retain the data mining component. We have access templates built on that data. But we use the template to form a provisioning request. On hire or job transfer, the manager receives a notice to go review the access request form. They can add/remove items at will. They can click to compare access with another specific individual on their team. But before any of this access is granted they click the “I say this person can have this access” button. Voila, no single person responsible for all electronic malfeasance within the company.

Women In STEM

Some Google engineer failed to heed the parable of Harvard President Larry Summers – suggest in any way that women and men are different, and there will be an uproar. What’s ironic is that the main jist of the guy’s monologue (available online) is that not discussing differences between men and women because doing so is insensitive yields diversity programs that are ill suited for their goal. And that companies make business decisions on how close to a 50/50 split they want to get. (If having parity in gender representation was the highest priority in hiring decisions, then a company would only interview female candidates until parity was reached.). And the general reaction online has essentially proved the guy’s point. A reasonable argument would have been challenging the research he cited. Doing so is a fairly easy task. Baron-Cohen, for instance, couldn’t even reproduce his own results. In other cases, the Google engineer conflates correlation and causation. Men don’t take paternity leave because of retribution — my husband was terminated after taking this two weeks of vacation after our daughter’s birth. That’s not even asking for paternity leave — that’s attempting to use vacation time as paternity leave. I experienced more stress as a woman entering an IT support department not because I have a female brain but because my capabilities were questioned (you’re going to fix my computer!?) and some coworkers felt entitled to make sexual advances towards me (I doubt any new male employee was asked to provide his measurements and describe his genitalia to provide a picture to accompany his coworker’s pleasuring himself to the individual’s voice on conference calls).

The mistakes people make, both in the case of Summers and this engineer, is mistaking population-wide averages for attributes of an individual and conflating ‘different’ for ‘inferior’. The engineer wasn’t wrong in one way – it is difficult to discuss gender norms and studies. Trying to divorce emotion from discussion of gender-specific behaviours and preferences isn’t a battle worth fighting. There have been too many badly formed studies designed to prove the superiority of some majority group for any new study to be approached seriously. But he could have made the same suggestions without the contentious topic of gender norms and diversity programs.

Gender aside, different people think differently and have different preferences. I don’t believe this is a contentious declaration. I have artistic friends, I have detail oriented friends, I have creative friends who are not artistic. I know people who love cats and people who love jumping out of perfectly functional aircraft. Introverts and extroverts.

Historically, computer software was not used by people. Programmers hired back in the 60’s and 70’s were not brought in as user experience designers. Text interfaces with obscure abbreviations and command line switches were perfectly acceptable code. They progressed in the field, moved up, and then hired more people like themselves. As computers were adopted, both in business and personally, computer software was slow to adopt ‘usability’ as a goal. Consider the old blue screen word processor. When I left University in 1996, I went to a temp agency in the hope of getting a paycheque that week. They had a computer competency test — figured I would ace it, I’d been running student IT support at the Uni for about eighteen months. I installed Windows 95, IRIX, and AIX and was fairly proficient using any of them. I served as a TA for intro to word processing an excel classes – knew Office 95 better than most of the instructors by year end. Then the temp agency sat me down in front of a computer with an ugly blue screen. What the hell?? I later discovered this old word processing package was common throughout businesses (Universities get grants and buy the latest cool ‘stuff’. Businesses reluctantly forked over a couple hundred grand ten years ago and are going to use that stuff until it decomposes into its component molecules.). People start out with a strip of paper over their function keys so they have a clue how to do anything beyond type on the ugly blue screen. Of course the temp agency was looking for competent computer users so didn’t have the quick ref strip. I couldn’t even start the test (open the file whatever.xtn).

Look at sendmail’s cf configuration file, or search for vim quick ref guides. Even git – sure there are GUI integrations, but the base of git is cryptic command line stuff that you commit to memory. This is not software developed by people who are people focused. Initially with the personal computer in the 80’s, usability was not a concern – “computer users” were in some way skilled and learned to work around the software. With public adoption of the Internet in the 90’s, and dramatically accelerating in the 2000’s and 2010’s — people began to use software. In mass. And new users demanded ‘easy’ to use, intuitive software. User experience engineering became a thing. Software was released to ‘regular’ users to obtain usability feedback.

But the developers behind the software are still, predominantly, the same personality types who developed code for ENIAC. This dichotomy creates an opportunity for the company’s recruitment and hiring teams to give our software an edge. As a company writing software that will be used by people, we think developers who lean toward people on the Things — People dimension, or who score as Social or Artistic on Holland’s personality types, etc provide value to the company. Since we have a lot of things / realistic or investigative types here already, we want our recruiting and hiring practices to create a balance with the other personality types. And we should look at ways to change our processes and make engineering work better align with the interests of people who are more people / cooperative and social or artistic.

Even if the argument was considered flawed, I don’t believe it would receive the widespread distribution and uproar the “it’s all about gender” version encountered. Someone could say “we’d rather make our current staff better at UX” or “we don’t think we need to change our practices to appeal to these other personality types”. Whatever. Even if he still offended his coworkers (I can too do artistic stuff!) or still managed to come off as entitled and whiny, I doubt the guy would have been fired.

Visual Studio Code

We found a free, open source code editor from Microsoft called Visual Studio Code — there are downloadable modules that include formatting for a variety of programming languages (c#, cpp, fortran), scripts (perl, php), and other useful formats like MySQL, Apache httpd config files. It also serves as a GUI front end to git. And that is something I’ve been trying to find since I inherited a git server at work — a way for people to avoid having to remember a dozen different git commands.

Business Practices To Avoid

Don’t ignore your customers. Seems obvious, but failing to engage customers undermines large corporations. I worked for one of Novell’s last big customers back in 2000-2010. We had the misfortune of being in the same territory as their biggest customer, FedEx, so got little sales attention. We were having problems managing computers without using the Active Directory domain — the dynamic local user Zen component that hooked the Novell GINA and created/maintained local user accounts had been used before an NT4 domain even existed within the company. In perusing their web site, I identified a product that perfectly met our needs *and* managed mobile devices (which was an up and coming ‘thing’ at the time). Why, I asked the sales guy, would you not pitch this product to us when we tell you about the challenges we are trying to address? No good answer, but it really was a rhetorical question. There wasn’t a downloadable demo available, you had to engage your sales rep to get a working demo copy — I asked for one, and he said he’d get one to me when he got back to his office.

Nothing. Emailed him a week later in case he just forgot. Oh, yeah, I’ll get that right out to you. A few weeks later, emailed him again. A few weeks later — well, let’s be serious here. We started using Exchange in 2000, and had an Active Directory domain licensed for all users anyway. We were willing to consider paying real money for the Novell product because the migration path was easier … but from a software licensing perspective, switching workstation authentication to AD was a 0$ thing. Needed a few new servers to handle authentication traffic – I think I went with five at about three thousand dollars each. Deployment, now that’s a nightmare. I wrote custom code to re-ACL the user profile directory and modify the registry to link the new user.domain SID to the re-ACL’d old profile directory. It got pushed out via automated software deployment and the failures would call in each morning. Even a 1% failure rate when you’re doing 10,000 computers a week is a lot of phone calls and workstation re-images. (At a subsequent employer, we made the same change but placed workstations into the domain as they were re-imaged for other reasons. New computer, you’re in the domain. Big problems with your OS, you’re in the domain. Eventually we had a couple hundred computers not yet in the domain and the individual users were contacted to schedule a reimage. Much cleaner process.)

The company didn’t last much longer — they purchased SuSE not much later. The sales guys came back – we used RHEL but would have happily bundled our Linux purchases into the big million dollar contract. How much are you looking to charge for updates? Dunno. How much is support? Dunno. Do you know anything about the company’s sales plan for SuSE? Not a thing. Well … glad you could stop by? I guess.

As far as software companies go, this is ancient history. But it’s something I think of a lot when dealing with Microsoft these days. There’s a free mechanism that allows you to use your existing Active Directory to store local workstation admin account passwords. Local workstations manage their own passwords — no two passwords are the same; you can read the individual computer’s password out of AD and provide it to the end user. Expire the computer’s local admin password and next time it communicates with the domain, the password will be changed. Never heard of it from the MS sales guy – someone found LAPS through random web searching. Advanced Group Policy Management that provides auditing and versioning for group policies – not something our MS reps mentioned. Visual Studio Code – yet another find based on random web searching. I know it isn’t the sales guy’s job to tell me about every little bit of free add-on code they have created, but isn’t it in their best interest to ensure that the products that we have become an intrinsic part of our business processes? I tell our SharePoint group that all the time — there are a lot of web based content management platforms. If all you use it for is avoiding web coding … well, I’ve got WordPress that does that. Or some Atlassian wiki thing. And some Jive wiki thing. And some Xerox document repository that has web pages. You need to make something unique to your product intrinsically entwined with business oeprations so no one would ever think of replacing your product.