Alan Doherty's Advice on how to setup/run a Mailserver well

Things you need do

These Requirements are not a minimum for operating a functional mailserver/Mail Transfer Agent {hereafter reffered to as MTA}, but are rather the minimum required for operating a trustable traceable authentic-looking and unlikely to be refused service MTA

One that will comply with the strictest requirements from those receiving your e-mail {and trying to verify wheither or not your mail comes from an MTA or a bot-infested PC}

Obviously if everyone followed ALL these steps telling mail from an MTA from mail from a bot would be simpler, unfortunatly not enough people do yet, and thus a lot of bot-spam gets delivered and a lot of badly configured 'legitimate' MTA mail gets refused, through the guesswork having to be done by recieving systems

Aside: you can avoid the lower requirements by using your connection-provider { hereafter ISPs} provided relay {for outgoing email} by either directing your clients at it or preferably configuring your MTA to send via them. But this means you depend on their reputation, which may be less than perfect occasionally, as it can be tainted by you or any of their other customers sending spam or them having mal-configured machines or DNS, but does offload the mail-queueing re-trying and storage costsbut as I assume you want to run an MTA well we will move on.

scope here is the MTA only not the content traveling via the MTA that is logically a seperate realm entirely, also remember not all MTAs receive mail and not all MTAs send mail, although admittedly most do some of both

main bullet points will be catagorised by MUST and SHOULD;
MUST: if not done /done incorrectly will count against the legitimacy of your MTA
SHOULD: will not normally detract from percieved legitimacy if not done, but if done correctly it will add to your percieved ligitimacy and positivly effect deliveries. may even mitigate against some errors in the MUST portion, such as generic looking Fully Qualified Reverse DNS Entry

items listed under SHOULD sub-catogorised as major or minor;
major: little maintenance overhead, major return, many people will notice/check
minor: little maintenance overhead, minor return, few people will notice/check

[must] Establish an Internet Organisational Identity

This means a domain name of your own, or use of a subdomain of another

If this part isn't obvious you might not be in a position to continue reading

[must] Establish a Consistent Traceable Reputable Address

This means Static IP{s}

first and foremost you need a static ip for both incomming and outgoing smtp, for incomming the requirement is obvious, for outgoing it will become so

[must] Connect the Address with the Organisational Identity, and make it verifiably non-forged

A Non-generic looking Forward Confirmed Reverse DNS Entry for your Static IP{s}

This involves first selecting a name NB:Do NOT use _ character!!
Non-generic looking means not looking like an automatically assigned name like ip.dsl.isp-name.tld {as some others block on generic looking PTR name
if you have IP{s} dedicated to mail-only use then say name it smtp, mail, mx, mta or something-indicating-its-role .your-domain.tld
if its not dedicated to email like say a single ip used by your entire office then go with something more generic like head-office.your-domain.tld, clients.your-domain.tld, dublin.your-domain.tld etc.
next get your DNS administrator {it may be you it may be your web-host it may be your ISPs to setup this name>IP "A" record and set the IP to be your IP
then getting your ISP to setup an IP>name record for your IP {this is known as a PTR record} with the name being the same
repeat this for any/all IPs used by your MTAs
if you have multiple IPs on one MTA it is best to point each PTR at the one name, then point the name at ALL the IPs
{though some badly run DNS hosting providers do not allow this standard option, in this case i would suggest moving DNS hosting provider {as its not a sign of good competence with a service so central to buisness availability} or avoiding using multiple IPs on the MTA as it will cause more issues with HELO/EHLO authenticity later
You can test your ip herefor many receivers this is a MUST-HAVE AOL,

[must] Continue to use your same Organisational Identity in a verifiable way in e-mail exchanges

A helo/ehlo identity that also is resolvable via DNS to your MTA's IP{s}

Select and use an e-mail client specific role name for this identity {it can be the same as above but it is better if it is a different name within the same domain*} and set up the new name>IP "A" record for ALL the IP{s} of this MTA as above.

Most MTA software allows this identity to be directly configured, but will default to the hostname of the machine if unset, for MTA specific instructions seek out info on google ;)

[should] Ensure others know that you explicitly intend this Identity to be verifiable, So they can identify and reject forgeries from others

This means protecting your helo/ehlo Identity by any other method possible

[major] SPF record for your chosen helo/ehlo identity

this means setting up an accurate SPF record for your helo/ehlo identity name, if you have the name>IP "A" record setup as indicated above then the spf for this name simply needs to be "v=spf1 A -all", if for some reason you have persisted and setup a helo/ehlo that dosen't point at all the IPs it may use {because of bad DNS hosting mentioned above} you can at least increase the IPs allowed via SPF record in this form "v=spf1 A ip4:###.###.###.### -all" where ###.###.###.### == an additional IP

[minor] CSA record for your chosen helo/ehlo identity

this means setting up an explicit CSA record authorising this name's use in helo/ehlo and verifying the IPs it is allowed from, CSA was a brilliant and apparently under implemented and abandoned/hardly verified protocol fo avoiding identity forgery, but will likely not actually effect your deliverability, but as it costs nothing other than effort its worth adding, for a full guide see here but simply put add a srv record like so _client._smtp.helo/ehlo-name.your-domain.tld. SRV 1 2 1 helo/ehlo-name.example.com. for each mta

[minor] RP record for your chosen helo/ehlo identity

this means setting up an accurate RP record for your chosen helo/ehlo identity name, in this day and age the e-mail portion can be '.' {.==null} the second part being a pointer to {name of} a txt record offering name phone and/or physical address and/or URL of same.

[should] {Indirectly related} explicity deny the use of other identities you control to be forged in e-mail if not intended to be used in e-mail

This means protecting your non SMTP Identities by any other method possible

[major] Null MX Entries

for any hosts receiving an "v=spf1 -all" entry you can achive similar effect {regarding forgeries using this name as right-rand side of @ in e-mail} for older servers that do-not read SPF records by adding an MX 0 . entry to them, as according to RFC's if a host has an MX it is to be used {in this case no mail} as if no mx is present they say delivery should be attempted to the IP thus the reciever has to assume the address MAY exist without this null MX record telling them directly it does not

[major] SPF record for your other DNS records

this means setting up an accurate SPF record for your other identity names for this name simply needs to be "v=spf1 -all", ie do not accept as a helo/ehlo or right-rand side of @ in e-mail

[minor] CSA record for the rest of your domain

once all your mta's have been authorised add a _client._smtp.your-domain.tld. SRV 1 1 1 your-domain.tld. entry {which basically says allow no helo/ehlo from any non-explicitly authorised hosts in my domain.

[minor] RP record for the rest of your domain

[must] be seen to value and protect your MTA's reputation

This means denying unauthorised use, and reacting/responding to abuse/complaints

this is technically unrelated to being verifiable-traceable, but what point is there being verifiable if your reputation has a negative effect on trust

Unauthorised use is a wide category and will be expanded on in another document but some things to consider doing, not all of these possible with all MTA software

Do not abuse others as part of filtering or any other system, I refer to badl setup autoresponders, accept-then-bounce-spam-filters, Challenge-Response-spam-filtering, sender-callout-verification and any other backscatter causes, I do not state that these are not possible to run in a limited way, but the abuse potential must be weighed and offset as much as humanly possible, tactics like only verify/challenge mail recieved via other verifiable reputable sources, let the untrusted source mail sit in quarentine for manual whitelisting
control who can relay mail {incomming port 25} via you, ie mail accepted by you for onward delivery should only be accepted from external-untrusted IP space if going to a user you are an incoming MX for, and even then you should consider rejecting attempts from other MTAs with a known bad reputation {discovered via DNSBL or other information sources}
control who can relay mail {ouitgoing port 25: discouraged} via youir server, remember machines on your own IP space that may need to be able to send out mail via your MTA are not necessarily clean and may be under the control of a malicious entity {1/3 of all PCs are part of a bot-net} thus if feasable only allow unauthenticated relay from selected IPs within your own IP space if no other method supported, eg certain IPs allowed to relay because the sending software dosn't support smtp auth or submission {port 587} {like certain black-box alarm systems and automated programs{additionally try and filter these to only certain envelope-senders/recipients}
control who can submit mail {outgoing port 587+auth}, the point about bots still applies to all other authenticated outgoing relay, whether limited to your own IP space or from those possesing a valid username and password located anywhere {depending on your policy}, ie. if the machine is compromised the ID/password is too.
Thus try limit its effectiveness where feasible by limiting each autenticated ID to only logon from a limited set of networks {where possible to pre-determine}
More importantly to only sending mail with an associated list of envelope-senders
consider the attempted use of non-associated envelope-senders, or the attempted login with the correct ID/Password from a network not on that ID's list, as an instant sign of a compromised ID/password and notify the user, often by diasabling smtp-auth while still allowing pop/imap and mailing the user to ask them to contact support to get their password changed
Additionally it ensures that when compromised users ID used with the associated envelope-sender {as otherwise the submission is rejected} when used {successfully} to send spam, will result in the user being alerted to the compromise when the bounces/complaints start happening.
use ratelimiting to automatically slow outgoing suspicious traffic
use content filters +quarantining on outgoing e-mail from your users {as most clients cannot handle rejection}, so quarantined mail can be released after manual inspection
monitor rejections/bounces for signs of a user's identity being used for spamming
react quickly to disconnect/disable or otherwise stop both unauthorised use that damages reputation, and authorised users that wish to abuse your MTA in ways also damaging, remind marketing that using that stolen list of address for cold-call e-mails is illegal/abuse
ensure someone accepts mail addressed for @your-helo/ehlo-identity, either your MTA if it has just the A record, or another by adding MX records directing the mail to another MTA
ensure you receive and monitor abuse@your-helo/ehlo-identity and postmaster@your-helo/ehlo-identity even if all envelope-senders using your MTA may be unrelated to your organisation or monitored at some other office, some will complain to you directly
ensure you monitor abuse@any-domain-you-are-primary-mx-for and postmaster@any-domain-you-are-primary-mx-for even if this is delivered to your support dept or some other distribution list ENSURE you are on that list if you accept and terminate mail for that domain
ensure these practices are upheld at any organisation your MTA may be providing service for ie. if you run backupmx.company.com, try and ensure the admin of primarymx.company.com is not ruining your MTA's reputation by operating badly. The same applies for downstream customers of your MTA where you provide outgoing-relay service for smaller MTAs
Finally if you are confident your house is run cleanly apply to organisations such as DNSWL.ORG for whitelisting

the FcRDNS PTR identity != the Verifiable HELO identity AND both within the same domain, arguments for adoption of this newer convention

* The reaason for using a different name in the HELO identity, to that found in the PTR record is non-obvious.
It is because, the reciever CAN already easily automatically verify that the IP's PTR identity is within the same organisation {domain} as the HELO used by the sending client.
{thus is just as veifiably trustable as using the identical name from a traceability point of view, as the "Forward Confirmed" PTR has tied the ip to the HELO domain, thus a HELO within the same domain is obviously non-forged}.

BUT unlike when they are identical, a malicious {think infected pc} client cannot easily appear to be a valid MTA by just looking up its own PTR and forging this as its HELO name. {as this is the default action of most bots}

Obviously this will only stop widespread forgeries from IP addresses that DO NOT Legitimatly HELO with their PTR identities, and have their PTR identities protected via SPF and CSA records instructing others that they should never be accepted in HELO identity ever from any IP

Less obviously As these 'newer standard' practices become the norm for legitimate senders it is easier for users to filter traffic from clients with suspicious markers such as no FcRDNS 'old standard' or when these exists, from clients sending from an identical {and thus easily found/forged} HELO name 'newer standard'

Last updated Jan. 2009 Alan Doherty