The one thing I have always been struggling with for the past decade: Sending emails from my own servers
This never ending saga is the result of the openness of the email sending protocol; it is based on trust and is transparent, because of this, it has become the number one method to reach anyone, very easily and quickly, as it was intended.
The idea was genius and obvious, so obvious in fact that now, anyone can send anything to everyone, very fast without consequences and at virtually no cost.
If you don’t understand where I am getting with this I shall make it plain and simple to understand: SPAMMERS
This has created a monumental problem, and spammers don’t care as long as they can sell to one in a million, send to 500’000’000 emails, get 500 responses, sell for $100.00, get it?
That is a problem the industry has been facing for the last 30 years and showing no signs of abating, therefore using various tools to identify the originator of an email message is the only way to at least mitigate the great flood of electrons down the great internet, these tools are checking that the:
- Sender of the email is who they say they are (SPF)
- Server sending the message is belonging to said sender (DKIM)
- Message header contains the two things listed above in the correct way combined with a valid DNS on said server and domain (DMARC)
- Message is not junk (SPAM or no SPAM?)
As far as I can tell, I think this is how it is supposed to work. But I never had any luck with successfully implementing the whole thing. For example, I have enormous problem trying to send emails to the Microsoft network comprising, outlook.com, live.com and hotmail.com. Yet when I report the issue to Microsoft Ticketing system, I always get a response basically telling me that there’s nothing “off hand” as they call it to prevent my messages getting through (more on this later).
The other problem I also have started to discover is that the rejected emails one is getting from the recipient’s mail server is pretty generic and does not seem to tell much of what exactly is the problem if at all. It’s just “a problem” and one needs to “figure it out”.
The other thorny aspect is IPV6, which is another problem altogether! So I have turned off IPV6 on my servers until I have time and resources to debug this problem also.
Lucky break
Only just recently I stumbled upon a very useful website that allows me to send an email and check what is going on with it and potentially why I struggle to send it to MSDN (the Microsoft network), so much so that, after many, many tweaks, I finally managed to send an email through to MSDN (albeit flagged as spam) nevertheless IT WENT THROUGH, HURRAY!!
Send them an email (free to send 5 messages per day, which was almost sufficient), see what wrong, fix and there you have it, 10/10 message quality!
The steps
Once you know what you need to do to fix your email sending problems, as long as your IP is not Blacklisted, bob’s your uncle.
Below are the required elements to ascertain your email sending cleanliness.
This Howto is for my own system which is made of Linux (Ubuntu) LAMP webservers managed via Webmin/Virtualmin and I outsource all my DNS with my Registrar (Namecheap).
Other setups will invariably be somewhat different, however I am suspecting that, should you be using Plesk or other similar Control panel systems, most of these issues might be already setup by default ( I know by experience at least with Cpanel it was less a problem) especially when you get a complete package from, say a registrar like Namecheap, as you’d expect them to have done it for you.
In my case, I like to go commando, by not relying on others to do the leg work, the onus is then on you to get it working, something that Virutalmin on an un-managed server gives you plenty of reasons to walk miles. But, as I said, once you know what to do it is relatively simple to enable your server to send clean emails to most networks.
Starting situation
This is the test result when setting up a mail box on one of my servers, it gives an idea of the sort of score you can expect from a default config on a Virtualmin Server
In more details in the screenshot below, one can see one of the main issue; DKIM, the other negative scoring is related to the MX record, as I just set it up for this test it might need to be propagated before the error is fixed.
So let’s get DKIM fixed for this domain by loading the website configuration and making the necessary changes to make DKIM work.
In the screenshot below it is already put in place by default for the whole server, we can therefore leave it as is, unless one wants to change the key…
So the next step is to edit the domain DNS record appropriately, by going on the registrar’s website of the domain and add the correct entry to the DNS record.
Lastly, it is also necessary to add the SPF and possibly the DMARC record for this domain, the default value is usually okay, depending if DNS is enabled or not, Virtualmin will provide a list of recommended DNS records under the “Server
Configuration -> DNS records” for that domain :
{domain}. IN TXT "v=spf1 a mx a:{domain} ip4:{xxx.xxx.xxx.xxx} ?all"
Where {domain} and {xxx.xxx.xxx.xxx} are the values to edit for the specific domain/IP
Once this is in place, validate the DNS, wait a little and re-test.
I almost got it right, there is just a little issue with the MX record, but I think that It will get resolved pretty quickly, whilst I was at it I also edited the DMARC record:
_dmarc{domain}. IN TXT "v=DMARC1; p=quarantine; pct=100; ruf=mailto:postmaster@{domain}; rua=mailto:postmaster@{domain}"
Last test
Once all these records are correctly set up. it is time to re-run the test, and, lo and behold!
The final hurdle is to test if the emails are making it through the dreaded OUTLOOK.COM address?
Nope :-P, oh well, apparently it is down to my Provider IP range being black listed, they tried to help but it has never worked, ultimately they recommended me to use SMTP2GO, I tried it and it works, but the point remains that I have a problem sending from one of my servers.
Conclusion
To conclude, I have actually learned something useful and am now confident that email deliver-ability issues I was facing without any clue as to why and simply gave up, are now easily fixable! Thank you Mail tester!
For more resources regarding setting up your own Web server using Virtualmin and what you need to check before starting, read my older post here more particularly the “Next Steps” section…
Addendum
I recently received an emaile regarding this post from Janis von Bleichert over at experte.com, they have developed a upgraded version of the mail-tester system, as he put it :
Like the original, it uses well-known spam filters and blacklists to check the spam score of a mail. But in addition, it also checks whether Gmail classifies the email as spam and into which inbox the mail is placed. Since Gmail is the world’s most used email provider, a positive assessment is critical for deliverability. In the last step, the tool also checks the correct configuration of the SPF and DKIM records.
Thanks to Janis to bring this useful tool to my attention,