bib – Gilbert Serex

Sorting the Email Sending Nightmare [ESN]

The one thing I have always been struggling with for the past decade: Sending emails from my own servers

This never ending saga is the result of the openness of the email sending protocol; it is based on trust and is transparent, because of this, it has become the number one method to reach anyone, very easily and quickly, as it was intended.

The idea was genius and obvious, so obvious in fact that now, anyone can send anything to everyone, very fast without consequences and at virtually no cost.

If you don’t understand where I am getting with this I shall make it plain and simple to understand: SPAMMERS

This has created a monumental problem, and spammers don’t care as long as they can sell to one in a million, send to 500’000’000 emails, get 500 responses, sell for $100.00, get it?

That is a problem the industry has been facing for the last 30 years and showing no signs of abating, therefore using various tools to identify the originator of an email message is the only way to at least mitigate the great flood of electrons down the great internet, these tools are checking that the:

Sender of the email is who they say they are (SPF)
Server sending the message is belonging to said sender (DKIM)
Message header contains the two things listed above in the correct way combined with a valid DNS on said server and domain (DMARC)
Message is not junk (SPAM or no SPAM?)

As far as I can tell, I think this is how it is supposed to work. But I never had any luck with successfully implementing the whole thing. For example, I have enormous problem trying to send emails to the Microsoft network comprising, outlook.com, live.com and hotmail.com. Yet when I report the issue to Microsoft Ticketing system, I always get a response basically telling me that there’s nothing “off hand” as they call it to prevent my messages getting through (more on this later).

The other problem I also have started to discover is that the rejected emails one is getting from the recipient’s mail server is pretty generic and does not seem to tell much of what exactly is the problem if at all. It’s just “a problem” and one needs to “figure it out”.

The other thorny aspect is IPV6, which is another problem altogether! So I have turned off IPV6 on my servers until I have time and resources to debug this problem also.

Lucky break

Only just recently I stumbled upon a very useful website that allows me to send an email and check what is going on with it and potentially why I struggle to send it to MSDN (the Microsoft network), so much so that, after many, many tweaks, I finally managed to send an email through to MSDN (albeit flagged as spam) nevertheless IT WENT THROUGH, HURRAY!!

Send them an email (free to send 5 messages per day, which was almost sufficient), see what wrong, fix and there you have it, 10/10 message quality!

The steps

Once you know what you need to do to fix your email sending problems, as long as your IP is not Blacklisted, bob’s your uncle.

Below are the required elements to ascertain your email sending cleanliness.

This Howto is for my own system which is made of Linux (Ubuntu) LAMP webservers managed via Webmin/Virtualmin and I outsource all my DNS with my Registrar (Namecheap).

Other setups will invariably be somewhat different, however I am suspecting that, should you be using Plesk or other similar Control panel systems, most of these issues might be already setup by default ( I know by experience at least with Cpanel it was less a problem) especially when you get a complete package from, say a registrar like Namecheap, as you’d expect them to have done it for you.

In my case, I like to go commando, by not relying on others to do the leg work, the onus is then on you to get it working, something that Virutalmin on an un-managed server gives you plenty of reasons to walk miles. But, as I said, once you know what to do it is relatively simple to enable your server to send clean emails to most networks.

Starting situation

This is the test result when setting up a mail box on one of my servers, it gives an idea of the sort of score you can expect from a default config on a Virtualmin Server

In more details in the screenshot below, one can see one of the main issue; DKIM, the other negative scoring is related to the MX record, as I just set it up for this test it might need to be propagated before the error is fixed.

So let’s get DKIM fixed for this domain by loading the website configuration and making the necessary changes to make DKIM work.

In the screenshot below it is already put in place by default for the whole server, we can therefore leave it as is, unless one wants to change the key…

The current default DKIM is all good to be used

So the next step is to edit the domain DNS record appropriately, by going on the registrar’s website of the domain and add the correct entry to the DNS record.

Lastly, it is also necessary to add the SPF and possibly the DMARC record for this domain, the default value is usually okay, depending if DNS is enabled or not, Virtualmin will provide a list of recommended DNS records under the “Server
Configuration -> DNS records” for that domain :

{domain}.	IN	TXT	"v=spf1 a mx a:{domain} ip4:{xxx.xxx.xxx.xxx} ?all"

Where {domain} and {xxx.xxx.xxx.xxx} are the values to edit for the specific domain/IP

Once this is in place, validate the DNS, wait a little and re-test.

I almost got it right, there is just a little issue with the MX record, but I think that It will get resolved pretty quickly, whilst I was at it I also edited the DMARC record:

_dmarc{domain}.	IN	TXT	"v=DMARC1; p=quarantine; pct=100; ruf=mailto:postmaster@{domain}; rua=mailto:postmaster@{domain}"

Last test

Once all these records are correctly set up. it is time to re-run the test, and, lo and behold!

The final hurdle is to test if the emails are making it through the dreaded OUTLOOK.COM address?

Nope :-P, oh well, apparently it is down to my Provider IP range being black listed, they tried to help but it has never worked, ultimately they recommended me to use SMTP2GO, I tried it and it works, but the point remains that I have a problem sending from one of my servers.

Use it if you cannot send emails after all these efforts

Conclusion

To conclude, I have actually learned something useful and am now confident that email deliver-ability issues I was facing without any clue as to why and simply gave up, are now easily fixable! Thank you Mail tester!

For more resources regarding setting up your own Web server using Virtualmin and what you need to check before starting, read my older post here more particularly the “Next Steps” section…

Addendum

I recently received an emaile regarding this post from Janis von Bleichert over at experte.com, they have developed a upgraded version of the mail-tester system, as he put it :

Like the original, it uses well-known spam filters and blacklists to check the spam score of a mail. But in addition, it also checks whether Gmail classifies the email as spam and into which inbox the mail is placed. Since Gmail is the world’s most used email provider, a positive assessment is critical for deliverability. In the last step, the tool also checks the correct configuration of the SPF and DKIM records.

A screenshot of Janis’s system in action

Thanks to Janis to bring this useful tool to my attention,

I have a life… (now?)

Even though I have not just been diagnosed with COVID-19 (some kind of influenza, speak to you in 10 years time), I am pleased to report that now more than 10 years after my initial Posting about this matter, I now do have a life.

And a Nice one for that matter 🙂

It was quite fun to read back what I wrote then, the state of the web, what was going on around it and my state of mind, today though, it is quite different: I don’t get involved much about what is going on on the web, except I still am on Facebook, only on the PC though, I rarely comment but like a lot of stuff… I removed it from my phone, like a lot of people seem to be doing these days:

I have also decided to screenshot articles, as link rot is very prevalent nowadays.

Funnily, I had completely forgotten about PLAXO, which is totally rotten dead.

So is Google Buzz (knew it, didn’t like it), Twitter, on the other hand, seems to do quite well, and there is Tik Tok (tried it, useless c**p if you ask me, but hey, your call). Will it stay? Jury’s out!

Instagram, also done it, no longer, I don’t see the point really… But my wife, uses it, as she’s (still) a marketing guru, but instead of big animals it is rather smaller furry (or shall I say hairy) ones now.

Which is also my life now, in my lovely renovated Perthshire Cottage at the edge of the stunning Scottish Highlands.

My keyboard is trying to tell me that I should also use #hashtags these days. Never saw this coming, but it kind of makes sense, if only some folks could spell, there is one thing I HATE above all, that is news reports (soon to be dead I also hear), littering their “report” with Twitter and other Instagram links.

If I want to read twitter, I blooming well will go on twitter to read them, which I don’t so don’t spam my reading pleasures with these ugly snippets of badly spelled nonsense, please! Another thing I would not miss at all.

The children are now in their late 20’s, one is working, one is on the dole (as a career) and I am desperately trying to boot my last one out of the house, left, came back, left came back and so on for the last 10 years. What is is with this generation? They blame me for all their bad choices, not me sorry, it YOUR life, all I will say is that gaming at home when you’re nearly 30 is not natural… we’ll see in 2035 perhaps it might have changed and I will be near retirement.

Apparently in 2036 there could have been a planetary disaster but it seems that fake newssss was the culprit. #99942Apophis here it comes aargh. but it may get my 40 something out of his room and look up, just maybe.

so enough for the once in a decade rant, see you in ten years I guess.

How to setup same DKIM settings for multiple (virtualmin) servers

You already have one server with a dkim key pair and use virtualmin “DomainKeys Identified Mail”:

Edit the dkim option on the source server and add the new domain (I personally use sub domains for all my servers) so for example if your initial mail server is s1.domain.com and your new server (the one you will copy the dkim key on) is s2.domain.com your “Domains to sign for” section must contain:

s1.domain.com

s1

s2.domain.com

s2

Save this on the source server. The key will be updated….
The private key on my ubuntu server is filed under: /etc/dkim.key
If your target server is the same [OS], first of all proceed to enable DKIM on the virtualmin “DomainKeys Identified Mail”, you may want to use the same details than the source server in the “Domains to sign for” section & “Save”, this will create the key pairs.
I personally then disabled dkim on the TARGET server before doing the next step, but it might not be required(!?).
When complete, edit the /etc/dkim.key on the TARGET server with vi or otherwise replace the private key with the one from the SOURCE server (you should make a backup of the file first, always do a backup!).
Go back to the virtualmin “DomainKeys Identified Mail” in the TARGET server page and enable the dkim outgoing email but with the option “Force generation of new private key?” to “NO” and “Save”

The private key will then be read from the /etc/dkim.key and used to generate the exact same public and DKIM DNS records for domains as the source server together with all the required settings to make it work.

You’re all set and the DNS can be edited if the DKIM DNS records for domains has changed, mine does not seem to have.

Synchronisation between two Virtualmin servers

When, like me, you get paranoid to losing your data or web server functionality…

I have created a little perl script to allow the synchronization of MySQL databases and /home between my Webservers.

The master server (where the script runs from) is the main production server, the salve server is on standby just in case.

I can appreciate that the passwords are clearly inserted in the script and it is a security issue. Nonetheless, I am the only admin and no other users have ftp or other privileges on both servers. I am sure that there is a way to prevent this but I am happy with the current situation and, obviously I am also making a regular backup of the master server on an external backup provider (I use rsync.net).

#!/usr/bin/perl
# Performs a synchronisation of home folder and dumps sql databases 
# from one Virtual server to another using rsync and secure shell 
# 
# Written by G.Serex Sharpnet UK (c) 03.12.2020 

# Var definitions 

############### SQL Config ############# 
# SQL root username 
$username = "root"; 
# Local SQL root password 
$password = "localmysqlpassword"; 
# Remote SQL root password 
$rpassword = "remotemysqlpassword"; 
# The dumped files path . (absolute path + trailing / please) 
$dumped_dbs_path = "/root/mysql/"; 
# The dumped file name 
$dumped_db = "dump.sql"; 
# Name of the database to exclude from the dump (here the mysql and sys are obviously dedicated to each server, so don't dump them!) 
$exclude_database = "mysql,sys,information_schema,performance_schema"; 

################ SSH Config ################# 

# The remote host name 
$remotehost = "ipaddress"; 

#The ssh username 
$sshusername = "root"; 

#The ssh port 
$sshport = "xx"; 

#____ E N D _ V A R _ D E F S. ________________ 

# First check and optimise the lot.

# A little house keeping 
system("/usr/bin/mysqlcheck --optimize --all-databases --auto-repair -u $username -p$password"); 

# Dump the dbs 
system("/usr/bin/mysqlpump -u $username -p$password --exclude-databases=$exclude_database --add-drop-table --result-file=$dumped_dbs_path$dumped_db"); 

# Transfer them abroad 
system("/usr/bin/rsync -avz -e 'ssh -p $sshport' $dumped_dbs_path $sshusername\@$remotehost:$dumped_dbs_path"); 

# Restore the dump abroad 

system("/usr/bin/ssh -p $sshport $sshusername\@$remotehost 'mysql -u root -p$rpassword < $dumped_dbs_path$dumped_db'"); 

# rsync the home directory 

system("/usr/bin/rsync -avz --delete -e 'ssh -p $sshport' /home/ $sshusername\@$remotehost:/home"); 
exit;

CSF/LFD Sasl Auth Failure

My Daily Logwatch reports a large amount of SMTP Authentication errors like the following:

--------------------- sasl auth daemon Begin ------------------------

 **Unmatched Entries** 

:auth failure: [user=smr@clarky.net] [service=smtp] [realm=clarky.net] [mech=pam] [reason=PAM auth error] 
: auth failure: [user=megaplan@clarky.net] [service=smtp] [realm=clarky.net] [mech=pam] [reason=PAM auth error] 
: auth failure: [user=pdf@clarky.net] [service=smtp] [realm=clarky.net] [mech=pam] [reason=PAM auth error] 
: auth failure: [user=development@clarky.net] [service=smtp] [realm=clarky.net] [mech=pam] [reason=PAM auth error]

This can cause a large amount of lines reported on the email report.

To counter this, here are the steps on an Ubuntu system.

1.Edit /etc/csf/csf.conf to find CUSTOM1_LOG and edit the content to:

CUSTOM1_LOG= "/var/log/mail.log"

2. Add the regex to catch the failed attempts against SASL by adding the following regular expression in /usr/local/csf/bin/regex.custom.pm:

 if (($lgfile eq $config{
CUSTOM1_LOG}) and ($line =~ /^\S+\s+\d+\s+\S+ \S+ postfix\/smtpd\[\d+\]: warning:.*\[(\d+\.\d+\.\d+\.\d+)\]: SASL [A-Z]*? authentication failed/)) {
    return ("Failed SASL login from",$1,"mysaslmatch","3","25","3600");
 }

3. restart CSF and LFD:

#csf -r
#service lfd restart
#systemctl restart lfd

After that, be prepared to get a lot of emails about the blocked IPs for the set time (default is 3600 seconds or 1 hour). They eventually taper out with the most offending IP’s getting permanently blocked 🙂

WordPress twenty seventeen different header images in different pages

spam blockers

Came across the challenge of having to customise the WordPress Twentyseventeen theme to cater for header images that match content of specific pages, let’s say, for example that one has a WP site with portraits of different animals in various pages of the site, whilst twenty seventeen theme allows you to upload a serie of header images that can be randomly rotated on the pages as the are loaded by the end user it does not specifically allow to have the same system dedicated to each pages. The following example remedies this.

After digging a bit and following some other posts on the web, here are the steps required to enable this feat.

Note that you must be able to edit the PHP files and thus be proficient in editing PHP and be familiar with WP inner workings…

First edit the wp-content/themes/twentyseventeen/header.php

(make a backup first!):

...

<div id="page" class="site">
 <a class="skip-link screen-reader-text" href="#content"><?php _e( 'Skip to content', 'twentyseventeen' ); ?></a>

<header id="masthead" class="site-header" role="banner">

<?php get_template_part( 'template-parts/header/header', 'image' ); ?>

 <?php if ( has_nav_menu( 'top' ) ) : ?>
 <div class="navigation-top">
 <div class="wrap">
 <?php get_template_part( 'template-parts/navigation/navigation', 'top' ); ?>
 </div><!-- .wrap -->
 </div><!-- .navigation-top -->
 <?php endif; ?>
</header><!-- #masthead -->

...

Remove or comment out the orange line and replace with the following code.

...

<div id="page" class="site">
 <a class="skip-link screen-reader-text" href="#content"><?php _e( 'Skip to content', 'twentyseventeen' ); ?></a>

<header id="masthead" class="site-header" role="banner">

<?php
// start simply copy header-image.php to header-dog.php and mod the file accordingly
if(is_page("dog-paintings-and-dog-portraits")) {

get_template_part( 'template-parts/header/header', 'dog' );

}
else{

get_template_part( 'template-parts/header/header', 'image' );

}
// end mod
 ?>

<?php if ( has_nav_menu( 'top' ) ) : ?>
 <div class="navigation-top">
 <div class="wrap">
 <?php get_template_part( 'template-parts/navigation/navigation', 'top' ); ?>
 </div><!-- .wrap -->
 </div><!-- .navigation-top -->
 <?php endif; ?>

...

in the example above the page is called

dog-paintings-and-dog-portraits

The header file called is called header-dog.php under the wp-content/themes/twentyseventeen/template-parts/header/ folder (which is a copy of the header-image.php and edited to deal with the other images)

if there are more than one page, simply add more conditionals ifs for each individual pages and copy header-image.php to match like above.

More to come as I write the code…

Windows 10 and superslow HDD’s -> image to SSD :-)

Windows 10 definitely sucks big time on a HDD, so I have just bought a Crucial MX500 1TB SSD and after hooking it up to my rig am simply using ddrescue from a Linux box to do the copy, ah Linux, what would I do without it? certainly live a much shorter life. Happy days…

GDPR & Cookie laws

Not going to talk in details about these two pieces of regulation other that it is a waste of my and everyone else time, like Europe, the Euro and all that jazz, good idea on paper, but forget it. is is useless and just a waste of time and money, I have ignored both and one is already dead, looking forward to have the GDPR dead soon too.

Mission Speakers refoaming

Got myself two pairs of Mission 700 from a back garden with the foam that was knackered. two pairs have been refurbished with the Monacor driver, the other is going to be re-foamed. The foam, ordered from an Ebay seller fits perfectly although it is time consuming as one must press the foam at the back of the cone for a good hour before it starts to stick…

Here are some photos of the process:

Author: bib