As I sat down to write another blog post about my life, I was attacked by a large squirrel, and installed DKIM. Look, squirrel!
OK, so the squirrel was the proverbial squirrel, of getting sidetracked by a series of tasks that seem to connect the dots together.
For me, it was the 3rd time in a week that I’d sat down to write a blog post, only to find that my server was unusable. Like dead in the water unusable.
I was finally able to SSH into my server, and found that my server load was in excess of 10… at times as high as 20.
After finally getting top to run, and a tail on the server logs, it was all too obvious what the culprit was. I was being brute force attacked on my WordPress blogs.
But a closer look reveals that the wp-login.php page isn’t cached, and because they are hitting it so regularly, the PHP-FPM processes cause a high load. This, by the way, could become a source of DDOS ( Distributed Denial of Service ) attacks if not properly shielded against
I installed the WordPress plugin Limit Login Attempts and WP Security Audit Log at some point, but it wasn’t really letting me know what’s going on. This could be that I don’t have it installed on all my blogs, but even the ones I have it on don’t seem to be triggering events anymore. I’m pretty sure I’ll be uninstalling these.
So I wrote up a bash script on my server that would identify two important things that would help me reduce these hacking attempts.
The first was list all the IP addresses and the number of times they are connecting. This tells me if there are IP’s that are doing strange things. Some of the high connections are just search engine bots, but there were a few sending strange queries ( bye, bye! )
The second was to tell me the IP address and number of attempts on the wp-login.php file. For this particular application, this is the more important portion of the script. Basically, anyone that has more than two attempts at the wp-login.php file from the same IP address I ban.
But I also look at the information as a whole, because sometimes the IP addresses are only running 2 queries on an IP address, but from block of IP’s, so really the block could be running hundreds of queries, and the entire IP block needs to be dropped at the firewall.
So now that I had the IP addresses, I needed to add them to the firewall to be dropped. I use iptables ( as do almost all Linux installations ), but I hadn’t really configured a firewall by hand in a few years… So off to the books to figure out the proper syntax, and not lock myself out.
My biggest fear actually is locking myself out, so I’m being uber careful. I don’t exactly have someone to call on my Amazon instance. I probably could call someone, or use the console in some way to get in, but it’s not the same as it was when I was on Hostgator and they could get into my firewall directly and fix it.
So I figured out how to add rules and list them, but for the life of me, I couldn’t find out how to restart the service. My personal laptop is Fedora, and has an iptables service. But on Debian, there’s no such thing.
I feel confident that there’s a system in place somewhere to save and reload the currently firewall rules, but I couldn’t find it. What I did find though, was an iptables init.d script for Debian that seems to be fairly complete, and works for me.
So now I can take my suspect IP addresses, and add them to my drop list and save them with my shiny new iptables script.
Problem solved. No more erratic WordPress login attempts.
End of story, right? Of course not, we don’t have DKIM installed yet
So I recognized that a cron job to show me all the suspect IP addresses of the last 24 hours might not be the most efficient, and that an IDS ( Intrusion Detection System ) might be in order.
Snort came up high on the list in terms of popularity and features, so I started the configuration process, but was quickly shutdown. Turns out that Snort has a really large foot print, and it couldn’t allocate enough memory on my puny little E2 Amazon micro instance.
So after some searching, I found PSAD ( Port Scan Analysis and Detection ), which is lighter weight, and feature rich. It didn’t take much to install, though I still need to tweak it because it’s sending me email alerts for every DNS lookup… seems DNS is considered a threat out of the box — LOL.
I shouldn’t laugh though, DNS is used for many attacks and hacks, but it’s clear to me that I’ve got something mis-configured, and almost seems like it’s for every email lookup on my Postfix server. For now I’ll deal with the false positives, but it makes reading the email alerts impossible, so real attempts will get hidden. I’ll need to fix this sooner than later.
EDIT: This was bothering me after I wrote this article. It looks like the common solution is to ignore UDP traffic on port 53… I chose to ignore TCP traffic on port 53 as well, because there are still a lot of servers using it, and larger traffic ( like zone transfers ) will use TCP as well.
So knowing that my server is being targeted, I decided I’d just tweak my Postfix config a bit, and add greylisting, RBL’s and a few other recommended items
Greylisting and RBL’s were the big things I was looking to get installed.
Greylisting is just a way to have emails be rejected temporarily. If they are legitimate emails, they will wait. If they are spam mail, they will either never try again, or try again right away. Either way, they aren’t allowed through. There are a few legitimate services ( Linked in was one we found, surprisingly enough ), that need to be whitelisted, but it should be few and far between.
I also ran into an issue where the Postfix logs say that no socket found. What? It’s there. It exists. And has proper permissions. So what’s that all about? Turns out Postfix was chrooted ( who knew? ) so I moved the socket directory into the chroot directory for Postfix, and all was good.
RBL’s ( Real Time Black List) are 3rd party services that maintain black lists. I have about half a dozen RBL’s that I have, and all of them must confirm that the email is OK. Any one of them that has it on their black list causes the email to be rejected as SPAM.
I also looked at the Postfix guide book, and added some items they recommended. But once I did that, all Hell broke lose, and messages came through from Google that said I had a mis configured mail server that was delivering spam, and they were blocking me.
OH CRAP – Google is now blocking me!
Sure, not the end of the world for some, but for me, and every email address I have, it’s a disaster. Why? Because all my email addresses are virtual email addresses that are forwards to Gmail accounts. This means that I wasn’t getting any mail whatsoever at this point.
So I did what any good admin would do… I ran around the room screaming and panicking in circles. Just kidding. My wife was asleep at the time, and would have used the baseball bat to help me sleep.
So I did some digging, and decided that I needed to adjust my SPF record. SPF ( Sender Policy Framework ) is a DNS mechanism that allows you to show you are the authority on that domain and not a spammer. It’s a validation technique.
I updated it per a Google recommendation I’d found, though it was only a slight variation from what I had already.
The other option was to install… <Drum Roll Please > … DKIM
DKIM ( Domain Key Identified Mail ) works by using a private/public key, the private key is on the server, and the public key gets installed into the TXT record of your domain. It’s similar in function to what SPF does, in that it helps you establish credibility for mail delivered from your domain.
I thought that by installing DKIM, it would help Google decide that I was one of the good guys, and that any misconfiguration I had previously could be forgiven.
I also read that 1024 bit kits was the minimum number of bits that Google will accept because of the security risks associated with anything less. So taking a prudent approach, I decided that a 2048 bit key would suffice.
All was good until I went to insert the key into the TXT field. I use Amazon Route53 for my DNS services, and it says that my text string was too long. 255 characters was the longest it would accept. WHAT? I had about 500 characters… how am I supposed to do this? I’m doomed, I’m never going to get Google to like me.
So I posted a quick blurb on FaceBook that I would have to explore other options besides Amazon Route53 because of this limitation. Fortunately a Facebook friend of my helped my out be saying I could chunk the data out into smaller pieces, and it would take it.
Sounds promising. So a few Google searches later, and I found a few articles that described this in intimate detail, the best one was the one on the Amazon forums that had an Amazon rep give the full solution to.
So with the TXT record for my DKIM key installed, I was able to finish the rest of the DKIM installation on the server side.
Mission accomplished. DKIM Installed.
The end effect? None really.
By the time DKIM was installed, Google had already recognized that my efforts were of good intention, and that the temporary bad config was just that – temporary.
And for me, DKIM is only read by Google, so I don’t even see the headers to know 100% for certain that it’s working ( but by all accounts it is )
My next step is to install physical users and accounts, so that I can install SASL, and move some email addresses into IMAP folders..
But for now, I have DKIM installed… all because I was attacked by a squirrel.