If you are like me and you run a public-facing SSH listener you will undoubtedly start to see a ton of failed SSH login attempts in /var/log/auth.log. If you are a paranoid cyber security professional like myself, then this really irks you. If it doesn’t irk you, then it needs to irk you. While your passwords are longer and more complex than 99% of those used, you still know that there is a tiny, slim-to-none chance that one of those login attempts might actually work.
Most of these failed SSH logins come from botnets that are looking for low-hanging fruit in the form of poorly secured Linux servers. When these botnets find one, they’ll log in, attempt to gain a foothold through any type of vulnerability and spread their botnet putridness. Luckily, security wary sysadmins have a lot of wonderful tools available to them. The one that I love to go to is Fail2Ban.
Fail2Ban enters the arena
Fail2Ban is a free utility that scans all sorts of logs looking for indicators of various types of malicious activity – multiple failed logins, attempts to leverage exploits, etc. When it detects a malicious attempt it temporarily adds the offender to the local system’s firewall for a set period of time.
Just by installing it ( sudo apt-get update && sudo apt-get install fail2ban ), the utility will immediately begin defending against the exact type of attempt to brute force a passworded SSH listener that I write about tonight.
It works great for most situations, however a lot of bot herders are well aware of Fail2Ban and have configured their bots to avoid detection by it by throttling back their attempts. Most bots don’t care and will continue to attempt brute forcing even with a drop rule on the target server’s firewall. But there are smart ones that realize that the default configuration of fail2ban puts a 10 minute ban on any IP that makes 5 failed login attempts over a 10 minute period. You’ll see some of the offenders make 4 attempts and wait 10 minutes or space out their attempts so that in any given 10 minute period, there are only 4 failed attempts. Combine this with 20 or more different IP addresses all doing the same thing and the number of password guesses against your SSH listener adds up pretty quickly. Pretty sneaky.
I don’t know about you, but I wear a tinfoil hat on my head when it comes to security of my systems. Luckily Fail2Ban has a juicy configuration file to dive into. Here’s what I did to drastically destroy chances of my servers being successfully brute forced:
Using my favorite lazy editor, I edited /etc/fail2ban/jail.conf. The parameters I changed and their values are below:
- bantime = 31536000
- findtime = 84600
Those values are in seconds. Yes, offenders are banned for an entire year. Yes, fail2ban now looks for 5 login failures in a 24 hour period. As today’s youth might say: Savage.
You can also whitelist IP addresses by adding whitelisted addresses, networks, or CIDR notations to the following line:
ignoreip = 127.0.0.1/8 xxx.xxx.xxx.xxx
If you are following along, don’t forget to restart Fail2Ban:
sudo service fail2ban restart
The theory behind this is two-fold:
- I don’t use a password to log into my SSH listeners so any attempts to authenticate with a password is not a legitimate login attempt. I use key-based authentication and you should too. DigitalOcean has a great guide on configuring key-based authentication for SSH.. If you don’t want to use key-based authentication, another option is two-factor authentication. DigitalOcean also has an outstanding and easy to follow guide on how to set it up using Google’s two-factor authentication platform.
- My hopes are that within a year the system owner will realize that their system is participating in a botnet and take action to remedy the situation. Additionally, a year of dropped packets should be enough to make that particular bot report to higher that my IP address is no longer responding.
The Aftermath
When making changes like this, Fail2Ban actually retroactively reviews the existing /var/log/auth.log file to look for offenders. If you don’t use two-factor authentication yet and you aren’t 100% certain that you haven’t violated the new, stricter rule, you might want to go start fresh and delete that file. If there is a chance that you failed to authenticate 5 times in a single day, then you may be facing a 1 year drop rule from your server, which could be problematic if SSH is your only way of accessing a remote server.
I noticed an immediate drop in failed SSH login attempts. Kibana in my ELK Stack gave me a nice visualization of the drop off. Before the change to Fail2Ban’s configuration I was seeing anywhere from 8 to 20 failed SSH login attempts every 10 minutes. Afterwards I am seeing 0 to 4. The number of IP addresses that were retroactively banned is huge and nearly doubled the size of my sudo iptables -L output.