Defend against fake google bots

I can think of some reasons why folks might use the Googlebot user agent on their non-Google bots, but I can’t think of any good, upstanding reasons to do it.

Here’s how one might find some fine folks who would do such a thing.

As of right now (May 2018), all valid Google Bot source IPs start with the same prefix, 66.249. This may change in the future, so if you’re having problems being crawled by Google, check to make sure you’re not blocking a new range they may have started using. OK, here’s the nitty-gritty.

Interesting that a few of the IPs from my logs indicated that Facebook in Ireland are using the Google user agent. Naughty! Anyway, if you want to test that you’re not blocking a valid Google address, then you need to do an IP lookup on some of these groups of addresses. And of course you can modify the above to scan the current log files instead of the archived gzipped files.

Here’s how I’m blocking the baddies (this isn’t original, I searched and found a version of this). This goes in your Apache config or .htaccess file:

Finding the most persistent, pernicious baddies by processing log files

Logwatch is a great utility for emailing me a summary of system logs over the last 24 hours. One of the things it shows are unsuccessful login attempts and their source IP addresses. But the default unsorted output is hard to analyze and take action on, since a single IP may appear many times in the output but at random locations.

It looks kind of like this (I’ve obscured the full IP to protect the guilty).

So, here we go. Create a shell script or alias with the following:
pbpaste | ggrep -Po '\b((?:\d{1,3}\.){3}\d{1,3})\s' | distribution

Once you’ve got the sections from the logwatch email copied to the clipboard, run this to see which source IPs are the top offenders. Since I’m using pbpaste and ggrep, it should be clear I’m on a Mac. This works on Linux using xsel --clipboard --output and grep, respectively.

And if you haven’t checked out distribution, you should. Super useful.

Allowing bookmarklets to work while NoScript is enabled

The NoScript extension is fantastic at enhancing one’s security while browsing. Sure, it’s a bit of pain to get used to needing to allow scripts for new websites visited (temporarily or permanently). But I wanted to use bookmarklets to post selected stuff in my Dokuwiki with the dokubookmark plugin.

The problem was, every time I hit the bookmarklet, The window popped up but instead of content, I got an error from NoScript saying that ABE had prevented the request. I wish I had looked at the message a little closer to begin with, because it was telling me the problem from the start. Specifically, what I’d missed at first was that it actually reports the rule that caused the failure. filtered by ABE: <LOCAL> Deny I had already tried adding rules to allow GET to my Dokuwiki instance, but I missed that ABE failed on the same rule (LOCAL) each time.

Finally, I smacked my forehead and put my rule above the LOCAL entry in the NoScript config, and all was well in the world again. Here is my rule:

You’ll most likely need to add an exclusion in XSS config as well.

Allow webapps to make outgoing requests

I was experiencing a pretty bad slowdown while trying to use the admin pages of a WordPress site recently. The load on the machine was quite low, so I began to suspect that it was trying to call out to external services (facebook, pinterest, etc) that might have been blocked by CSF (configserver firewall).

I started playing around with tcpdump and friends and then realized that the information I was looking for (blocked outgoing requests) was already being logged in /var/log/kern.log on our Ubuntu system (same on Debian). Continue reading “Allow webapps to make outgoing requests”

Fix for LFD error in syslog

I noticed that I was getting emails from LFD (part of the ConfigServer Firewall package) about failing to find some added check line it was sending to syslog.

The syslog message looks like this:
lfd[%d]: *SYSLOG CHECK* Failed to detect check line [%s] sent to SYSLOG

Of course I’ve replaced the pid with %d and the check string that it’s looking for with %s, since that will vary.

The fix is simple. Just like how you may need to adjust the path in /etc/csf/csf.conf to the real location of the ipset binary, you also may need to set where your SYSLOG messages are going. On an Ubuntu system, that means /var/log/syslog rather than /var/log/messages. Then just run csf -r to restart LFD with the new settings.

/var/log/messages appears in more than just csf.conf. Since /var/log/messages doesn’t exist on my system, I’m just going to symlink it to syslog and see what happens.

OK, I thought better of it and just modified csf.syslogs and csf.logfiles. I deleted that messages symlink in /var/log next. LFD was still being a little bitch after I restarted using csf -r, so I ran service lfd stop and then started it again.

Switching from APF to CSF

CSF logoI was enjoying trying out APF on my Raspberry Pi, but I noticed that it wasn’t blocking repeat attackers the way I wanted it to. fail2ban was working the way it was supposed to work, but it only blocks temporarily, and I never figured out why the gamin back-end to continuously monitor log files didn’t work reliably. I tried to work around that with some extra iptables rules, but was still getting hammered by folks. It made me sad.

ConfigServer Security & Firewall, CSF, has been great so far. Reading through the main config file takes time but that’s good because it’s so well documented. I admit I’m not digging the extensive tuning needed to stop the seemingly endless squawking about IDS-related features (process resources, funny process names, custom cron scripts, etc.) so for now that’s turned off. I may fine-tune it soon.

Other things I like about CSF: optional automatic updates, built-in connection limits and rate limits, the idea of having separate allowed and ignored groups (allowed group may still be banned if not also in the ignored group, which is a nuanced distinction), lots of flexibility & customization, and it also has IPSET support for ultra-fast rule matching!

Fix the broken APF package on Debian/Ubuntu

R-fx networks logoThe Debian / Ubuntu package for Advanced Policy Firewall (APF) seems a bit unmaintained. By default it won’t run without some initial tweaking. Note that they probably want everyone to just download and run the installer from their site nowadays, but that’s not how I roll (usually).

In functions.apf, change the line


That allows the basic functionality of the software to work. Next, for the sake of upgrade-ability, I copy /etc/apf-firewall/conf.apf to /etc/apf-firewall/ Then the only change needed to the installed config is to source the .my file. Here’s the bottom of the file:

Since it won’t work if you try to source the internals.conf file twice, you need to make sure that the last line in the .my file is commented or removed. Now you can edit the other values in the .my file to your liking. Remember to turn off devel mode and change /etc/default/apf-firewall when you’re satisfied with any config changes, then restart the service in the usual way.

APF, fail2ban & more

APF is wonderful for a good-enough firewall solution for a lot of people. But what if you also want the power of another great tool, fail2ban?

The problem is, fail2ban wants to make changes directly to iptables, which APF is maintaining. Rules that fail2ban writes will be overwritten by APF. I found the solution is pretty straightforward: fail2ban with APF. This works really well.

No, I still didn’t think it was enough. Why? Because the “gamin” backend of fail2ban is buggy on Debian / Ubuntu, and tends to stop working after some time. So, I use the polling backend, which means it seems to wait about 30 seconds or so between checks of the log files. Today I saw a very persistent bot become unbanned and immediately get banned again, about 30-40 seconds later. How many attempts had it made on the SSH server while unbanned? I didn’t even stop to check, I just wanted to find a way to react more quickly.

Search for “limit connection rate linux” or whatnot on the Googs and you’ll find a number of sites with basically the same solution. It looks like this:

This works well if you’re manually (or using home-brewed scripts) to tamp down the baddies. But again it’s modifying iptables directly, which we don’t want. A couple more searches yielded a promising page that said to add any custom rules you need to APF’s postroute.rules file. However, using the plain -I or -A options to iptables without a line number doesn’t work right, since the order of rules, and what happens at the top and bottom of a ruleset are significant. So we need to add the rules just above the first blanket SSH ACCEPT rule. Here’s what I did in postroute.rules:

Notice that I also added the --name option, to prevent a conflict with the default name that APF uses.

Raspberry Pi SSH cipher speed

I was curious to see how quickly I could transfer files to my Pi using SSH rather than FTP. Obviously using FTP is way faster than almost any other method, but still I wanted to see how fast I could transfer data over SSH.

Here’s the time it took to transfer a 50 MB file to my Pi using different SSH ciphers.

I later re-tested the aes128-ctr cipher and it took about a second less than what I’d recorded initially. This boils down to:

  • Don’t use triple-DES ever, for both performance and security reasons
  • Most other ciphers give about the same performance, and are generally considered secure
  • arcfour is the fastest class of ciphers, but there is less trust in it from the crypto community. If you’re going to use it, try to avoid the base arcfour cipher and instead use the 128 or 256 version, which tosses out some of the initial bits as a precaution

monkeysphere project to add PKI to all the services

monkeysphere logo looks pretty cool. This project claims to want to apply the PKI web-of-trust to different services like web browsing and SSH. By querying the public keys stored on key servers, you wouldn’t need to guess that the remote site was providing their actual key the first time you connect, like you normally would when connecting to a new server or from a new client via SSH. You know what I’m talking about:

Yeah, that’s what I’m talking about. There’s no guarantee that the host you’re connecting to is the one you think it is unless you already know what the fingerprint is or you’re already using some other method for key exchange. The nice thing about this project is that they claim that there are absolutely no modifications needed to SSH to get this to work.

Link: monkeysphere