The big vertical refactor

At the request of some organisations I added some code to scan servers for a particular problem. These organisations are marked as critical infrastructure, and they often have access to material that is in a responsible disclosure procedure before it’s available to the general public. If you ever have this problem along with PoC code (any language will do), I'm interested :-)

I’m happy to do this of course, but the old flood detection problem reared its ugly head again. This problem often occurs with larger organisations that run multiple servers on the same subnet behind the same firewall. If the firewall has flood detection enabled, a ShadowTrackr node that hits multiple ips in the same subnet behind that firewall will be blocked. The blocking is usually only for 5 minutes, but that’s enough to generate a lot of useless messages on the timeline. It’s a bit like shitposting on Twitter.

So far I’ve done scans horizontally, just like cencys (pdf alert). The solution for flood detection so far was a fancy algorithm that divided all checks and scans over the worker nodes in such a way that no node would ever hit multiple ips in the same subnet range within 5 seconds. 5 seconds are the default flood interval setting found in most big corporate firewalls. When adding more custom scans, this solution isn’t working anymore and the fancy algorithm will become a drag on the database while we’re scaling up too.

I’ve been wanting to refactor some parts of the code to offload more work from the database to the nodes anyway, and this seems like a good time to do it. This is what I’m spending most time on now, together with more vertically based host checks and scans. It’ll take some time to do properly and this means I’ll have put a hold on some other ideas for now. Bugs get priority of course, so please keep sending those.

More sources for news and dataleaks

Your keywords are now used for monitoring more sources in the categories you have already specified. On top of that, we've also added more countries. The US, UK and Netherlands were already available, and Australia, Canada and Germany are new. The newsites we monitor are typically the biggest national newspapers and some local newspaper in big cities.

We've also scoured the internet for more copy-paste sites to monitor. If you're not sure why this is useful, read up on it at haveibeenpwnd.com/pastes. There are sites that claim they will search 90+ datadumpsites for you. That is a bold claim, so we checked it.

A lot of the the 90+ copy-paste sites listed are dead links, have errors or have the domainname for sale. For those that do work there is no API or feed with recent posts available to search them. You can post a message and directly after posting you'll get a url linking to your post. If you don't share this url openly yourself, nobody will know about the post or de data you dumped in it. The 9 remaining sites that were useful are added to ShadowTrackr, but to be honest these are mostly sites to share code. In our experience spicy datadumps are rarely found on specific code sharing sites.

We have been monitoring 10 (from today this is 19) of the bigger pastesites for months now with keywords like "botnet". About 80% of the hits we get are from pastebin.com, which is by far the largest of the dumpsites if you look at both volume and the rate at which new posts appear. It's good to look around for changes in de dumpsite landscape every now and then, but at the moment the effort is unlikely to pay off and we better use the time for other items on the todo list.

If you do miss a newsite or dumpsite that you want us to monitor, or if you want to check if your favorite site is included, please contact support.

Well, that was stupid. According to our site stats there are more visitors from India and Ireland than from Canada. Newssites from those countries are now added too.

Why you will like Canary Tokens

Ever heard of Canary tokens? These are the digital equivalent of the canaries that were used to detect gas in coal mines. The coal mine canary dies of the gas before miners do and give the miners time to save themselves. Similarly, the canary tokens you plant in your systems can give you a heads up that a hacker is snooping around. This is typically in the reconnaissance fase and you might even have a chance to prevent lateral movement or exfiltration.

As with the real life canary, you can have a really low false positive rate. Given the huge amounts of false positives leading to "alert fatigue" in security teams this is really welcome. The trick of course is that you set up your canaries in such a way that no regular user is likely to trigger them. If you have a word document named "vulnerability report" in your home directory, nobody other than you should open it. And you of course no that opening this fake document sets of a trap and you leave it alone. If you're using the same trick to find snoopers in group directories, you better make sure that everyone in the group knows about the trap. If you're exclusively looking for evil hackers and not for internal leaks you could also consider having the directory hidden from normal users.

There are lots of different types of canaries available: fake documents, URL, trigger, DNS triggers SQL Server triggers, AWS triggers and more. You can easily expand on these with any action that you can translate in one of the existing triggers. An example is the SQL Server trigger, where a certain SQL statement will cause the database server to do a DNS lookup that in turn triggers the trap.

You can easily setup a canary token on canarytokens.org. The server running there seems to be running a bit behind on the github project, and we're thinking of running an instance at ShadowTrackr.com or another domain that might be a bit less of a giveway for those that monitor dns traffic. You can use ShadowTrackr to turn the webhooks that canarytokens call into push messages on your phone and log them to have an audit trail for post mortems. See this use case for an example.