Website scanning in-depth

Scanning a website seems easy, and it is if you just do a one-off, single scan for a url.

Things get more interesting when you host your website on multiple servers (for better performance or reliability). You probably also have both ipv4 and ipv6 addresses available. Your website runs on HTTPS, and you want your visitors to be able to find you without typing in the protocol too. So you also have HTTP configured. That’s two protocol versions, on two versions of ip addresses and maybe multiple hosts.

Some websites run in the cloud. You can limit this to a specific cloud in a specific country (which most governments do with their websites), but you can also have the cloud provider figure out what the best spot is. If you do this with Azure, your website will get served from the nearest Azure cloud. ShadowTrackr has nodes all over the world, and this means we’ll be able to detect your website in multiple clouds. That’s on purpose of course, but it does complicate things.

Then there are CDNs like Cloudflare and Akamai. You host your website on a server where the CDN can reach you, and they handle all your visitor requests. You’ll need a trick to point your visitors at the CDN of course, and this is where it gets ugly for scanners. There are multiple ways of doing this and these can be mixed and matched. On top of this some CDN hire subcontractors that are really hard to attribute and you might end up detecting Vodafone instead of Akamai.

It seemed so easy to scan a website, but in practice it can get really complex. The goal has always been for ShadowTrackr to detect all your website instances on all internet-reachable hosts, including clouds and CDNs. I had underestimated how complex this is and did not achieve the goal from the start. After getting it wrong a couple of times, this week’s update features a much improved algorithm. This might result in a storm of new websites being found on your timeline. I’m on it and regularly clean things up until they are all properly ingested and monitored.

If you do find irregularities, or have any other questions, drop me a line.

UX and GUI improvements

The last two weeks were mostly spent on improving the graphical user interface (GUI) and user experience (UX).

ShadowTrackr is continuously growing. Sometimes functionality is improved, sometimes features are added, and things get "bolted on". Over time, clear and readable webpages will devolve into Frankenpages.

Two weeks of tender love and care should have fixed that now. I’m happy with the new layout. Warnings and errors are more consistent, and many asset properties are now clickable links. These links will generate a search query and show you all other assets with the same properties.

Finally, the exports have improved too, try mailing yourself a certificate, website, host or domain page. Just click "mail report" in the context menu (three dots, right upper corner). These reports are intended to mail to the IT staff that has to fix things. You can directly mail them from your account, without having to go trough copy-paste hell. Hopefully this lower the bar and encourages you to improve your security posture even more.

I hope you like the update as much as I do. If you have any suggestions or special requests, please let me know!

CVE checks on detected software

Some of you might have seen the CVE mentions on the software reports page. They have silently been in beta for a while, and now it’s time to step up.

ShadowTrackr runs automatic checks to detect software running on your websites and servers, and it always suprised me how often it’s possible to determine the exact version number too. Since we have this information anyway, we should do something useful with it, and that’s were the CVE checks come in.

As of today we maintain a current database of all released CVEs. Everytime we find software and version information on one of your assets, we’ll check it against the known vulnerabilities in the CVE database. If we find a vulnerability, you’ll get a warning. And if it’s a critical vulnerability (meaning a CVSS score above 9), we’ll list it as a problem and urge you to fix it immediately.

Any evil hacker who wants to have a go at you will go through the same process of finding out what software you run and checking if there are any known vulnerabilities. So, this should be a major step in reducing your attack surface. Check out the results on the software report page and get started!