One common challenge with version detection and inference-based vulnerability analysis – the kind we do in Ident and Core- is graceful handling of versions which appear to be vulnerable baed on their versions, but in reality have been backported.
What do we mean by backporting? Well, some operating systems (*ahem* Red Hat *ahem*) will apply security fixes and patches to previous version of a software package, but not update the software version. Rather, they update their own package version. The software keeps reporting the vulnerable version, but users are protected. This is commonly referred to as backporting.
This is actually a good thing for users, in general, but can result in some false positives when performing unauthenticated vulnerability assessments. So we need to prepare for it. To address this, we added a bit of sophistication to the ident library, which allows us to detect this behavior in cases where we know the operating system, and respond accordingly.
Below, you can see how our dynamic version detection for PHP, OpenSSL and Apache, all handle this by looking for the existence of RHEL, Red Hat or CentOS, and appending the “(Backported)” bit to the version string. Note that this might result in some false negatives, so we retain the version string for users.
Oh. Hey! Wow. You look, better, even, … i mean … you’re practically glistening. It’s been a year, hasn’t it? You must be working out. What have we been up to? Oh. I’m glad you asked! (PS – if you want to get straight to the goods, go here.)
Ready to go? Let’s dig in.
One underlying but prevailing theme of this release is “scaling up”. As we operationalized more engines over the last year to support our efforts in the Intrigue.io service, we needed a proper process management system, and to split out supporting components to their own managed processes. These are services such as:
Apache Tika (for parsing pretty much every file format on the planet, safely)
An EventMachine-based DNS Resolver for super fast resolution
And once these components were properly managed, database optimization became a focus, getting into the gory guts of Postgres and finally driving the ability to past a million entities per project. (Try the machines feature after running “create_entity” on your domain of choice :], and you’ll see.)
So with that work in place, we focused on a new and improved “Standalone” Docker Image as part of this release, which (finally, we know!) uses Docker’s volume support to allow you to save your data locally. No more starting from scratch each time you spin up an image!
Another key feature of this release is the all-new issue tracking system. Issues are now a first class object – like Entities – and are our way to capture vulnerabilities, misconfigurations, and other findings which should be brought to the attention of an analyst.
This release also adds some other oft-requested features including SSL by default and a much more in-depth automated scoping engine. More on that below.
Even with several major new features in this release, it’s hard to overstate how much has changed in the codebase over last 12 months. And we’re not slowing down. As always with a new release, this one brings tons of new tasks, entities, improvements and bugfixes (…read on for details)
One major feature since the last release that will be very visible when you use a Machine is the automated scoping functionality.
Scoping takes the seeds (Entities) you provide and uses them as guidance for what to scan, and more importantly, what NOT to scan. In previous versions, thiis was a blacklist, but now there’s some smarts behind it.
You’ll notice right away on the entities page that some are hidden by default. This is the scoping engine.
You can view them by selecting “Show unscoped” and “Show hidden” on the search screen.
Give it a try and let us know what you think!
New Discovery Tasks
Okay, so this bit is going to get a little long. And, while it’s been a year, many of these tasks were built and refined over just the last 3 months, thanks in no small part to @anasbensalah who joined as a committer this year.
This v0.7 release includes 23 shiny new tasks, bringing the current total to 124 discovery tasks.
Ready to dig in? The new tasks are in alphabetical order below and each individually linked to their implementation for those brave enough to dive into the codebase.
dns_lookup_dkim – Attempts to identify all known DKIM records by iterating through known selectors.
dns_morph – Uses the excellent DNSMORPH to find permuted domains.
email_brute_gmail_glxu – Uses an enumeration bug in the mail/glxu endpoint on gmail to check account existence.
gitrob – Uses the excellent Gitrob to search a given GithubAccount for committed secrets.
As if that wasn’t enough, the following new tasks help determine if a given Domain or DnsRecord is compromised or otherwise blocked by utilizing the content blocking provided in the respective provider’s DNS service. They’re all very similar in implementation, but may provide different results depending on the provider. These is more great work from Anas.
Reading carefully above, you might notice some of the tasks are introducing new entity types, and for that matter, new use cases.
This release brings two new entities. First, the “AnalyticsId” which represents an id from an analytics provider like NewRelic or Google. Secondly the “FileHash” entity brings us the ability to represent an md5 or sha1 hash as an entity.
Definitely check the tasks creating these entities (search_spyonweb, and search_alienvault_otx_hashes, respectively) above and have a play around with them. Feedback is very welcome. If you find them useful or have ideas on ways we could improve, let us know and we’ll add support for more providers and hash types.
Major Improvements to Tasks
The following were significantly overhauled during the course of this release, and worth checking out again if you have tried the task previously. These now have a lot more functionality.
Luckily we had no bugs in the last release, so this one will continue that tradition. (Just kidding, there were simply way too many to mention. You know how to find them.)
New Vuln Checks
If you were following along over the last year, you probably noticed a significant amount of effort went into testing for vulnerabilities and misconfigurations.
The 0.7 release brings 9 new vuln check tasks, each linked below.
Now that we have a better system for finding and reporting them (blog post forthcoming), you can expect to see more of this kind of shiny goodness in the future.
This release has been well over a year in the making and would not have been possible without the following contributors of code, ideas, and bugs. Make sure to say thank you the next time you see these fine folks.
@kennasecurity (for being, well, awesome and helping this project grow)
So that’s it you say? Well, it’s as much as we could recollect of the blur that was 2019. There’s surely a bunch of neat stuff that we’ve forgotten and you’ll discover when you get started. So with that, go get started now!
Try it out and send feedback via Email, Slack or Twitter. Have fun, and let’s not let it go another year before we do this again!