Oh. Hey! Wow. You look, better, even, … i mean … you’re practically glistening. It’s been a year, hasn’t it? You must be working out. What have we been up to? Oh. I’m glad you asked! (PS – if you want to get straight to the goods, go here.)
Ready to go? Let’s dig in.
One underlying but prevailing theme of this release is “scaling up”. As we operationalized more engines over the last year to support our efforts in the Intrigue.io service, we needed a proper process management system, and to split out supporting components to their own managed processes. These are services such as:
- Headless Chrome (for screen grabs, fingerprinting JS, etc)
- Apache Tika (for parsing pretty much every file format on the planet, safely)
- An EventMachine-based DNS Resolver for super fast resolution
And once these components were properly managed, database optimization became a focus, getting into the gory guts of Postgres and finally driving the ability to past a million entities per project. (Try the machines feature after running “create_entity” on your domain of choice :], and you’ll see.)
So with that work in place, we focused on a new and improved “Standalone” Docker Image as part of this release, which (finally, we know!) uses Docker’s volume support to allow you to save your data locally. No more starting from scratch each time you spin up an image!
Another key feature of this release is the all-new issue tracking system. Issues are now a first class object – like Entities – and are our way to capture vulnerabilities, misconfigurations, and other findings which should be brought to the attention of an analyst.
This release also adds some other oft-requested features including SSL by default and a much more in-depth automated scoping engine. More on that below.
Even with several major new features in this release, it’s hard to overstate how much has changed in the codebase over last 12 months. And we’re not slowing down. As always with a new release, this one brings tons of new tasks, entities, improvements and bugfixes (…read on for details)
One major feature since the last release that will be very visible when you use a Machine is the automated scoping functionality.
Scoping takes the seeds (Entities) you provide and uses them as guidance for what to scan, and more importantly, what NOT to scan. In previous versions, thiis was a blacklist, but now there’s some smarts behind it.
You’ll notice right away on the entities page that some are hidden by default. This is the scoping engine.
You can view them by selecting “Show unscoped” and “Show hidden” on the search screen.
Give it a try and let us know what you think!
New Discovery Tasks
Okay, so this bit is going to get a little long. And, while it’s been a year, many of these tasks were built and refined over just the last 3 months, thanks in no small part to @anasbensalah who joined as a committer this year.
This v0.7 release includes 23 shiny new tasks, bringing the current total to 124 discovery tasks.
Ready to dig in? The new tasks are in alphabetical order below and each individually linked to their implementation for those brave enough to dive into the codebase.
- dns_lookup_dkim – Attempts to identify all known DKIM records by iterating through known selectors.
- dns_morph – Uses the excellent DNSMORPH to find permuted domains.
- email_brute_gmail_glxu – Uses an enumeration bug in the mail/glxu endpoint on gmail to check account existence.
- gitrob – Uses the excellent Gitrob to search a given GithubAccount for committed secrets.
- saas_google_calendar_check – Checks to see if public Google Calendar exists for a given user.
- search_alienvault_otx_hashes – This task searches AlienVault OTX via API and checks for information related to a FileHash.
- search_binaryedge – This task hits the BinaryEdge API for a given IpAddress, DnsRecord, or Domain, and creates new entities such as NetworkServices and Uri, as well as associated host details.
- search_binaryedge_risk_score – This task hits the BinaryEdge API and provides a risk score detail for a given IPAddress. It can optionally create an issue for high risk IPs.
- search_binaryedge_torrents – This task hits the BinaryEdge API for a given IPAddress, and tells us if the address has seen activity consistent with torrenting
- search_dehashed – This task hits the Dehashed API for leaked accounts.
- search_grayhat_warfare – This task hits the Grayhat Warfare API and finds AwsS3Buckets.
- search_hunter_io – This task hits the Hunter.io API. EmailAddresses are created for a given domain.
- search_spyonweb – This task hits the SpyOnWEB API for hosts sharing the same IPAddress, Domains, or AnalyticsId.
- uri_brute_focused_content – Check for juicy content based on the site’s technology stack (This is a special task, part discovery and part vuln check, so it’s listed below, as well).
- uri_check_subdomain_hijack – Checks for a specific string on a gievn uri, and creates a hijackable subdomain issue if it matches.
- well_known_gather_and_parse – Checks for files in the /.well-known/ directory, as defined in RFC5785.
- wordpress_enumerate_plugins -If the provided Uri is running WordPress (as fingerprinted by Ident), this’ll enumerate the plugins
- wordpress_enumerate_users – If the target’s running WordPress, this’ll enumerate the users
As if that wasn’t enough, the following new tasks help determine if a given Domain or DnsRecord is compromised or otherwise blocked by utilizing the content blocking provided in the respective provider’s DNS service. They’re all very similar in implementation, but may provide different results depending on the provider. These is more great work from Anas.
New Entity Types
Reading carefully above, you might notice some of the tasks are introducing new entity types, and for that matter, new use cases.
This release brings two new entities. First, the “AnalyticsId” which represents an id from an analytics provider like NewRelic or Google. Secondly the “FileHash” entity brings us the ability to represent an md5 or sha1 hash as an entity.
Definitely check the tasks creating these entities (search_spyonweb, and search_alienvault_otx_hashes, respectively) above and have a play around with them. Feedback is very welcome. If you find them useful or have ideas on ways we could improve, let us know and we’ll add support for more providers and hash types.
Major Improvements to Tasks
The following were significantly overhauled during the course of this release, and worth checking out again if you have tried the task previously. These now have a lot more functionality.
Luckily we had no bugs in the last release, so this one will continue that tradition. (Just kidding, there were simply way too many to mention. You know how to find them.)
New Vuln Checks
If you were following along over the last year, you probably noticed a significant amount of effort went into testing for vulnerabilities and misconfigurations.
The 0.7 release brings 9 new vuln check tasks, each linked below.
- uri_brute_focused_content – Also mentioned above, this discovery task also checks for many types of vulns. A few highlighted issues it can identify: Jira “Filters” Info Disclosure, Palo Alto GlobalProtect, Laravel Configuration Leak, Splunk Info Disclosure
Now that we have a better system for finding and reporting them (blog post forthcoming), you can expect to see more of this kind of shiny goodness in the future.
This release has been well over a year in the making and would not have been possible without the following contributors of code, ideas, and bugs. Make sure to say thank you the next time you see these fine folks.
- @kennasecurity (for being, well, awesome and helping this project grow)
- @bensalah_anas (many, many new tasks and improvements)
- @mosesrenegade (well_known_gather_and_parse task)
- @securitykitten (improvements to scrape_publicwww, Dockerfiles)
- @shpendk (Docker volume support)
- @bcoles (fingerprinting / ident improvements)
- @attritionorg (fix to cve names)
- @netevert (dnsmorph creator)
- @michenriksen (gitrob creator)
- @jgamblin (inspiration and ideas on many new tasks)
- @danielherrera (inspiration for uri_brute_focused_content, email_brute_gmail task)
- @orange_8361 and @mehqq_ (inspiration on VPN vuln checks)
- @en0f (inspiration and details on weblogic vulns)
- @digidems team (inspiration for dns_lookup_dkim task)
- @x0rz (inspiration for email_brute_gmail task)
So that’s it you say? Well, it’s as much as we could recollect of the blur that was 2019. There’s surely a bunch of neat stuff that we’ve forgotten and you’ll discover when you get started. So with that, go get started now!