Intrigue Core v0.8 Released!

Announcing the immediate availability of Intrigue Core v0.8.0, the open asset and exposure discovery engine, and the core that powers the Intrigue.io Attack Surface Management platform.

Our 2020 – like many of yours – started out pretty chaotic, but with myself and the rest of the team grounded at home, it was a year of building and improving the platform. This release is a direct outcome of that year of heads-down development, and a signal that positive things can come out of the dumpster fire that was 2020. 

On the team front, Shpend Kurtishaj joined us mid-year as our first full time developer, and brought in new ideas and execution, building upon the excellent work of the existing team. Anas and Maxim were our key open source contributors this year, and yours truly also found a way to make the project full time, so you can expect to see much more goodness over the coming 12 months.

Here’s to a positive, safe, and healthy 2021!

New Features

In this (yet again) truly MASSIVE release, you’ll find the following key features: 

  • NEW! Supported VMWare and VirtualBox images
  • In-App Workflows powered by user-definable YAML files
  • Improved Vulnerability Discovery Capabilities
  • Asynchronous DNS and HTTP bringing new levels of speed
  • 30 new integrations and discovery capabilities 
  • 26 new vulnerability and misconfiguration checks

New Feature – Workflows

The most immediately distinctive feature of this release is something we’ve wanted to add support for, for many years: Automated Workflows. Workflows fully replace the now-legacy concept of “machines” in the platform with a simpler, friendlier YAML syntax but largely the same functionality. Meaning, they’re recursive by default, and when a new entity is created in a project with a workflow is attached to it, it will automatically schedule and run the relevant tasks that the workflow specifies. This, in combination with UX support for workflows makes it easier than ever to discover the attack surface of organizations; and further, for users to build out custom automation on top of the raw capabilities and tasks of Intrigue Core. Check it out:

You’ll notice now, after creating a new project, that you’re directed to the workflows screen and encouraged to enter as many “hints” as you can, and Core takes what you offer and builds upon it using the selected workflow: And if you like the old way of doing it (where a workflow is started on each new entity that’s discovered from a first import or task), you can do that too:

While there’s much to discuss about the other new features and capabilities, we’ll leave the deep dive for a follow-on post. In the meantime, check out some of the new capabilities below.

New Feature – VMWare and Virtualbox Images

One of the most common pieces of feedback is a request to supply a pre-built image of Intrigue Core for ease of use getting started. By popular demand, we now support both VirtualBox and VMWare images, and you can find them in the Getting Started section.

More New Capabilities

Many of these new capabilities are worthy of a post on their own, and there’s simply so many, that all we can do is point out major highlights and trends, such as:

  • Integrations to search and pull metadata from Mobile App stores like iOS and Android
  • Unauthenticated and automated API Endpoint discovery
  • Content discovery on app endpoints using @joohoi’s Ffuf, link extraction and other entity-identification techniques
  • Integration of great open source tooling like Subfinder, and Naabu
  • Deeper integration with great services such as C99, BinaryEdge, Spyse, and Zetalytics (there are so many now!)
  • Authenticated Integrations to pull DNS Zones (aws_route53, cloudflare_dns, etc)

The full list of individual new tasks in v0.8.0 is as follows

  • AWS Route53
  • Cloudflare Zones
  • DNS Search TLS Cert Names
  • Naabu Scan
  • SaaS ServiceNow Check
  • SaaS ServiceNow Open KB Articles
  • Search 42matters API for Android/iOS apps
  • Search Apptweak API for Android/iOS apps
  • Search Azure Blob
  • Search BinaryEdge Open Databases
  • Search c99 Subdomainfinder
  • Search DnSimple
  • Search Farsight DNSDB
  • Search Hostio
  • Search Mnemonic
  • Search NeutrinoAPI
  • Search Recon.dev
  • Search Spyse
  • Search Spyse Cert
  • Search Spyse Domain
  • Search WhoisXMLAPI (Reverse Whois)
  • Subfinder
  • URI Brute Generic Content
  • URI Bruteforce Vhosts
  • URI Check API Endpoint
  • URI Check Retire.js
  • URI Extract Linked Hosts
  • URI Extract Tokens
  • URI Ffuf Content Discovery
  • WordPress Enumerate Leaked Logs

If you’d like to know more, you can find descriptions for each task here.

New Entities

To support all these great capabiltiies, you’ve gotta be able to represent the data types, and thus, the following new entities have been added since the last release. While an entity itself might not be exciting, the ability to open up new use cases brings fun challenges and you can expect even more entities in 2021.

  • AndroidApp – Android Mobile Application
  • ApiEndpoint – A HTTP based API endpoint
  • IosApp – IOS Mobile Application
  • MailServer – A Mailserver (MX)
  • UniqueKeyword – A globally unique keyword that can be reliably searched
  • UniqueToken – An api key or analytics id

New Vulnerability Checks

On some days this last year, it felt like literally every webapp and/or network appliance was under threat. 2020 did bring the “wow” CVEs, such as the F5 BigIP bug or RCEs in Sharepoint, Exchange, GlobalProtect…. yep… wow. The checks we now support are below, and the best thing is that these are all automatically driven by fingerprinting. If you find a GlobalProtect instance, and vulnerability checks are enabled for a project, it’ll automatically be tested. Attack surface enumeration should be easy – and ACCURATE – and these checks go a long way in making that a reality.

  • vuln/atlassian_dataexposure_cve_2020_14179
  • vuln/cisco_asa_limited_file_read_cve_2020_3452
  • vuln/cisco_asa_path_traversal_cve_2018_0296
  • vuln/citrix_netscaler_codeinjection_cve_2020_8194
  • vuln/craft_cms_seomatic_cve_2020_9757
  • vuln/f5_bigip_configuration_utility_cve_2020_5902
  • vuln/hadoop_yarn_unathenticated_resourcemanager
  • vuln/icewarp_xss_cve_2020_8512
  • vuln/microsoft_exchange_cve_2020_0688
  • vuln/microsoft_exchange_cve_2020_16875
  • vuln/microsoft_sharepoint_cve_2020_16952
  • vuln/mobileiron_multiple_cves
  • vuln/nextjs_path_traversal_cve_2020_5284
  • vuln/paloalto_globalprotect_check_cve_2020_2021
  • vuln/saas_gitlab_open_reg_check
  • vuln/solarwinds_orion_code_compromise
  • vuln/sonatype_nexus_cve_2020_10204
  • vuln/sonicwall_cve_2020_5135
  • vuln/telerik_crypto_weakness_cve_2017_9248
  • vuln/tomcat_ghostcat_cve_2020_1938
  • vuln/tomcat_persistent_manager_cve_2020_9484
  • vuln/wordpress_file_manager_command_injection_rce
  • vuln/wordpress_loginizer_cve_2020_27615

New Threat Checks

While threat discovery and enrichment is still a nascent use case for the engine, this release brings more goods (thank you Anas!), with even more direct integrations of high quality threat feeds to verify if a given IoC (entity) was found in their database, and where possible – the reason why. Expect this use case will continue to steadily improve in the new year.

  • threat/search_apility
  • threat/search_badips
  • threat/search_blcheck_list
  • threat/search_blocklistde
  • threat/search_dshield
  • threat/search_emerging_threats
  • threat/search_fraudguard
  • threat/search_greynoise
  • threat/search_ibm_x_force
  • threat/search_ipqs
  • threat/search_ipqs_emailaddress
  • threat/search_pulsedive
  • threat/search_talos_blacklist

BUGFIXES 

Luckily we had no bugs in the last release, so this one will continue that illustrious tradition of perfect and bug-free software. (Just kidding, there were simply way too many to mention. You know how to find them.) Security fixes, feature fixes, and all around improving the user experience were a big focus.

THANK YOU 

No core release to date has been simple, and this one has been well over a year in the making. It would not have been possible without the following fine folks, and so a thank you is well deserved:

  • First and foremost, Thank you to Intrigue.io customers, for your support and ideas that make this open source project grow!
  • Thank you to my wonderful wife Jessica and to all of the contributors’ families for supporting the significant time this project requires
  • Thank you @shpendk, for joining us as the first full-time contributor, and for tackling the ugliest challenges of the codebase
  • Thank you @bensalah_anas for consistently driving powerful new cases and capabilities in the platform
  • Thank you Maxim Gofnung, for digging right into the guts of the code with huge enthusiasm
  • Thank you @joeuser47 for the friendly and helpful support in our slack channel
  • Thank you to the folks building powerful open source tooling, particularly @errbysam, @joohoi and @pdiscoveryio
  • Thank you to the teams building innovative APIs, including Zetalytics, Spyse, SecurityTrails, BinaryEdge, Greynoise, Recon.dev and so many more
  • Thank you to the researchers who regularly share techniques and ideas … @th3g3nt3lman, @nahamsec, and so many others
  • Thank you @ebellis and @kennasecurity for your incredible long-term support of this project
  • Finally, thank you to the many open source users and contributors who have provided feedback, support, ideas.

So with that …. and the piece of mind that 2021 is looking up – bringing even more capabilities and velocity to this project, go and get started now! Try it out and send feedback via Email, Slack, or Twitter. Have fun, and keep us posted with any and all feedback!

-jcran

Intrigue Core v0.7 Released!

Oh. Hey! Wow. You look, better, even, … i mean … you’re practically glistening. It’s been a year, hasn’t it? You must be working out. What have we been up to? Oh. I’m glad you asked! (PS – if you want to get straight to the goods, go here.)

Ready to go? Let’s dig in.

One underlying but prevailing theme of this release is “scaling up”. As we operationalized more engines over the last year to support our efforts in the Intrigue.io service, we needed a proper process management system, and to split out supporting components to their own managed processes. These are services such as:

  • Headless Chrome (for screen grabs, fingerprinting JS, etc)
  • Apache Tika (for parsing pretty much every file format on the planet, safely)
  • An EventMachine-based DNS Resolver for super fast resolution

And once these components were properly managed, database optimization became a focus, getting into the gory guts of Postgres and finally driving the ability to past a million entities per project. (Try the machines feature after running “create_entity” on your domain of choice :], and you’ll see.)

So with that work in place, we focused on a new and improved “Standalone” Docker Image as part of this release, which (finally, we know!) uses Docker’s volume support to allow you to save your data locally. No more starting from scratch each time you spin up an image!

Another key feature of this release is the all-new issue tracking system. Issues are now a first class object – like Entities – and are our way to capture vulnerabilities, misconfigurations, and other findings which should be brought to the attention of an analyst.

This release also adds some other oft-requested features including SSL by default and a much more in-depth automated scoping engine. More on that below.

Even with several major new features in this release, it’s hard to overstate how much has changed in the codebase over last 12 months. And we’re not slowing down. As always with a new release, this one brings tons of new tasks, entities, improvements and bugfixes (…read on for details)

Automated Scoping

One major feature since the last release that will be very visible when you use a Machine is the automated scoping functionality.

Scoping takes the seeds (Entities) you provide and uses them as guidance for what to scan, and more importantly, what NOT to scan. In previous versions, thiis was a blacklist, but now there’s some smarts behind it.

Try it by using the “create_entity” task and the ” machine with a few iterations.

You’ll notice right away on the entities page that some are hidden by default. This is the scoping engine.

You can view them by selecting “Show unscoped” and “Show hidden” on the search screen.

Automated scoping results in amazonaws.com entities (and others) being hidden by default…

Give it a try and let us know what you think!

New Discovery Tasks

Okay, so this bit is going to get a little long. And, while it’s been a year, many of these tasks were built and refined over just the last 3 months, thanks in no small part to @anasbensalah who joined as a committer this year.

This v0.7 release includes 23 shiny new tasks, bringing the current total to 124 discovery tasks.

Ready to dig in? The new tasks are in alphabetical order below and each individually linked to their implementation for those brave enough to dive into the codebase.

  • dns_lookup_dkim – Attempts to identify all known DKIM records by iterating through known selectors.
  • dns_morph – Uses the excellent DNSMORPH to find permuted domains.
  • email_brute_gmail_glxu – Uses an enumeration bug in the mail/glxu endpoint on gmail to check account existence.
  • gitrob – Uses the excellent Gitrob to search a given GithubAccount for committed secrets.
  • saas_google_calendar_check – Checks to see if public Google Calendar exists for a given user.
  • search_alienvault_otx_hashes – This task searches AlienVault OTX via API and checks for information related to a FileHash.
  • search_binaryedge – This task hits the BinaryEdge API for a given IpAddress, DnsRecord, or Domain, and creates new entities such as NetworkServices and Uri, as well as associated host details.
  • search_binaryedge_risk_score – This task hits the BinaryEdge API and provides a risk score detail for a given IPAddress. It can optionally create an issue for high risk IPs.
  • search_binaryedge_torrents – This task hits the BinaryEdge API for a given IPAddress, and tells us if the address has seen activity consistent with torrenting
  • search_dehashed – This task hits the Dehashed API for leaked accounts.
  • search_grayhat_warfare – This task hits the Grayhat Warfare API and finds AwsS3Buckets.
  • search_hunter_io – This task hits the Hunter.io API. EmailAddresses are created for a given domain.
  • search_spyonweb – This task hits the SpyOnWEB API for hosts sharing the same IPAddress, Domains, or AnalyticsId.
  • uri_brute_focused_content – Check for juicy content based on the site’s technology stack (This is a special task, part discovery and part vuln check, so it’s listed below, as well).
  • uri_check_subdomain_hijack – Checks for a specific string on a gievn uri, and creates a hijackable subdomain issue if it matches.
  • well_known_gather_and_parse – Checks for files in the /.well-known/ directory, as defined in RFC5785.
  • wordpress_enumerate_plugins -If the provided Uri is running WordPress (as fingerprinted by Ident), this’ll enumerate the plugins
  • wordpress_enumerate_users – If the target’s running WordPress, this’ll enumerate the users

As if that wasn’t enough, the following new tasks help determine if a given Domain or DnsRecord is compromised or otherwise blocked by utilizing the content blocking provided in the respective provider’s DNS service. They’re all very similar in implementation, but may provide different results depending on the provider. These is more great work from Anas.

New Entity Types

Reading carefully above, you might notice some of the tasks are introducing new entity types, and for that matter, new use cases.

This release brings two new entities. First, the “AnalyticsId” which represents an id from an analytics provider like NewRelic or Google. Secondly the “FileHash” entity brings us the ability to represent an md5 or sha1 hash as an entity.

Definitely check the tasks creating these entities (search_spyonweb, and search_alienvault_otx_hashes, respectively) above and have a play around with them. Feedback is very welcome. If you find them useful or have ideas on ways we could improve, let us know and we’ll add support for more providers and hash types.

Major Improvements to Tasks

The following were significantly overhauled during the course of this release, and worth checking out again if you have tried the task previously. These now have a lot more functionality.

  • search_have_i_been_pwned
  • search_phishtank
  • search_shodan
  • search_certspotter
  • scrape_publicwww
  • search_alienvault_otx
  • import/umbrella_top_sites

Bugfixes

Luckily we had no bugs in the last release, so this one will continue that tradition. (Just kidding, there were simply way too many to mention. You know how to find them.)

New Vuln Checks

If you were following along over the last year, you probably noticed a significant amount of effort went into testing for vulnerabilities and misconfigurations.

The 0.7 release brings 9 new vuln check tasks, each linked below.

Now that we have a better system for finding and reporting them (blog post forthcoming), you can expect to see more of this kind of shiny goodness in the future.

Thank You!!!

This release has been well over a year in the making and would not have been possible without the following contributors of code, ideas, and bugs. Make sure to say thank you the next time you see these fine folks.

So that’s it you say? Well, it’s as much as we could recollect of the blur that was 2019. There’s surely a bunch of neat stuff that we’ve forgotten and you’ll discover when you get started. So with that, go get started now!

Try it out and send feedback via Email, Slack or Twitter. Have fun, and let’s not let it go another year before we do this again!

Intrigue Core v0.6 Released!

Today marks the release of Intrigue Core v0.6, bringing a bunch of new functionality including: automatic inference of CVEs and vulnerabilities on discovered applications, new entities such as “Finding” and “Domain” types, and usability features such as the ability to import a list and new analysis views to easily see expired certs or out-of-compliance cipher suites.

See below for details on how to get it, and enjoy!

Major Features 

  • Added a vulnerability inference capability based on ident’s fingerprinting
  • Added support for a “Domain” entity, representing a top-level domain (vs a standard DnsRecord)
  • Added initial support for “Finding” entity, enabling tasks to easily surface actionable findings
  • Added the ability to import and run a set of tasks on a list (thanks @hollywoodmarks!)
  • Added new analysis views (ciphers, javascript, cves)
  • Added support for task “Notifiers” & an initial (Slack) notifier
  • Adjusted application fingerprinting to a new standalone library, “intrigue-ident”
  • Adjusted Enrichment tasks to run in-line, eliminating a variety of race conditions when running machines

Minor Features

  • Added support for go-based utilities in the image via util/bootstrap.sh setup script
  • Added support for RDAP, enabling new RIR Whois lookups (Afrinic, Lacnic, Apnic)
  • Adjusted handling of network services – all types are now subtype of “NetworkService”
  • Adjusted handling of saas services – all types are now subtype of “WebAccount”
  • Adjusted base image to Ubuntu 16.04 (util/bootstrap.sh setup)
  • Upgraded to Bundler 2
  • Upgraded to latest GeoLite2-City

Major Bugs

  • Fixed bug causing the system to throw a runtime error when an API key is missing
  • Fixed bug in util/bootstrap.sh script that would cause a hang due to grub-pc (thanks @bpmcdevitt!)
  • Fixed bug that would cause memory leaks in Chrome headless browser teardown
  • Mounted an out of control rollercoaster of regex bugs and arrived victorious
  • … literally hundreds of other minor bugs

New Tasks:

A huge thanks to the following folks who submitted PRs and/or contributed to this release:

You can download and run Intrigue Core v0.6 immediately using one of the following guides:

If you’re interested in contributing to the effort to make Intrigue Core the best OSINT and security intelligence gathering framework around, please jump in our chat and say hello!

Intrigue Core v0.5 Released!

Announcing the immediate availability of Intrigue Core v0.5. This release is  heavy with system level improvements and bugfixes. There are also key investments made in this release with the removal of (the now-deprecated) PhantomJS and the  integration of Chrome Headless browser for screenshots and javascript fingerprinting.

A number of security-related improvements were made as well, with all platforms (EC2, Docker, Vagrant) all relying on the same core system setup scripts and no longer using “rbenv sudo” to run as the root user.

This release continues to make progress on fingerprinting with a number of new fingerprints being added.

Enjoy, and Happy 4th!

– jcran

New Functionality: 

  • Update to Ruby 2.5.1
  • Moved headless browsing components from PhantomJS to Chrome Headless
  • Consolidated & simplified system bootstrap for Docker / Vagrant / AMI
  • Removal of rbenv sudo
  • Support for dynamic task queues
  • Lots of new fingerprints (pfsense, telerik, atlassian, etc)
  • Faster exports (streaming JSON) & dynamically generated export UI

New Tasks:

How to get it 

You can download and run Intrigue Core 0.5 immediately on the following platforms.

Announcing … Intrigue Core v0.4!

Announcing the immediate release of Intrigue Core v0.4!

In this release, you’ll find:

If that weren’t enough, we added a total of 19 new modules:

This release also had a ton of work over the last few weeks as we prepared for RSA 2018. At RSA, Ed Bellis & I discussed “Recon for Defenders” and offered up a few specific CVEs and software that defenders must be very quick to patch – particularly when it’s available for scanning.

As part of that work, we spun up around over 100 simultaneous instances of Intrigue Core, and used these instances to scan the F500 using the “org_asset_discovery_active” strategy and a single domain seed. After running for 10 hours total, we had the world’s first ~complete attack surface scan of the entire F500. Pretty sweet.

We then anonymized and released the data from those tests. As you dig into them, you’ll notice a large number of servers and applications exposed at the perimeter that were still running vulnerable versions of this software at the time of testing.

Digging through the results, I realized that Core’s fingerprinting capabilities needed a lot of work, and so shortly after the talk, I sat down and overhauled the application fingerprinter, creating a pluggable system. Now, for each URI that the system wants to fingerprint, any piece of software can plug in a set of checks. This architecture us to minimize the number of HTTP requests we make, while still supporting a large number of fingerprints.

Now that v0.4 is available,  you can now immediately download and run Core through the normal AMI, Dockerfile, or (new in this release) in a local or remote VM using Vagrant!