Using workflows to discover attack surface

So you’ve gotten an instance of intrigue-core up and running using the Getting Started guide, but what now!? Give workflows a try. Here’s now.

Create a new project, let’s set this one up on with a name of Mastercard – they run a public bounty on Bugcrowd, and likely have a lot of interesting systems on the Internet.

Once created, you’ll drop into the “Start a Workflow” page:

Hit “Profile an Organization” and we’lll add in as many Mastercard domains that we know exist. Bounty program pages can be instructive for finding domains and other seeds. For now, we’ll just start with mastercard.com, but it’s always better to have more seeds.

Now, hit submit, and notice you’re dropped to the “logs” page so you can monitor what’s happening.

You’ll want to refresh a couple times, and as you do, you’ll see more tasks being automatically scheduled.

Now this is all good and well, but where are the results? For that, you’ll want to hit “Entities” in the top menu. When you do, you’ll see something like the following. Now, you might not recognize every entity in here, and that’s okay. If you look to the far right, you’ll see some are scoped, and some are not. This is Core’s auto-scoping capability at work. If it finds something that it’s not sure belongs to our target organization, it’ll remain unscoped.

In fact, let’s try hitting ‘Only Scoped’ in the search section on the left.and filter only by DnsRecord by selecting that entity in the ‘Types’ list. There we go, that looks a bit more familiar.

Also notice the details give us a bit more info about what each entity is, and we can always click into an entity to learn more about it. The enrichment process ensures that we have a lot of detail per entity.

  • The Ancestors section tells us what the original “seed” entity we entered that resulted in us finding this entity. There can be many ancestor entities.
  • The Aliased Entities section tells us what other entities this DnsRecord resolved to. This creates a group and makes it easier to see if this is a service, or a host, or some combination thereof.
  • The Tasks that Created this Entity section shows us that this domain was found through the dns_brute_sub task, in other words, through Subdomain Bruteforce!
  • The Tasks Run on this Entity section tells us what tasks were auto-queued and run with this entity. This is entirely based on the workflow that we chose ‘Profile an Organization’. If we chose to hit the blue button in the upper right and start a new task, that would also show up in this list.

Finally, notice that full details are preserved in the “Entity Details” section – details can be arbitrary length and help give us more context about the entity.

Now, hit entities again, and let’s take a look at the ‘grouped’ view, now that we know what “Aliased Entities” mean. This is a helpful way to better understand a given organization’s infrastructure. In this case, it looks like there are quite a few subdomains resolving to the 216.119.209.64 address, it is likely a load balancer.

Okay, well let’s slice it a different way now. Hit the Analysis -> Domains page, and you’ll see the top level domains, sorted by count, which gives us a view of the first and third party domains around Mastercard:

If we’re interested to see, for example, priceless.com we can click on that and it’ll drop us back into search:

Now we might want to add the priceless.com domain into this scan. While it’s almost always better to have the full list of domains at the beginning, we can add it, by browsing to ‘Start’ -> ‘Start a new workflow’ and adding priceless.com in. However, if we’re not sure about a given domain, we might want to find everything that mentions priceless.com by CTRL-clicking on the entity type and searching all entity types:

Aha, that top entity https://www.priceless.com:443 might tell us more, let’s click on tha. Looks like it was fingerprinted with Facebook, and Jquery, so it’s probably a marketing or company page. When we browse, you can see a screenshot has already been taken, as well as the certificate stored in the etails, giving us a very clear indication that this a Mastercard assets. And in fact, it’s already been scoped automatically.

Okay, so let’s add it to the project by kicking off another workflow on it

And we add the domain as a seed and hit submit.

In this way, you can continuously kick off workflows and keep the system iterating on new entities you discover. .Workflows are incredibly powerful, as they automate the individual tasks inside Intrigue Core. For more infomration on workflows, see our Workflow help page.

Intrigue Core v0.8 Released!

Announcing the immediate availability of Intrigue Core v0.8.0, the open asset and exposure discovery engine, and the core that powers the Intrigue.io Attack Surface Management platform.

Our 2020 – like many of yours – started out pretty chaotic, but with myself and the rest of the team grounded at home, it was a year of building and improving the platform. This release is a direct outcome of that year of heads-down development, and a signal that positive things can come out of the dumpster fire that was 2020. 

On the team front, Shpend Kurtishaj joined us mid-year as our first full time developer, and brought in new ideas and execution, building upon the excellent work of the existing team. Anas and Maxim were our key open source contributors this year, and yours truly also found a way to make the project full time, so you can expect to see much more goodness over the coming 12 months.

Here’s to a positive, safe, and healthy 2021!

New Features

In this (yet again) truly MASSIVE release, you’ll find the following key features: 

  • NEW! Supported VMWare and VirtualBox images
  • In-App Workflows powered by user-definable YAML files
  • Improved Vulnerability Discovery Capabilities
  • Asynchronous DNS and HTTP bringing new levels of speed
  • 30 new integrations and discovery capabilities 
  • 26 new vulnerability and misconfiguration checks

New Feature – Workflows

The most immediately distinctive feature of this release is something we’ve wanted to add support for, for many years: Automated Workflows. Workflows fully replace the now-legacy concept of “machines” in the platform with a simpler, friendlier YAML syntax but largely the same functionality. Meaning, they’re recursive by default, and when a new entity is created in a project with a workflow is attached to it, it will automatically schedule and run the relevant tasks that the workflow specifies. This, in combination with UX support for workflows makes it easier than ever to discover the attack surface of organizations; and further, for users to build out custom automation on top of the raw capabilities and tasks of Intrigue Core. Check it out:

You’ll notice now, after creating a new project, that you’re directed to the workflows screen and encouraged to enter as many “hints” as you can, and Core takes what you offer and builds upon it using the selected workflow: And if you like the old way of doing it (where a workflow is started on each new entity that’s discovered from a first import or task), you can do that too:

While there’s much to discuss about the other new features and capabilities, we’ll leave the deep dive for a follow-on post. In the meantime, check out some of the new capabilities below.

New Feature – VMWare and Virtualbox Images

One of the most common pieces of feedback is a request to supply a pre-built image of Intrigue Core for ease of use getting started. By popular demand, we now support both VirtualBox and VMWare images, and you can find them in the Getting Started section.

More New Capabilities

Many of these new capabilities are worthy of a post on their own, and there’s simply so many, that all we can do is point out major highlights and trends, such as:

  • Integrations to search and pull metadata from Mobile App stores like iOS and Android
  • Unauthenticated and automated API Endpoint discovery
  • Content discovery on app endpoints using @joohoi’s Ffuf, link extraction and other entity-identification techniques
  • Integration of great open source tooling like Subfinder, and Naabu
  • Deeper integration with great services such as C99, BinaryEdge, Spyse, and Zetalytics (there are so many now!)
  • Authenticated Integrations to pull DNS Zones (aws_route53, cloudflare_dns, etc)

The full list of individual new tasks in v0.8.0 is as follows

  • AWS Route53
  • Cloudflare Zones
  • DNS Search TLS Cert Names
  • Naabu Scan
  • SaaS ServiceNow Check
  • SaaS ServiceNow Open KB Articles
  • Search 42matters API for Android/iOS apps
  • Search Apptweak API for Android/iOS apps
  • Search Azure Blob
  • Search BinaryEdge Open Databases
  • Search c99 Subdomainfinder
  • Search DnSimple
  • Search Farsight DNSDB
  • Search Hostio
  • Search Mnemonic
  • Search NeutrinoAPI
  • Search Recon.dev
  • Search Spyse
  • Search Spyse Cert
  • Search Spyse Domain
  • Search WhoisXMLAPI (Reverse Whois)
  • Subfinder
  • URI Brute Generic Content
  • URI Bruteforce Vhosts
  • URI Check API Endpoint
  • URI Check Retire.js
  • URI Extract Linked Hosts
  • URI Extract Tokens
  • URI Ffuf Content Discovery
  • WordPress Enumerate Leaked Logs

If you’d like to know more, you can find descriptions for each task here.

New Entities

To support all these great capabiltiies, you’ve gotta be able to represent the data types, and thus, the following new entities have been added since the last release. While an entity itself might not be exciting, the ability to open up new use cases brings fun challenges and you can expect even more entities in 2021.

  • AndroidApp – Android Mobile Application
  • ApiEndpoint – A HTTP based API endpoint
  • IosApp – IOS Mobile Application
  • MailServer – A Mailserver (MX)
  • UniqueKeyword – A globally unique keyword that can be reliably searched
  • UniqueToken – An api key or analytics id

New Vulnerability Checks

On some days this last year, it felt like literally every webapp and/or network appliance was under threat. 2020 did bring the “wow” CVEs, such as the F5 BigIP bug or RCEs in Sharepoint, Exchange, GlobalProtect…. yep… wow. The checks we now support are below, and the best thing is that these are all automatically driven by fingerprinting. If you find a GlobalProtect instance, and vulnerability checks are enabled for a project, it’ll automatically be tested. Attack surface enumeration should be easy – and ACCURATE – and these checks go a long way in making that a reality.

  • vuln/atlassian_dataexposure_cve_2020_14179
  • vuln/cisco_asa_limited_file_read_cve_2020_3452
  • vuln/cisco_asa_path_traversal_cve_2018_0296
  • vuln/citrix_netscaler_codeinjection_cve_2020_8194
  • vuln/craft_cms_seomatic_cve_2020_9757
  • vuln/f5_bigip_configuration_utility_cve_2020_5902
  • vuln/hadoop_yarn_unathenticated_resourcemanager
  • vuln/icewarp_xss_cve_2020_8512
  • vuln/microsoft_exchange_cve_2020_0688
  • vuln/microsoft_exchange_cve_2020_16875
  • vuln/microsoft_sharepoint_cve_2020_16952
  • vuln/mobileiron_multiple_cves
  • vuln/nextjs_path_traversal_cve_2020_5284
  • vuln/paloalto_globalprotect_check_cve_2020_2021
  • vuln/saas_gitlab_open_reg_check
  • vuln/solarwinds_orion_code_compromise
  • vuln/sonatype_nexus_cve_2020_10204
  • vuln/sonicwall_cve_2020_5135
  • vuln/telerik_crypto_weakness_cve_2017_9248
  • vuln/tomcat_ghostcat_cve_2020_1938
  • vuln/tomcat_persistent_manager_cve_2020_9484
  • vuln/wordpress_file_manager_command_injection_rce
  • vuln/wordpress_loginizer_cve_2020_27615

New Threat Checks

While threat discovery and enrichment is still a nascent use case for the engine, this release brings more goods (thank you Anas!), with even more direct integrations of high quality threat feeds to verify if a given IoC (entity) was found in their database, and where possible – the reason why. Expect this use case will continue to steadily improve in the new year.

  • threat/search_apility
  • threat/search_badips
  • threat/search_blcheck_list
  • threat/search_blocklistde
  • threat/search_dshield
  • threat/search_emerging_threats
  • threat/search_fraudguard
  • threat/search_greynoise
  • threat/search_ibm_x_force
  • threat/search_ipqs
  • threat/search_ipqs_emailaddress
  • threat/search_pulsedive
  • threat/search_talos_blacklist

BUGFIXES 

Luckily we had no bugs in the last release, so this one will continue that illustrious tradition of perfect and bug-free software. (Just kidding, there were simply way too many to mention. You know how to find them.) Security fixes, feature fixes, and all around improving the user experience were a big focus.

THANK YOU 

No core release to date has been simple, and this one has been well over a year in the making. It would not have been possible without the following fine folks, and so a thank you is well deserved:

  • First and foremost, Thank you to Intrigue.io customers, for your support and ideas that make this open source project grow!
  • Thank you to my wonderful wife Jessica and to all of the contributors’ families for supporting the significant time this project requires
  • Thank you @shpendk, for joining us as the first full-time contributor, and for tackling the ugliest challenges of the codebase
  • Thank you @bensalah_anas for consistently driving powerful new cases and capabilities in the platform
  • Thank you Maxim Gofnung, for digging right into the guts of the code with huge enthusiasm
  • Thank you @joeuser47 for the friendly and helpful support in our slack channel
  • Thank you to the folks building powerful open source tooling, particularly @errbysam, @joohoi and @pdiscoveryio
  • Thank you to the teams building innovative APIs, including Zetalytics, Spyse, SecurityTrails, BinaryEdge, Greynoise, Recon.dev and so many more
  • Thank you to the researchers who regularly share techniques and ideas … @th3g3nt3lman, @nahamsec, and so many others
  • Thank you @ebellis and @kennasecurity for your incredible long-term support of this project
  • Finally, thank you to the many open source users and contributors who have provided feedback, support, ideas.

So with that …. and the piece of mind that 2021 is looking up – bringing even more capabilities and velocity to this project, go and get started now! Try it out and send feedback via Email, Slack, or Twitter. Have fun, and keep us posted with any and all feedback!

-jcran