DomainRankingList

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

DomainRankingList

★ 8 Open GitHub ↗

Domain ranking lists based on the Cisco Umbrella Top 1M, cleaned up and processed for blue team use.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Domain Ranking List for Blue Teams

## Domain Rankings and SOC Use Cases

Enriching your security events with external data is a great way to weed out false positives and prioritize investigations. Domain rankings – lists of domains by popularity - have long been a popular method of checking the validity or shiftiness of domains and IP addresses. Unfortunately, we rarely examine the utility or validity of these lists before jamming them into our tools or processes. Depending on your specific use case, there is a lot of noise that you might not want to accept as “good” regardless of which domain ranking list you use.

We thought it would be worthwhile to analyze one of the more popular free domain ranking lists - the Cisco Umbrella Top 1M - and see if we could develop a methodology for making a revised list that is more relevant and actionable for defensive use cases. We also wanted to spend some time looking at the more unusual data that shows up in Internet-wide DNS requests to help blue teamers understand some of what they can expect (or not expect) to see in everyday traffic.

## Down the Rabbit Hole We Go

Much like DNS itself, domain research tends to be recursive. As noted by Scheitle, Jelten, et. al.[i], the Umbrella list is based on DNS queries only and includes queries not generated by user browsing. This also means that it tracks requests for any Internet service, not just web sites. The Umbrella list experiences a large amount of volatility day over day and month over month.

The long-term volatility is to be expected, but one day volatility of 20% and the turn-over of close to 1/3 of the list every three days is substantial. Beyond three months, the list overlap settles into a core set of entries that comprise ~40% of the list.

Because the Umbrella list is not limited to browser queries, it is subject to both typos, device mis-configurations, and plenty of other bad DNS requests. This makes for lots of weirdness showing up in the list – even at the top.

## Analyzing the List

The mix of FQDNs and domain names does not make for a very useful “top 1 million most popular domains” list (Source: Umbrella Blog). We want to reduce the list to just domains, while still retaining some measure of the FQDNs present on the list for each domain. Our goal is to create a derivative list that features the following values:

- Highest rank for an entry associated with the domain (e.g. 1)
 - Domain name (e.g. google.com)
 - Count of occurrences in the original Umbrella list (e.g. 1726 *.google.com occurrences)
 - Highest ranking entry for that domain in the list (e.g. google.com – not www.google.com)
 - Associated TLD, gTLD, or reserved ccXLD (top, second, third, or Nth level domain extension) (e.g. com)

Upon creation, our new condensed list only contains 321,855 entries for 4/21/20, a reduction of more than two-thirds after removing invalid, non-routable, and vendor/device domains.

## Files In This Repo

We encourage you to explore both the original Cisco Umbrella Top 1M list, as well as our revised and contextualized Bionic Umbrella list available here. The files available include:
 
 - the original Umbrella list
 - invalid (non-useful) entries in the original list
 - Bionic’s revised and contextualized Umbrella list
 - The public domain suffix list used for FQDN parsing

DomainRankingList

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README