KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agent #developer-tools #codex #typescript #cli #deepseek-harness #claude-code #ai-agents #mcp #rust

metahub

★ 177 Open GitHub ↗

MetaHub is an automated contextual security findings enrichment and impact evaluation tool for vulnerability management.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

MetaHub is an automated contextual security findings enrichment and impact evaluation tool for vulnerability management. You can use it with AWS Security Hub or any ASFF-compatible security scanner. Stop relying on useless severities and switch to impact scoring definitions based on YOUR context.

# Table of Contents

- [Description](#description)
- [Quick Run](#quick-run)
- [Context](#context)
- [Impact](#impact)
- [High Level Architecture](#high-level-architecture)
- [Use Cases](#use-cases)
- [Configuration](#customizing-configuration)
- [Run with Python](#run-with-python)
- [Run with Docker](#run-with-docker)
- [Run with Lambda](#run-with-lambda)
- [Run with Security Hub Custom Action](#run-with-security-hub-custom-action)
- [AWS Authentication](#aws-authentication)
- [Configuring Security Hub](#configuring-security-hub)
- [Configuring Context](#configuring-context)
- [Inputs](#Inputs)
- [Output](#outputs)
- [Filters](#filters)
- [Security Hub Actions](#security-hub-actions)
- [Contributing](#contributing)

# Description

**MetaHub** is an open-source security tool for **impact-contextual vulnerability management**. It can automate the process of **contextualizing** security findings based on your environment and your needs, YOUR **context**, identifying **ownership**, and calculate an **impact scoring** based on it that you can use for defining prioritization (where should you start?) and automations like remediations, alerts or tickets. The tool is for AWS environments and you can use it with [AWS Security Hub](https://aws.amazon.com/security-hub) or any [ASFF](https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-findings-format.html) compatible scanners (like [Prowler](https://github.com/prowler-cloud/prowler)).

> :information_source: Explore and extend the new [MetaHub Dashboards using Powerpipe](#use-it-with-powerpipe)!

**MetaHub** describes your [**context**](#context) by connecting to your affected resources in your affected accounts. It can describe information about your AWS account and organization, the affected resources tags, related CloudTrail events, your affected resource configurations **and all their associations**: if you are contextualizing a security finding affecting an EC2 Instance, **MetaHub** will not only connect to that instance itself but also its IAM Roles; from there, it will connect to the IAM Policies associated with those roles. It will connect to the Security Groups and analyze all their rules, the VPC and the Subnets where the instance is running, the Volumes, the Auto Scaling Groups, and more. You can apply [**filters**](#filters) to automate detecting other resources with the same properties and do in-depth investigations.

After fetching all the information from your **context**, **MetaHub** will evaluate the [**impact**](#impact) conditions for all your resources: [**exposure**](#exposure), [**access**](#access), [**encryption**](#encryption), [**status**](#status), [**environment**](#environment), [**application**](#application), and [**owner**](#owner) and based on those calculations and in addition to the information about the security [**findings**](#findings) affecting the resource all **together**, **MetaHub** will generate a **score** for each finding and affected resource.

Check the following dashboard generated by **MetaHub**. You have the affected resources, grouping all the security findings affecting them together and the original severity of each finding. After that, you have the **score** and all the criteria **MetaHub** evaluated to generate that **score**. All this information is filterable, sortable, groupable, downloadable, and customizable.

The following is the JSON [output](#outputs) for an EC2 instance; see how **MetaHub** organizes all the information about its **context** together, under [**associations**](#associations), [**config**](#config), [**tags**](#tags), [**account**](#account) and [**cloudtrail**](#cloudtrail), and finally the [**impact**](#impact) key with the **score** and all the criteria evaluated to generate that **score**.

**MetaHub** provides a range of ways to list, manage, and output your security findings for investigation, suppression, updating, and integration with other tools. It is designed for use as a [CLI](#run-with-python) tool or within automated workflows, such as [AWS Lambda functions](#run-with-lambda). It supports different **[outputs](#outputs)**, some of them [**programatic json**](#json), but also powerful [**html**](#html), [**xlsx**](#xlsx) and [**csv**](#csv) that you can [customize](#customize-html-csv-or-xlsx-outputs).

If you use **AWS Security Hub**, MetaHub integrates smoothly and extends its functionalities. It can be used as a **[Security Hub Custom Action](#run-with-security-hub-custom-action)**, it supports **[AWS Security Hub filters](security-hub-filtering)**, you can manage the **[workflow status of your findings](#updating-workflow-status)**, and you can even **[enrich your findings directly in AWS Security Hub](#enriching-findings)**.

**MetaHub** is designed to be used with AWS and supports **multi-account setups**. You can run the tool from any environment by assuming roles in your AWS Security Hub `master` and your `child/service` accounts. This allows you to fetch aggregated data from multiple accounts using your AWS Security Hub multi-account implementation while also fetching and enriching those findings with data from the accounts where your affected resources are running. Refer to [Configuring Security Hub](#configuring-security-hub) for more information.

# Quick Run

Read your security findings from AWS Security Hub with the default filters and executes the default context options:

```bash
./metahub
```

Read a specific (filtered by Id) security finding from AWS Security Hub and executes the default context options:

```bash
./metahub --sh-filters Id=arn:aws:securityhub:us-east-1:123456789012:security-control/CloudFront.1/finding/8bd4d049-dcbc-445b-a5d1-595d8274b4c1
```

Read all the security findings affecting one resource which are ACTIVE (filtered by ResourceId and RecordState) from AWS Security Hub and executes the default context options:

```bash
./metahub --sh-filters RecordState=ACTIVE ResourceId=arn:aws:ec2:eu-west-1:123456789012:subnet/subnet-0b7d243ff90ebc03e
```

Read all the security findings affecting an AWS Account which are ACTIVE (filtered by AwsAccountId and RecordState) for resources with a tag `Environment` and the value `stg` and executes the context options `config` and `tags`:

```bash
./metahub --sh-filters RecordState=ACTIVE AwsAccountId=123456789012 --mh-filters-tags Environment=stg --context config tags
```

# Quick Run (Reading findings from a input ASFF file)

Read your security findings from Prowler as an input file and executes the default context options:

```bash
python3 prowler.py aws -M json-asff -q
./metahub --inputs file-asff --input-asff /path/to/prowler-findings.json.asff
```

Read your security findings from Powerpipe as an input file and executes the default context options:

```bash
powerpipe benchmark run aws_compliance.benchmark.all_controls --export asff
./metahub --inputs file-asff --input-asff /path/to/powerpipe-findings.json.asff
```

Read your security findings from Trivy as an input file and executes the default context options:

```bash
export AWS_REGION=us-west-1
export AWS_ACCOUNT_ID=012345678901
trivy image --format template --template "@contrib/asff.tpl" -o trivy-findings.json.asff public.ecr.aws/n2p8q5p4/metahub:stable
./metahub --inputs file-asff --input-asff /path/to/trivy-findings.json.asff
```

# Context

In **MetaHub**, **context** refers to information about the affected resources like their **configuration**, **associations**, **logs**, **tags** and **account**.

MetaHub doesn't stop at the affected resource but also analyzes any associated or attached resources. For instance, if a security finding exists on a Security Group, **MetaHub** will analyze the Security Group and everything else associated with it, like the EC2 instances using it. For each associated resource, **MetaHub** will fetch its context. If the Security Group is attached to an EC2 Instance, **MetaHub** will analyze the instance and all its associations, like the IAM Roles and policies. From a single security finding on a Security Group, **MetaHub** will fetch the context of the Security Group, the EC2 Instance, the IAM Roles, and the IAM Policies. This is critical for understanding the impact of your security findings.

The **context** module has five main parts: [**config**](#config) (which includes [**associations**](#associations)), [**tags**](#tags), [**cloudtrail**](#cloudtrail), and [**account**](#account). By default only **config**, **tags** and **account** are enabled, but you can change this behavior using the option `--context` (e.g. use `--context config tags cloudtrail account` for enabling all context keys, or `--context config` for enabling only the config and associations key.):

## Config

Under the `config` key, you can find important configuration from the affected resource. For example, if the affected resource is an S3 Bucket, you will find information about its bucket policy, its ACLs, its encryption configuration, and more. If the affected resource is an EC2 Instance, you will find information about its key, its public and private IP, its metadata and more. The configuration information that **MetaHub** fetches is defined by resource type. If you want to add more configuration information see [contributing](#contributing).

You can filter your findings based on config outputs using the option: `--mh-filters-config <key> {True/False}` (see [config filters](#config-filters)).

<details>
 <summary>Example for an S3 bucket config key</summary>

```json
"config": {
  "resource_policy": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "Test",
        "Effect": "Allow",
        "Principal": {
          "Service": "config.amazonaws.com"
        },
        "Action": "s3:GetBucketAcl",
        "Resource": "arn:aws:s3:::metahub-bucket",
        "Condition": {
          "StringEquals": {
            "AWS:SourceAccount": "123456789012"
          }
        }
      },
    ]
  },
  "website_enabled": false,
  "bucket_acl": [
    {
      "Grantee": {
        "DisplayName": "gabriel.soltz",
        "ID": "1234564bd76c6c64080717b68eafaa588b41706daaf22d3d0705b398bd7cbd57",
        "Type": "CanonicalUser"
      },
      "Permission": "FULL_CONTROL"
    }
  ],
  "cannonical_user_id": "1234564bd76c6c64080717b68eafaa588b41706daaf22d3d0705b398bd7cbd57",
  "public_access_block_enabled": {
    "BlockPublicAcls": true,
    "IgnorePublicAcls": true,
    "BlockPublicPolicy": true,
    "RestrictPublicBuckets": true
  },
  "account_public_access_block_enabled": false,
  "public": false,
  "bucket_encryption": [
    {
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "AES256"
      },
      "BucketKeyEnabled": false
    }
  ]
},
```

</details>

## Associations

Under the `associations` key, you will find all the associated resources of the affected resource. For example, if the affected resource is an EC2 Instance, you will find resources like: Security Groups, IAM Roles, Volumes, VPC, Subnets, Auto Scaling Groups. If the affected resource is a IAM Role, you will find resources like IAM Policies, IAM Users, IAM Groups, and more. Each time MetaHub finds an association, it will connect to the associated resource again and fetch its own context. Associations are key to understanding the context and impact of your security findings.

You can filter your findings based on associations outputs using the option: `--mh-filters-config <key> {True/False}` (see [config filters](#config-filters)).

<details>
 <summary>Example for an EC2 instance assocations key</summary>

```json
"associations": {
  "security_groups": {
    "arn:aws:ec2:eu-west-1:123456789012:security-group/sg-020cc749a58678e05": {
      "associations": {
        "vpcs": {
          "arn:aws:ec2:eu-west-1:123456789012:vpc/vpc-03cc56a1c2afb5760": {
            "associations": {
              "subnets": {
                "arn:aws:ec2:eu-west-1:123456789012:subnet/subnet-03d86f1ccd7729d85": {},
                "arn:aws:ec2:eu-west-1:123456789012:subnet/subnet-0ccfb8dea658f49ec": {},
                "arn:aws:ec2:eu-west-1:123456789012:subnet/subnet-05e85a7b0ec9e404c": {},
                "arn:aws:ec2:eu-west-1:123456789012:subnet/subnet-0e177ea95bcc76256": {}
              }
            },
            "config": {
              "cidr": "172.10.0.0/16",
              "default": false,
              "public": null
            }
          }
        },
        "network_interfaces": {
          "arn:aws:ec2:eu-west-1:123456789012:network-interface/eni-041a6e5bb59c336ee": {}
        },
        "instances": {
          "arn:aws:ec2:eu-west-1:123456789012:instance/i-018daeedcf06398c0": {}
        }
      },
      "config": {
        "public_ips": [
          "100.100.100.100"
        ],
        "managed_services": [],
        "its_referenced_by_a_security_group": false,
        "security_group_rules": [
          {
            "SecurityGroupRuleId": "sgr-08cdc9fdac8fd1a5b",
            "GroupId": "sg-020cc749a58678e05",
            "GroupOwnerId": "123456789012",
            "IsEgress": true,
            "IpProtocol": "-1",
            "FromPort": -1,
            "ToPort": -1,
            "CidrIpv4": "0.0.0.0/0",
            "Tags": []
          },
          {
            "SecurityGroupRuleId": "sgr-0e6cd39169dc137ab",
            "GroupId": "sg-020cc749a58678e05",
            "GroupOwnerId": "123456789012",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 22,
            "ToPort": 22,
            "CidrIpv4": "0.0.0.0/0",
            "Tags": []
          }
        ],
        "public": true,
        "default": false,
        "attached": true,
        "resource_policy": null
      }
    },
  },
  "iam_roles": {
    "arn:aws:iam::123456789012:role/eu-west-1-stg-backend-iam-role": {
      "associations": {
        "iam_policies": {
          "arn:aws:iam::123456789012:policy/eu-west-1-stg-backend-iam-policy-cw": {
            "associations": {
              "iam_roles": {
                "arn:aws:iam::123456789012:role/eu-west-1-stg-backend-iam-role": {}
              },
              "iam_groups": {},
              "iam_users": {}
            },
            "config": {
              "name": "eu-west-1-stg-backend-iam-policy-cw",
              "description": false,
              "customer_managed": true,
              "attached": true,
              "public": null,
              "resource_policy": {
                "Version": "2012-10-17",
                "Statement": [
                  {
                    "Effect": "Allow",
                    "Action": [
                      "logs:CreateLogGroup",
                      "logs:CreateLogStream",
                      "logs:PutLogEvents",
                      "logs:DescribeLogStreams"
                    ],
                    "Resource": [
                      "arn:aws:logs:*:*:*"
                    ]
                  }
                ]
              }
            }
          },
        }
      },
      "config": {
        "iam_inline_policies": {},
        "instance_profile": "arn:aws:iam::123456789012:instance-profile/eu-west-1-stg-backend-iam-profile",
        "trust_policy": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "",
              "Effect": "Allow",
              "Principal": {
                "Service": "ec2.amazonaws.com"
              },
              "Action": "sts:AssumeRole"
            }
          ]
        },
        "permissions_boundary": false,
        "public": null,
        "resource_policy": null
      }
    }
  },
  "volumes": {
    "arn:aws:ec2:eu-west-1:123456789012:volume/vol-0371a09e338f582da": {
      "associations": {
        "instances": {
          "arn:aws:ec2:eu-west-1:123456789012:instance/i-018daeedcf06398c0": {}
        }
      },
      "config": {
        "encrypted": true,
        "attached": true,
        "public": null,
        "resource_policy": null
      }
    }
  },
  "autoscaling_groups": {
    "arn:aws:autoscaling:eu-west-1:123456789012:autoScalingGroup/stg-backend-20201205160228428400000002": {
      "associations": {
        "instances": {
          "arn:aws:ec2:eu-west-1:123456789012:instance/i-018daeedcf06398c0": {}
        },
        "launch_templates": {
          "arn:aws:ec2:eu-west-1:123456789012:launch-template/lt-06b73d2e77f10446f": {}
        },
        "launch_configurations": {}
      },
      "config": {
        "name": "stg-backend-20201205160228428400000002",
        "public": null,
        "resource_policy": null
      }
    }
  },
  "vpcs": {
    "arn:aws:ec2:eu-west-1:123456789012:vpc/vpc-03cc56a1c2afb5760": {
      "associations": {
        "subnets": {
          "arn:aws:ec2:eu-west-1:123456789012:subnet/subnet-03d86f1ccd7729d85": {},
          "arn:aws:ec2:eu-west-1:123456789012:subnet/subnet-0ccfb8dea658f49ec": {},
          "arn:aws:ec2:eu-west-1:123456789012:subnet/subnet-05e85a7b0ec9e404c": {},
          "arn:aws:ec2:eu-west-1:123456789012:subnet/subnet-0e177ea95bcc76256": {}
        }
      },
      "config": {
        "cidr": "172.10.0.0/16",
        "default": false,
        "public": null
      }
    }
  },
  "subnets": {
    "arn:aws:ec2:eu-west-1:123456789012:subnet/subnet-03d86f1ccd7729d85": {
      "associations": {
        "route_tables": {
          "arn:aws:ec2:eu-west-1:123456789012:route-table/rtb-0ebae6462f919943d": {
            "associations": {},
            "config": {
              "default": false,
              "route_to_internet_gateway": [
                {
                  "DestinationCidrBlock": "0.0.0.0/0",
                  "GatewayId": "igw-0790540d8d726f9d4",
                  "Origin": "CreateRoute",
                  "State": "active"
                }
              ],
              "route_to_nat_gateway": [],
              "route_to_transit_gateway": [],
              "route_to_vpc_peering": [],
              "public": null
            }
          }
        },
        "network_interfaces": {
          "arn:aws:ec2:eu-west-1:123456789012:network-interface/eni-0e8918fa31d2acd55": {},
          "arn:aws:ec2:eu-west-1:123456789012:network-interface/eni-0f6c936934fd9d6a6": {},
          "arn:aws:ec2:eu-west-1:123456789012:network-interface/eni-041a6e5bb59c336ee": {},
          "arn:aws:ec2:eu-west-1:123456789012:network-interface/eni-0e5f6cfdc7c286224": {}
        },
        "instances": {
          "arn:aws:ec2:eu-west-1:123456789012:instance/i-018daeedcf06398c0": {}
        }
      },
      "config": {
        "cidr": "172.11.11.0/24",
        "map_public_ip_on_launch_enabled": true,
        "default": false,
        "public": true,
        "resource_policy": null,
        "public_ips": [
          "100.100.100.100",
        ],
        "managed_services": [
          "ELB app/stg-alb-backend/3567d780bd062d75",
          "Interface for NAT Gateway nat-0805d9808347bba69"
        ],
        "attached": true
      }
    }
  },
}
```

</details>

## Tags

Under the `tags` key, you will find all the tags associated with the affected resource. **MetaHub** relies on [AWS Resource Groups Tagging API](https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/overview.html) to query the tags associated with your resources. Note that not all AWS resource type supports this API. You can check [supported services](https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/supported-services.html).

Tags are a crucial part of understanding your context. Tagging strategies often include:

- Environment (like Production, Staging, Development, etc.)
- Data classification (like Confidential, Restricted, etc.)
- Owner (like a team, a squad, a business unit, etc.)
- Compliance (like PCI, SOX, etc.)

If you follow a proper tagging strategy, you can filter and generate interesting outputs. For example, you could list all findings related to a specific team and provide that data directly to that team.

You can filter your findings based on tags outputs using the option: `--mh-filters-tags TAG=VALUE` (see [tags filters](#tags-filters)).

<details>
 <summary>Example for an EC2 instance tag key</summary>

```json
"tags": {
  "aws:autoscaling:groupName": "stg-backend-20201205160228428400000002",
  "environment": "stg",
  "terraform": "true",
  "aws:ec2launchtemplate:version": "8",
  "aws:ec2launchtemplate:id": "lt-06b73d2e77f10446f",
  "Name": "stg-backend"
}
```

</details>

## CloudTrail

Under the key `cloudtrail`, you will find critical Cloudtrail events related to the affected resource, such as creation events. The Cloudtrail events we look for are defined by resource type, and you can add, remove, or change them by editing the configuration file [resources.py](lib/config/resources.py). For example, for an affected resource of type Security Group, MetaHub will look for the following events: `CreateSecurityGroup` (Security Group Creation event) and `AuthorizeSecurityGroupIngress` (Security Group Rule Authorization event).

<details>
 <summary>Example for an EC2 instance cloudtrail key</summary>

```json
"cloudtrail": {
  "RunInstances": {
    "Username": "root",
    "EventTime": "2023-11-15 06:10:07+01:00",
    "EventId": "4f122d76-812d-4438-bc33-3585a9e863cf"
  }
}
```

</details>

<details>
 <summary>Example for a DynamoDB table cloudtrail key</summary>

```json
"cloudtrail": {
  "CreateTable": {
    "Username": "gabriel.soltz",
    "EventTime": "2023-12-05 14:34:25+01:00",
    "EventId": "7110e3ae-09a3-44b9-929a-1775e0fbedcf"
  }
}
```

</details>

## Account

Under the key `account`, you will find information about the account where the affected resource is runnning, like if it's part of an AWS Organizations, information about their contacts, and more.

</details>

<details>
 <summary>Example for account key</summary>

```json
"account": {
  "Alias": "metahub-demo",
  "AlternateContact": {
    "AlternateContactType": "SECURITY",
    "EmailAddress": "gabriel@domain.com",
    "Name": "Gabriel",
    "PhoneNumber": "+1234567890",
    "Title": "Security"
  },
  "Organizations": {
    "Arn": "arn:aws:organizations::123456789012:organization/o-12349772jb",
    "Id": "o-12349772jb",
    "MasterAccountId": "123456789012",
    "MasterAccountEmail": "gabriel.soltz@domain.com",
    "FeatureSet": "ALL",
    "DelegatedAdministrators": {},
    "Details": {
      "ParentId": "r-k123",
      "ParentType": "ROOT",
      "OU": "ROOT",
      "Policies": {
        "p-FullAWSAccess": {
          "Name": "FullAWSAccess",
          "Arn": "arn:aws:organizations::aws:policy/service_control_policy/p-FullAWSAccess",
          "Type": "SERVICE_CONTROL_POLICY",
          "Description": "Allows access to every operation",
          "AwsManaged": true,
          "Targets": []
        }
      }
    }
  }
},
```

</details>

# Impact

The impact module in MetaHub focuses on understanding the 7 key properties of the affected resource: [**exposure**](#exposure), [**access**](#access), [**encryption**](#encryption), [**status**](#status), [**environment**](#environment), [**application**](#application), and [**owner**](#owner) and combining their values with the values of all the security [findings](#findings) affecting the same resource and their severities to generate a **score**. The impact score is a number between 0 and 100, where 100 is the highest impact.

## Exposure

**Exposure** evaluates how the the affected resource is exposed to other networks. For example, if the affected resource is effectively public or just public, if it is part of a VPC, if it has a public IP or if it is protected by a firewall or a security group.

| **Possible Statuses**   | **Value** | **Description**                                                                                                |
| ----------------------- | :-------: | -------------------------------------------------------------------------------------------------------------- |
| 🔴 effectively-public   |   100%    | The resource is effectively public from the Internet.                                                          |
| 🟠 restricted-public    |    40%    | The resource is public, but there is a restriction like a Security Group.                                      |
| 🟠 unrestricted-private |    30%    | The resource is private but unrestricted, like an open security group.                                         |
| 🟠 launch-public        |    10%    | These are resources that can launch other resources as public. For example, an Auto Scaling group or a Subnet. |
| 🟢 restricted           |    0%     | The resource is restricted.                                                                                    |
| 🔵 unknown              |     -     | The resource couldn't be checked                                                                               |

<details>
 <summary>Example for an effectively-public resource</summary>

```json
"exposure": {                             --> The exposure key
  "effectively-public": {                 --> The exposure value, effectively-public
    "entrypoint": "66.66.66.66",          --> The entrypoint to the resource from the Internet (Ip, Domain, etc.)
    "unrestricted_ingress_rules": [       --> The unrestricted ingress rules, if any
      {
        "SecurityGroupRuleId": "sgr-0553206714e321b87",
        "GroupId": "sg-0a15a46e47f07d139",
        "GroupOwnerId": "123456789012",
        "IsEgress": false,
        "IpProtocol": "tcp",
        "FromPort": 22,
        "ToPort": 22,
        "CidrIpv4": "0.0.0.0/0",
        "Tags": []
      }
    ],
    "unrestricted_egress_rules": [       --> The unrestricted egress rules, if any
      {
        "SecurityGroupRuleId": "sgr-007b509667896ebe3",
        "GroupId": "sg-0a15a46e47f07d139",
        "GroupOwnerId": "123456789012",
        "IsEgress": true,
        "IpProtocol": "-1",
        "FromPort": -1,
        "ToPort": -1,
        "CidrIpv4": "0.0.0.0/0",
        "Tags": []
      },
    ],
    "resource_public_config": true       --> The public configuration of the resource
  }
}
```

</details>

## Access

**Access** evaluates the resource policy layer. MetaHub checks every available policy including: IAM Managed policies, IAM Inline policies, Resource Policies, Bucket ACLS, and any association to other resources like IAM Roles which its policies are also analyzed . An unrestricted policy is not only an itsue itself of that policy, it afected any other resource which is using it.

| **Possible Statuses**      | **Value** | **Description**                                                                                                                              |
| -------------------------- | :-------: | -------------------------------------------------------------------------------------------------------------------------------------------- |
| 🔴 unrestricted            |   100%    | The principal is unrestricted, without any condition or restriction.                                                                         |
| 🔴 untrusted-principal     |    70%    | The principal is an AWS Account, not part of your trusted accounts.                                                                          |
| 🟠 unrestricted-principal  |    40%    | The principal is not restricted, defined with a wildcard. It could be conditions restricting it or other restrictions like s3 public blocks. |
| 🟠 cross-account-principal |    30%    | The principal is from another AWS account.                                                                                                   |
| 🟠 unrestricted-actions    |    30%    | The actions are defined using wildcards.                                                                                                     |
| 🟠 dangerous-actions       |    30%    | Some dangerous actions are defined as part of this policy.                                                                                   |
| 🟠 unrestricted-service    |    10%    | The policy allows an AWS service as principal without restriction.                                                                           |
| 🟢 restricted              |    0%     | The policy is restricted.                                                                                                                    |
| 🔵 unknown                 |     -     | The policy couldn't be checked.                                                                                                              |

<details>
 <summary>Example for an unrestricted-actions resource</summary>

```json
"access": {                                --> The access key
  "unrestricted-actions": {                --> The access value, unrestricted-actions
    "wildcard_actions": {                  --> The wildcard policies, if any
      "arn:aws:iam::123456789012:policy/eu-west-1-stg-iam-policy-dynamodb-cache": [
        {
          "Action": [
            "dynamodb:*"                   --> The wildcard action
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:dynamodb:eu-west-1:123456789012:table/table",
          ]
        }
      ],
  }
}
```

</details>

## Encryption

**Encryption** evaluate the different encryption layers based on each resource type. For example, for some resources it evaluates if `at_rest` and `in_transit` encryption configuration are both enabled.

| **Possible Statuses** | **Value** | **Description**                                                     |
| --------------------- | :-------: | ------------------------------------------------------------------- |
| 🔴 unencrypted        |   100%    | The resource is not fully encrypted.                                |
| 🟢 encrypted          |    0%     | The resource is fully encrypted including any of it's associations. |
| 🔵 unknown            |     -     | The resource encryption couldn't be checked.                        |

<details>
 <summary>Example for an unencrypted resource</summary>

```json
"encryption": {                                --> The encryption key
  "unencrypted": {                             --> The encryption value, unencrypted
    "unencrypted_resources": [                 --> the unencrypted resources associated with the affected resource, if any
      "arn:aws:ec2:eu-west-1:012345678901:volume/vol-0ac713ec808a8d8bd"
    ],
    "resource_encryption_config": null         --> The encryption configuration of the resource, if it has any
  }
}
```

</details>

## Status

**Status** evaluate the status of the affected resource in terms of attachment or functioning. For example, for an EC2 Instance we evaluate if the resource is running, stopped, or terminated, but for resources like EBS Volumes and Security Groups, we evaluate if those resources are attached to any other resource.

| **Possible Statuses** | **Value** | **Description**                                           |
| --------------------- | :-------: | --------------------------------------------------------- |
| 🟠 attached           |   100%    | The resource supports attachment and is attached.         |
| 🟠 running            |   100%    | The resource supports running and is running.             |
| 🟠 enabled            |   100%    | The resource supports enabled and is enabled.             |
| 🟢 not-attached       |    0%     | The resource supports attachment, and it is not attached. |
| 🟢 not-running        |    0%     | The resource supports running and it is not running.      |
| 🟢 not-enabled        |    0%     | The resource supports enabled and it is not enabled.      |
| 🔵 unknown            |     -     | The resource couldn't be checked for status.              |

<details>
 <summary>Example for a running resource</summary>

```json
"status": {                             --> The status key
  "running": {                          --> The status value, running
    "status": "running",                --> The status configuration of the resource, if it has any
    "attached": null                    --> The attachment configuration of the resource, if it has any
  }
}
```

</details>

## Environment

**Environment** evaluates the environment where the affected resource is running. By default, MetaHub defines 3 environments: `production`, `staging`, and `development`, but you can add, remove, or modify these environments based on your needs. MetaHub evaluates the environment based on the tags of the affected resource, the account id or the account alias. You can define your own environemnts definitions and strategy in the configuration file (See [Customizing Configuration](#customizing-configuration)).

| **Possible Statuses** | **Value** | **Description**                                  |
| --------------------- | :-------: | ------------------------------------------------ |
| 🟠 production         |   100%    | It is a production resource.                     |
| 🟢 staging            |    30%    | It is a staging resource.                        |
| 🟢 development        |    0%     | It is a development resource.                    |
| 🔵 unknown            |     -     | The resource couldn't be checked for enviroment. |

<details>
 <summary>Example for a production resource matched by Tags</summary>

```json
"environment": {                          --> The environment key
  "production": {                         --> The environment value, production
    "tags": {                             --> The parameters used for evaluating the environment, in this case tags
      "Env": "prod"                       --> The tag and key found used for evaluating the environment
    }
  }
}
```

</details>

## Application

**Application** evaluates the application that the affected resource is part of. MetaHub relies on the AWS [myApplications](https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/aws-myApplications.html) feature, which relies on the Tag `awsApplication`, but you can extend this functionality based on your context for example by defining other tags you use for defining applications or services (like `Service` or any other), or by relying on account id or alias. You can define your application definitions and strategy in the configuration file (See [Customizing Configuration](#customizing-configuration)).

| **Possible Statuses** | **Value** | **Description**                                   |
| --------------------- | :-------: | ------------------------------------------------- |
| 🔵 unknown            |     -     | The resource couldn't be checked for application. |

<details>
 <summary>Example for a resource matched by myApplication Tag</summary>

```json
"application": {                          --> The application key
  "payments-app": {                       --> The application value, payments-app
    "tags": {                             --> The parameters used for evaluating the environment, in this case the tag awsApplication
      "awsApplication": "arn:aws:resource-groups:eu-west-1:123456789012:group/app1/0c8vpbjkzeeffsz2cqgxpae7b2"   --> The tag and key found used for evaluating the environment
    }
  }
}
```

</details>

## Owner

**Owner** focuses on ownership detection. It can determine the owner of the affected resource in various ways. This information can be used to automatically assign a security finding to the correct owner, escalate it, or make decisions based on this information. An automated way to determine the owner of a resource is critical for security teams. It allows them to focus on the most critical issues and assign them as fast as possible to the right people in automated workflows. You can define your owner definitions and strategy in the configuration file (See [Customizing Configuration](#customizing-configuration)).

| **Possible Statuses** | **Value** | **Description**                             |
| --------------------- | :-------: | ------------------------------------------- |
| 🔵 unknown            |     -     | The resource couldn't be checked for owner. |

<details>
 <summary>Example for a resource matched by Account Id</summary>

```json
"owner": {                                --> The owner key
  "payments-team": {                      --> The owner value, payments-app
    "account": {                          --> The parameters used for evaluating the environment, in this case the account
        "account_ids": ["123456789012"],  --> The account ids found used for evaluating the environment
        "account_aliases": [],
    },
  }
}
```

</details>

## Findings

As part of the impact score calculation, we also evaluate the total ammount of security findings and their severities affecting the resource. We use the following formula to calculate this metric:

```sh
(SUM of all (Finding Severity / Highest Severity) with a maximum of 1)
```

For example, if the affected resource has two findings affecting it, one with `HIGH` and another with `LOW` severity, the **Impact Findings Score** will be:

```sh
SUM(HIGH (3) / CRITICAL (4) + LOW (0.5) / CRITICAL (4)) = 0.875
```

# High level architecture

**MetaHub** reads your security findings from AWS Security Hub or any ASFF-compatible security scanner. It then queries the affected resources directly in the affected account to provide additional context. Based on that context, it calculates it's impact. Finally, it generates different outputs based on your needs.

# Use Cases

Some use cases for MetaHub include:

- [MetaHub integration with Prowler as a local scanner for context enrichment](https://medium.com/@gabriel_87/metahub-use-cases-part-i-metahub-integration-with-prowler-as-a-local-scanner-for-context-f3540e18eaa1)
- Automating Security Hub findings suppression based on Tagging
- Integrate MetaHub directly as Security Hub custom action to use it directly from the AWS Console
- Created enriched HTML reports for your findings that you can filter, sort, group, and download
- Create Security Hub Insights based on MetaHub context

# Customizing Configuration

**MetaHub** uses configuration files that let you customize some checks behaviors, default filters, and more. The configuration files are located in [lib/config/](lib/config).

Things you can customize:

- [lib/config/configuration.py](lib/config/configuration.py): This file contains the default configuration for MetaHub. You can change the default filters, the default output modes, the environment definitions, and more.

- [lib/config/impact.py](lib/config/impact.py): This file contains the values and it's weights for the impact formula criteria. You can modify the values and the weights based on your needs.

- [lib/config/reources.py](lib/config/resources.py): This file contains definitions for every resource type, like which CloudTrail events to look for.

# Run with Python

**MetaHub** is a Python3 program. You need to have Python3 installed in your system and the required Python modules described in the file `requirements.txt`.

Requirements can be installed in your system manually (using pip3) or using a Python virtual environment (suggested method).

## Run it using Python Virtual Environment

1. Clone the repository: `git clone git@github.com:gabrielsoltz/metahub.git`
2. Change to repostiory dir: `cd metahub`
3. Create a virtual environment for this project: `python3 -m venv venv/metahub`
4. Activate the virtual environment you just created: `source venv/metahub/bin/activate`
5. Install Metahub requirements: `pip3 install -r requirements.txt`
6. Run: `./metahub -h`
7. Deactivate your virtual environment after you finish with: `deactivate`

Next time, you only need steps 4 and 6 to use the program.

Alternatively, you can run this tool using Docker.

# Run with Docker

MetaHub is also available as a Docker image. You can run it directly from the public Docker image or build it locally.

The available tagging for MetaHub containers are the following:

- `latest`: in sync with master branch
- `<x.y.z>`: you can find the releases [here](https://github.com/gabrielsoltz/metahub/releases)
- `stable`: this tag always points to the latest release.

For running from the public registry, you can run the following command:

```sh
docker run -ti public.ecr.aws/n2p8q5p4/metahub:latest ./metahub -h
```

## AWS credentials and Docker

If you are already logged into the AWS host machine, you can seamlessly use the same credentials within a Docker container. You can achieve this by either passing the necessary environment variables to the container or by mounting the credentials file.

For instance, you can run the following command:

```sh
docker run -e AWS_DEFAULT_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN -ti public.ecr.aws/n2p8q5p4/metahub:latest ./metahub -h
```

On the other hand, if you are not logged in on the host machine, you will need to log in again from within the container itself.

## Build and Run Docker locally

Or you can also build it locally:

```sh
git clone git@github.com:gabrielsoltz/metahub.git
cd metahub
docker build -t metahub .
docker run -ti metahub ./metahub -h
```

# Run with Lambda

**MetaHub** is Lambda/Serverless ready! You can run MetaHub directly on an AWS Lambda function without any additional infrastructure required.

Running MetaHub in a Lambda function allows you to automate its execution based on your defined triggers. For example, you can:

- Trigger the MetaHub Lambda function each time there is a new security finding to enrich that finding back in AWS Security Hub.
- Trigger the MetaHub Lambda function each time there is a new security finding for suppression based on Context.
- Trigger the MetaHub Lambda function to identify the affected owner of a security finding based on Context and assign it using your internal systems.
- Trigger the MetaHub Lambda function to create a ticket with enriched context.

## Deploying Lambda

The terraform code for deploying the Lambda function is provided under the `terraform/` folder.

Just run the following commands:

```sh
cd terraform
terraform init
terraform apply
```

The code will create a zip file for the Lambda code and a zip file for the Python dependencies that we will use as Lambda layer. It will also create the Lambda function and all the required resources.

The Terraform code will also create a Security Hub custom action and an EventBridge rule to trigger the Lambda function when the custom action is executed. See below.

## Customize Lambda behaviour

You can customize MetaHub options for your lambda by editing the file [lib/lambda.py](lib/lambda.py). You can change the default options for MetaHub, such as the filters, the Meta\* options, and more.

## Lambda Permissions

Terraform will create the minimum required permissions for the Lambda function to run locally (in the same account). If you want your Lambda to assume a role in other accounts (for example, you will need this if you are executing the Lambda in the Security Hub master account that is aggregating findings from other accounts), you will need to specify the role to assume, adding the option `--mh-assume-role` in the Lambda function configuration (See previous step) and adding the corresponding policy to allow the Lambda to assume that role in the lambda role.

# Run with Security Hub Custom Action

**MetaHub** can be run as a [Security Hub Custom Action](https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-cwe-custom-actions.html). This allows you to run MetaHub directly from the Security Hub console for a selected finding or for a selected set of findings.

The custom action will then trigger a Lambda function that will run MetaHub for the selected findings.

When you trigger the Lambda using the Security Hub Custom Action, the lambda will read the selected findings for it's context, and it will execute once for each finding. By default, no action will be taken on the findings, but you can change this behavior. See [Customize Lambda behavior](#customize-lambda-behaviour).

The Security Hub custom action is deployed as part of the Terraform code. See [Deploying Lambda](#deploying-lambda) for more information.

<details>
 <summary>If you want to deploy it manually, you can follow the steps below:</summary>

1. In Security Hub, choose Settings and then choose Custom Actions.
2. Choose Create custom action.
3. Provide a Name, Description, and Custom action ID for the action.
4. Choose Create custom action. (Make a note of the Custom action ARN. You need to use the ARN when you create a rule to associate with this action in EventBridge.)
5. In EventBridge, choose Rules and then choose Create rule.
6. Enter a name and description for the rule.
7. For the Event bus, choose the event bus that you want to associate with this rule. If you want this rule to match events that come from your account, select default. When an AWS service in your account emits an event, it always goes to your account's default event bus.
8. For Rule type, choose a rule with an event pattern and then press Next.
9. For Event source, choose AWS events.
10. For the Creation method, choose Use pattern form.
11. For Event source, choose AWS services.
12. For AWS service, choose Security Hub.
13. For Event type, choose Security Hub Findings - Custom Action.
14. Choose Specific custom action ARNs and add a custom action ARN.
15. Choose Next.
16. Under Select targets, choose the Lambda function
17. Select the Lambda function you created for MetaHub.
</details>

# AWS Authentication

Ensure you have AWS credentials set up on your local machine (or from where you will run MetaHub).

For example, you can use `aws configure` option.

```sh
aws configure
```

Or you can export your credentials to the environment.

```sh
export AWS_DEFAULT_REGION="us-east-1"
export AWS_ACCESS_KEY_ID= "ASXXXXXXX"
export AWS_SECRET_ACCESS_KEY= "XXXXXXXXX"
export AWS_SESSION_TOKEN= "XXXXXXXXX"
```

# Configuring Security Hub

- If you are running MetaHub for a single AWS account setup (AWS Security Hub is not aggregating findings from different accounts), you don't need to use any additional options; MetaHub will use the credentials in your environment. Still, if your IAM design requires it, it is possible to log in and assume a role in the same account you are logged in. Just use the options `--sh-assume-role` to specify the role and `--sh-account` with the same AWS Account ID where you are logged in.

- `--sh-region`: The AWS Region where Security Hub is running. If you don't specify a region, it will use the one configured in your environment. If you are using AWS Security Hub Cross-Region aggregation, you should use that region as the --sh-region option so that you can fetch all findings together.

- `--sh-account` and `--sh-assume-role`: The AWS Account ID where Security Hub is running and the AWS IAM role to assume in that account. These options are helpful when you are logged in to a different AWS Account than the one where AWS Security Hub is running or when running AWS Security Hub in a multiple AWS Account setup. Both options must be used together. The role provided needs to have enough policies to get and update findings in AWS Security Hub (if needed). If you don't specify a `--sh-account`, MetaHub will assume the one you are logged in.

- `--sh-profile`: You can also provide your AWS profile name to use for AWS Security Hub. When using this option, you don't need to specify `--sh-account` or `--sh-assume-role` as MetaHub will use the credentials from the profile. If you are using `--sh-account` and `--sh-assume-role`, those options take precedence over `--sh-profile`.

## IAM Policy for Security Hub

This is the minimum IAM policy you need to read and write from AWS Security Hub. If you don't want to update your findings with MetaHub, you can remove the `securityhub:BatchUpdateFindings` action.

```sh
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "security hub:GetFindings",
                "security hub:ListFindingAggregators",
                "security hub:BatchUpdateFindings",
                "iam:ListAccountAliases"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
```

# Configuring Context

If you are running MetaHub for a multiple AWS Account setup (AWS Security Hub is aggregating findings from multiple AWS Accounts), you must provide the role to assume for Context queries because the affected resources are not in the same AWS Account that the AWS Security Hub findings. The `--mh-assume-role` will be used to connect with the affected resources directly in the affected account. This role needs to have enough policies for being able to describe resources.

## IAM Policy for Context

The minimum policy needed for context includes the managed policy `arn:aws:iam::aws:policy/SecurityAudit` and the following actions:

- `tag:GetResources`
- `lambda:GetFunction`
- `lambda:GetFunctionUrlConfig`
- `cloudtrail:LookupEvents`
- `account:GetAlternateContact`
- `organizations:DescribeAccount`
- `iam:ListAccountAliases`

# Inputs

MetaHub can read security findings directly from AWS Security Hub using its API. If you don't use Security Hub, you can use any ASFF-compatible scanner. Most cloud security scanners support the ASFF format like Prolwer, Steampipe, Trivy, and more.

If you want to read from an input ASFF file, you need to use the option (`--inputs file-asff`) and provide the path to the file. You can provide multiple files separated by a space.:

```sh
./metahub.py --inputs file-asff --input-asff path/to/the/file.json.asff path/to/the/file2.json.asff
```

You also can combine AWS Security Hub findings with input ASFF files specifying both inputs (`--inputs file-asff securityhub`). MetaHub will process all findings together and end up with a single output.:

```sh
./metahub.py --inputs file-asff securityhub --input-asff path/to/the/file.json.asff
```

When using a file as input, you can't use the option `--sh-filters` for filter findings, as this option relies on AWS API for filtering. You can't use the options `--update-findings` or `--enrich-findings` as those findings are not in the AWS Security Hub. If you are reading from both sources at the same time, only the findings from AWS Security Hub will be updated.

MetaHub also implements some **fixing mechanisms** for the ASFF format, when they are not correctly formatted. This is a best-effort approach to make the ASFF as useful as possible, but it's not perfect and needs to be fixed in the source scanner.

- If the key `Region` is missing from the Resources and from the Root Level, MetaHub will calculate the region based on the ARN of the affected resource.
- If the ASFF file is not correctly setting the ASFF Resource Type, MetaHub will calculate it based on the ARN of the affected resource using the library [aws-arn](https://github.com/gabrielsoltz/aws-arn)
- If any other field is missing like `SeverityLabel`, `Workflow`, `RecordState`, `Compliance`, `Id`, `ProductArn` or `StandardsControlArn`, MetaHub will set them to `Unknown`.

# Outputs

**MetaHub** can generate different programmatic and visual outputs. By default, all output modes are enabled: `json-short`, `json-full`, `json-statistics`, `json-inventory`, `html`, `csv`, and `xlsx`. If you want only to generate a specific output mode, you can use the option `--output-modes` with the desired output mode. The outputs will be saved in the `outputs/` folder with the execution date.

For example, if you only want to generate the output `json-short`, you can use:

```sh
./metahub.py --output-modes json-short
```

If you want to generate `json-short`, `json-full` and `html` outputs, you can use:

```sh
./metahub.py --output-modes json-short json-full html
```

**MetaHub** organizes the security findings affecting the same resource all together under the `findings` key for in an attempt to avoid Shadowing (when two checks refer to the same issue, but one in a more generic way than the other one) and Duplication (when you use more than one scanner and get the same problem from more than one.). You can see this behaviour clear in the outputs `json-short`, `json-full` and `html`.

- [JSON](#json)
- [HTML](#html)
- [CSV](#csv)
- [XLSX](#xlsx)
- [SQLite](#sqlite)

## JSON

> :info: For exploring a JSON output interactively, you can use the tool [fx](https://github.com/antonmedv/fx).

### JSON-Full

Shows the affected resource using it's ARN as the key and the findings affecting it as a list under the key `findings`. In adittion to the findings, you will also get the following keys: `ResourceType`, `Region`, `AwsAccountId`, `associations`, `config`, `tags`, `account`, `cloudtrail`, and `impact`.

```json
"arn:aws:ec2:eu-west-1:1234567890:instance/i-0a40b2be25dbac0ac": {           --> The affected resource ARN
  "findings": [                                                              --> The findings affecting the resource
    {
      "EC2 instances should use Instance Metadata Service Version 2 (IMDSv2)": {   --> The finding title
        "SeverityLabel": "HIGH",                                                   --> The finding severity label
        "Workflow":{                                                               --> The finding workflow
          "Status": "NEW",
        },
        "RecordState": "ACTIVE",                                                   --> The finding record state
        "Compliance":{                                                             --> The finding compliance
          "Status": "FAILED",
        },
        "Id": "arn:aws:securityhub:eu-west-1:123456789012:security-control/EC2.8/finding/a1d4f19f-453e-4c3c-b486-8443c73e84f1",
        "ProductArn": "arn:aws:securityhub:eu-west-1::product/aws/securityhub",
      }
    },
    {"EC2 instances should be managed by AWS Systems Manager": {...}},        --> Another finding title
    {"EC2 instances should not have a public IPv4 address": {...}}          --> Another finding title
  ],
  "ResourceType": "AwsEc2Instance",                                         --> The affected resource type
  "Region": "eu-west-1",                                                    --> The affected resource region
  "AwsAccountId": "1234567890",                                             --> The affected resource account id
  "associations": {                                                         --> The associations of the affected resource
    "security_groups": {},                                                  --> The security groups associated with the affected resource
    "iam_roles": {},                                                        --> The IAM roles associated with the affected resource
    "volumes": {},                                                          --> The volumes associated with the affected resource
    "autoscaling_groups": {},                                               --> The autoscaling groups associated with the affected resource
    "vpcs": {},                                                             --> The VPCs associated with the affected resource
    "subnets": {},                                                          --> The subnets associated with the affected resource
  },
  "config": {                                                              --> The configuration of the affected resource (based on it's type)
    "public_ip": "200.200.200.200",                                       --> The public IP of the affected resource, if any
    "private_ip": "10.10.10.10",                                        --> The private IP of the affected resource, if any
    "key": "ssh-key",                                                  --> The key used for the affected resource, if any
    "metadata_options": {                                             --> The metadata options of the affected resource, if any
      "State": "applied",
      "HttpTokens": "required",
      "HttpPutResponseHopLimit": 1,
      "HttpEndpoint": "enabled",
      "HttpProtocolIpv6": "disabled",
      "InstanceMetadataTags": "disabled"
    },
  },
  "tags": {                                                            --> The tags of the affected resource
    "Name": "test",                                                  --> The tag key and value
    "Env": "prod",
    "awsApplication": "arn:aws:resource-groups:eu-west-1:1234567890:group/app1/0c8vpbjkzeeffsz2cqgxpae7b2"
  },
  "account": {                                                            --> The account of the affected resource
    "Alias": "prod",                                                --> The account alias
    "AlternateContact": {},                                        --> The alternate contact of the account, if any
    "Organizations": {                                            --> The organization of the account, if any
      "Id": "o-1234567890",
      "Arn": "arn:aws:organizations::1234567890:organization/o-1234567890/o-1234567890",
      "MasterAccountId": "1234567890",
      "MasterAccountArn": "arn:aws:organizations::1234567890:account/o-1234567890/1234567890",
      "MasterAccountEmail": "",
      "Details": {
        "ParentId": "p-k1234567890",
        "ParentType": "ROOT",
        "OU": "ROOT",
        "Policies": {                                           --> The policies of the account, if any
          "p-FullAWSAccess": {...}                              --> The policy name and policy
        }
      }
    },
  },
  "cloudtrail": {                                                     --> The CloudTrail events affecting the affected resource
    "RunInstances": {                                                --> The CloudTrail event name
      "Username": "test",                                            --> The username of the event, if any
      "EventTime": "2021-01-01T00:00:00Z",
      "EventId": "12345678-1234-1234-1234-123456789012",
    }
  },
  "impact": {                                                        --> The impact of the affected resource
    "exposure": {...},                                             --> The exposure impact
    "access": {...},                                              --> The access impact
    "encryption": {...},                                         --> The encryption impact
    "status": {...},                                            --> The status impact
    "environment": {...},                                      --> The environment impact
    "application": {...},                                     --> The application impact
    "owner": {...},                                          --> The owner impact
    "findings": {...},                                      --> The findings impact
    "score": {...}                                         --> The total impact score
  }
}
```

### JSON-Short

```json
"arn:aws:ec2:eu-west-1:1234567890:instance/i-0a40b2be25dbac0ac": {           --> The affected resource ARN
  "findings": [                                                              --> The findings affecting the resource
    "EC2 instances should use Instance Metadata Service Version 2 (IMDSv2)", --> The finding title
    "EC2 instances should be managed by AWS Systems Manager",                --> The finding title
    "EC2 instances should not have a public IPv4 address"                    --> The finding title
  ],
  "ResourceType": "AwsEc2Instance",                                         --> The affected resource type
  "Region": "eu-west-1",                                                    --> The affected resource region
  "AwsAccountId": "1234567890",                                             --> The affected resource account id
  "associations": {                                                         --> The associations of the affected resource
    "security_groups": {},                                                  --> The security groups associated with the affected resource
    "iam_roles": {},                                                        --> The IAM roles associated with the affected resource
    "volumes": {},                                                          --> The volumes associated with the affected resource
    "autoscaling_groups": {},                                               --> The autoscaling groups associated with the affected resource
    "vpcs": {},                                                             --> The VPCs associated with the affected resource
    "subnets": {},                                                          --> The subnets associated with the affected resource
  },
  "config": {                                                              --> The configuration of the affected resource (based on it's type)
    "public_ip": "200.200.200.200",                                       --> The public IP of the affected resource, if any
    "private_ip": "10.10.10.10",                                        --> The private IP of the affected resource, if any
    "key": "ssh-key",                                                  --> The key used for the affected resource, if any
    "metadata_options": {                                             --> The metadata options of the affected resource, if any
      "State": "applied",
      "HttpTokens": "required",
      "HttpPutResponseHopLimit": 1,
      "HttpEndpoint": "enabled",
      "HttpProtocolIpv6": "disabled",
      "InstanceMetadataTags": "disabled"
    },
  },
  "tags": {                                                            --> The tags of the affected resource
    "Name": "test",                                                  --> The tag key and value
    "Env": "prod",
    "awsApplication": "arn:aws:resource-groups:eu-west-1:1234567890:group/app1/0c8vpbjkzeeffsz2cqgxpae7b2"
  },
  "account": {                                                            --> The account of the affected resource
    "Alias": "prod",                                                --> The account alias
    "AlternateContact": {},                                        --> The alternate contact of the account, if any
    "Organizations": {                                            --> The organization of the account, if any
      "Id": "o-1234567890",
      "Arn": "arn:aws:organizations::1234567890:organization/o-1234567890/o-1234567890",
      "MasterAccountId": "1234567890",
      "MasterAccountArn": "arn:aws:organizations::1234567890:account/o-1234567890/1234567890",
      "MasterAccountEmail": "",
      "Details": {
        "ParentId": "p-k1234567890",
        "ParentType": "ROOT",
        "OU": "ROOT",
        "Policies": {                                           --> The policies of the account, if any
          "p-FullAWSAccess": {...}                              --> The policy name and policy
        }
      }
    },
  },
  "cloudtrail": {                                                     --> The CloudTrail events affecting the affected resource
    "RunInstances": {                                                --> The CloudTrail event name
      "Username": "test",                                            --> The username of the event, if any
      "EventTime": "2021-01-01T00:00:00Z",
      "EventId": "12345678-1234-1234-1234-123456789012",
    }
  },
  "impact": {                                                        --> The impact of the affected resource
    "exposure": {...},                                             --> The exposure impact
    "access": {...},                                              --> The access impact
    "encryption": {...},                                         --> The encryption impact
    "status": {...},                                            --> The status impact
    "environment": {...},                                      --> The environment impact
    "application": {...},                                     --> The application impact
    "owner": {...},                                          --> The owner impact
    "findings": {...},                                      --> The findings impact
    "score": {...}                                         --> The total impact score
  }
}
```

### JSON-Inventory

Show a list of all resources with their ARN.

```json
[
  "arn:aws:sagemaker:us-east-1:ofuscated:notebook-instance/obfuscated",
  "arn:aws:sagemaker:eu-west-1:ofuscated:notebook-instance/obfuscated"
]
```

### JSON-Statistics

Show statistics for each field/value. In the output, you will see each field/value and the number of occurrences; for example, the following output shows statistics for six findings.

```json
{
  "Title": {
    "SageMaker.1 Amazon SageMaker notebook instances should not have direct internet access": 2,
    "SageMaker.2 SageMaker notebook instances should be launched in a custom VPC": 2,
    "SageMaker.3 Users should not have root access to SageMaker notebook instances": 2
  },
  "SeverityLabel": {
    "HIGH": 6
  },
  "Workflow": {
    "NEW": 6
  },
  "RecordState": {
    "ACTIVE": 6
  },
  "Compliance": {
    "FAILED": 6
  },
  "ProductArn": {
    "arn:aws:security hub:eu-west-1::product/aws/security hub": 3,
    "arn:aws:security hub:us-east-1::product/aws/security hub": 3
  },
  "ResourceType": {
    "AwsSageMakerNotebookInstance": 6
  },
  "AwsAccountId": {
    "obfuscated": 6
  },
  "Region": {
    "eu-west-1": 3,
    "us-east-1": 3
  },
  "ResourceId": {
    "arn:aws:sagemaker:eu-west-1:ofuscated:notebook-instance/obfuscated": 3,
    "arn:aws:sagemaker:us-east-1:ofuscated:notebook-instance/obfuscated": 3
  }
}
```

## HTML

You can create rich HTML reports of your findings, adding your context as part of them.

HTML Reports are interactive in many ways:

- You can add/remove columns.
- You can sort and filter by any column.
- You can auto-filter by any column
- You can group/ungroup findings
- You can also download that data to xlsx, CSV, HTML, and JSON.

## CSV

You can create CSV reports of your findings, adding your context as part of them.

## XLSX

Similar to CSV but with more formatting options.

## Customize HTML, CSV or XLSX outputs

You can customize which Context keys to unroll as columns for your HTML, CSV, and XLSX outputs using the options `--output-tag-columns` and `--output-config-columns` (as a list of columns). If the keys you specified don't exist for the affected resource, they will be empty. You can also configure these columns by default in the configuration file (See [Customizing Configuration](#customizing-configuration)).

For example, you can generate an HTML output with Tags and add "Owner" and "Environment" as columns to your report using the:

```sh
./metahub --output-modes html --output-tag-columns Owner Environment
```

## SQLite

**MetaHub** can save the findings and context in a SQLite database. This database can be used for further analysis, reporting, and integration with other tools, like [Powerpipe](https://powerpipe.io/). The database will be saved in the `outputs/` folder with the execution date and extension `.db`.

### Use it with Powerpipe

You can use the SQLite database generated by MetaHub with [Powerpipe](https://powerpipe.io/). You can find MetaHub PowerPipe mod under the folder `powerpipe/`.

To use it, you need to have Powerpipe [installed in your system](https://powerpipe.io/downloads).

Once installed, you can run the following command:

```sh
powerpipe server --database sqlite:../outputs/metahub-20240331-182942.db
```

### Build your own dashboards

Building dashboards with Powerpipe is supper easy and powerfull. You can check the [Powerpipe documentation](https://powerpipe.io/docs) for more information.

Below you will find the tables available in the SQLite database.

#### Resources

| Column Name                     | Data Type | Description                                                                                    |
| ------------------------------- | --------- | ---------------------------------------------------------------------------------------------- |
| resource_arn                    | VARCHAR   | The Amazon Resource Name (ARN) of the resource, serving as the primary key.                    |
| resource_type                   | VARCHAR   | The type of the resource.                                                                      |
| resource_region                 | VARCHAR   | The region the resource is located in.                                                         |
| resource_account_id             | VARCHAR   | The account ID associated with the resource. Foreign key that references accounts(account_id). |
| resource_account_alias          | VARCHAR   | The alias of the account associated with the resource.                                         |
| resource_tags                   | TEXT      | Tags associated with the resource.                                                             |
| resource_exposure               | VARCHAR   | The exposure level of the resource.                                                            |
| resource_access                 | VARCHAR   | The access level of the resource.                                                              |
| resource_encryption             | VARCHAR   | The encryption status of the resource.                                                         |
| resource_status                 | VARCHAR   | The current status of the resource.                                                            |
| resource_application            | VARCHAR   | The application associated with the resource.                                                  |
| resource_environment            | VARCHAR   | The environment (e.g., production, staging) the resource is in.                                |
| resource_owner                  | VARCHAR   | The owner of the resource.                                                                     |
| resource_score                  | INTEGER   | A score associated with the resource.                                                          |
| resource_findings_score         | INTEGER   | A score based on findings associated with the resource.                                        |
| resource_findings_critical      | INTEGER   | The number of critical findings.                                                               |
| resource_findings_high          | INTEGER   | The number of high severity findings.                                                          |
| resource_findings_medium        | INTEGER   | The number of medium severity findings.                                                        |
| resource_findings_low           | INTEGER   | The number of low severity findings.                                                           |
| resource_findings_informational | INTEGER   | The number of informational findings.                                                          |

#### Findings

| Column Name              | Data Type | Description                                                        |
| ------------------------ | --------- | ------------------------------------------------------------------ |
| finding_id               | VARCHAR   | The unique identifier for the finding, serving as the primary key. |
| finding_title            | VARCHAR   | The title of the finding.                                          |
| finding_severity         | VARCHAR   | The severity of the finding.                                       |
| finding_workflowstatus   | VARCHAR   | The workflow status of the finding.                                |
| finding_recordstate      | VARCHAR   | The record state of the finding.                                   |
| finding_compliancestatus | VARCHAR   | The compliance status of the finding.                              |
| finding_productarn       | VARCHAR   | The ARN of the product generating the finding.                     |
| finding_resource_arn     | VARCHAR   | The ARN of the associated resource.                                |

#### Accounts

| Column Name                     | Data Type | Description                                                                   |
| ------------------------------- | --------- | ----------------------------------------------------------------------------- |
| account_id                      | INTEGER   | The unique identifier for the account, serving as the primary key.            |
| account_alias                   | VARCHAR   | The alias of the account.                                                     |
| account_organizations_id        | VARCHAR   | The ID of the organization the account belongs to.                            |
| account_organizations_arn       | VARCHAR   | The ARN of the organization the account belongs to.                           |
| account_master_account_id       | VARCHAR   | The ID of the master account, if this account is part of an AWS Organization. |
| account_master_account_email    | VARCHAR   | The email address associated with the master account.                         |
| account_alternate_contact_type  | VARCHAR   | The type of alternate contact (e.g., billing, security).                      |
| account_alternate_contact_name  | VARCHAR   | The name of the alternate contact.                                            |
| account_alternate_contact_email | VARCHAR   | The email address of the alternate contact.                                   |
| account_alternate_contact_phone | VARCHAR   | The phone number of the alternate contact.                                    |
| account_alternate_contact_title | VARCHAR   | The title of the alternate contact.                                           |

# Filters

You can filter the security findings and resources that you get from your source in different ways and combine all of them to get exactly what you are looking for, then re-use those filters to create automations, alerts, reports, and more.

- [Security Hub Filtering](#security-hub-filtering)
- [Security Hub Filtering using YAML templates](#security-hub-filtering-using-yaml-templates)
- [Config Filters](#config-filters)
- [Tags Filters](#tags-filters)
- [Impact Filters](#impact-filters)

## Security Hub Filtering

MetaHub supports filtering AWS Security Hub findings in the form of `KEY=VALUE` filtering for AWS Security Hub using the option `--sh-filters`, the same way you would filter using AWS CLI but limited to the `EQUALS` comparison. If you want another comparison, use the option `--sh-template` [Security Hub Filtering using YAML templates](#security-hub-filtering-using-yaml-templates).

You can check available filters in [AWS Documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/securityhub.html#SecurityHub.Client.get_findings)

```sh
./metahub --sh-filters <KEY=VALUE>
```

If you don't specify any filters, default filters are applied: `RecordState=ACTIVE WorkflowStatus=NEW`

Passing filters using this option resets the default filters. If you want to add filters to the defaults, you need to specify them in addition to the default ones. For example, adding SeverityLabel to the default filters:

```sh
./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW
```

If a value contains spaces, you should specify it using double quotes: `"ProductName="Security Hub"`

You can add how many different filters you need to your query and also add the same filter key with different values:

Examples:

- Filter by Severity (CRITICAL):

```sh
./metaHub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW SeverityLabel=CRITICAL
```

- Filter by Severity (CRITICAL and HIGH):

```sh
./metaHub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW SeverityLabel=CRITICAL SeverityLabel=HIGH
```

- Filter by Severity and AWS Account:

```sh
./metaHub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW SeverityLabel=CRITICAL AwsAccountId=1234567890
```

- Filter by Check Title:

```sh
./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW Title="EC2.22 Unused EC2 security groups should be removed"
```

- Filter by AWS Resource Type:

```sh
./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW ResourceType=AwsEc2SecurityGroup
```

- Filter by Resource ID:

```sh
./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW ResourceId="arn:aws:ec2:eu-west-1:01234567890:security-group/sg-01234567890"
```

- Filter by Finding Id:

```sh
./metahub --sh-filters Id="arn:aws:security hub:eu-west-1:01234567890:subscription/aws-foundational-security-best-practices/v/1.0.0/EC2.19/finding/01234567890-1234-1234-1234-01234567890"
```

- Filter by Compliance Status:

```sh
./metahub --sh-filters ComplianceStatus=FAILED
```

## Security Hub Filtering using YAML templates

**MetaHub** lets you create complex filters using YAML files (templates) that you can re-use when needed. YAML templates let you write filters using any comparison supported by AWS Security Hub like "EQUALS' | 'PREFIX' | 'NOT_EQUALS' | 'PREFIX_NOT_EQUALS". You can call your YAML file using the option `--sh-template <<FILE>>`.

You can find examples under the folder [templates](templates)

- Filter using YAML template default.yml:

```sh
./metaHub --sh-template templates/default.yml
```

## Config Filters

**MetaHub** supports **Config filters** (and associations) using `KEY=VALUE` where the value can only be `True` or `False` using the option `--mh-filters-config`. You can use as many filters as you want and separate them using spaces. If you specify more than one filter, you will get all resources that match **all** filters.

Config filters only support `True` or `False` values:

- A Config filter set to **True** means `True` or with data.
- A Config filter set to **False** means `False` or without data.

Config filters run after AWS Security Hub filters:

1. MetaHub fetches AWS Security Findings based on the filters you specified using `--sh-filters` (or the default ones).
2. MetaHub executes Context for the AWS-affected resources based on the previous list of findings
3. MetaHub only shows you the resources that match your `--mh-filters-config`, so it's a subset of the resources from point 1.

Examples:

- Get all Security Groups (`ResourceType=AwsEc2SecurityGroup`) with AWS Security Hub findings that are ACTIVE and NEW (`RecordState=ACTIVE WorkflowStatus=NEW`) only if they are associated to Network Interfaces (`network_interfaces=True`):

```sh
./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW ResourceType=AwsEc2SecurityGroup --mh-filters-config network_interfaces=True
```

- Get all S3 Buckets (`ResourceType=AwsS3Bucket`) only if they are public (`public=True`):

```sh
./metahub --sh-filters ResourceType=AwsS3Bucket --mh-filters-config public=False
```

## Tags Filters

**MetaHub** supports **Tags filters** in the form of `KEY=VALUE` where KEY is the Tag name and value is the Tag Value. You can use as many filters as you want and separate them using spaces. Specifying multiple filters will give you all resources that match **at least one** filter.

Tags filters run after AWS Security Hub filters:

1. MetaHub fetches AWS Security Findings based on the filters you specified using `--sh-filters` (or the default ones).
2. MetaHub executes Tags for the AWS-affected resources based on the previous list of findings
3. MetaHub only shows you the resources that match your `--mh-filters-tags`, so it's a subset of the resources from point 1.

Examples:

- Get all Security Groups (`ResourceType=AwsEc2SecurityGroup`) with AWS Security Hub findings that are ACTIVE and NEW (`RecordState=ACTIVE WorkflowStatus=NEW`) only if they are tagged with a tag `Environment` and value `Production`:

```sh
./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW ResourceType=AwsEc2SecurityGroup --mh-filters-tags Environment=Production
```

## Impact Filters

**MetaHub** supports **Impact filters**, you can filter by the [impact keys](#impact) calculated by MetaHub. You can use as many filters as you want and separate them using spaces. If you specify more than one filter, you will get all resources that match **all** filters.

Examples:

- Filter all Security Findings affecting resources with exposure calculated as effectively-public:

```sh
./metahub --mh-filters-impact exposure=effectively-public
```

- Filter all Security Findings affecting resources with status calculated as not-attached:

```sh
./metahub --mh-filters-impact status=not-attached
```

# Security Hub Actions

## Updating Workflow Status

You can use **MetaHub** to update your AWS Security Hub Findings workflow status (`NOTIFIED,` `NEW,` `RESOLVED,` `SUPPRESSED`) with a single command. You will use the `--update-findings` option to update all the findings from your MetaHub query. This means you can update one, ten, or thousands of findings using only one command. AWS Security Hub API is limited to 100 findings per update. Metahub will split your results into 100 items chucks to avoid this limitation and update your findings beside the amount.

For example, using the following filter: `./metahub --sh-filters ResourceType=AwsSageMakerNotebookInstance RecordState=ACTIVE WorkflowStatus=NEW` I found two affected resources with three finding each making six Security Hub findings in total.

Running the following update command will update those six findings' workflow status to `NOTIFIED` with a Note:

```sh
./metahub --update-findings Workflow=NOTIFIED Note="Enter your ticket ID or reason here as a note that you will add to the finding as part of this update."
```

The `--update-findings` will ask you for confirmation before updating your findings. You can skip this confirmation by using the option `--no-actions-confirmation`.

## Enriching Findings

You can use **MetaHub** to enrich back your AWS Security Hub Findings with Context outputs using the option `--enrich-findings`. Enriching your findings means updating them directly in AWS Security Hub. **MetaHub** uses the `UserDefinedFields` field for this.

By enriching your findings directly in AWS Security Hub, you can take advantage of features like Insights and Filters by using the extra information not available in Security Hub before.

For example, you want to enrich all AWS Security Hub findings with `WorkflowStatus=NEW`, `RecordState=ACTIVE`, and `ResourceType=AwsS3Bucket` that are `public=True` with Context outputs:

```sh
./metahub --sh-filters RecordState=ACTIVE WorkflowStatus=NEW ResourceType=AwsS3Bucket --mh-filters-checks public=True --enrich-findings
```

The `--enrich-findings` will ask you for confirmation before enriching your findings. You can skip this confirmation by using the option `--no-actions-confirmation`.

# Contributing

You can follow this guide if you want to contribute to the Context module [guide](docs/context.md).

metahub

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README