gatherer

KO | EN

gitlite — search

#javascript #hacktoberfest #typescript #golang #machine-learning #rust #deep-learning #linux #nodejs #cli #vue #pytorch #mcp #ios

(★ 21)

Gather domains as a precursor to scanning

gatherer Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Gatherer #

[![GitHub Build Status](https://github.com/cisagov/gatherer/workflows/build/badge.svg)](https://github.com/cisagov/gatherer/actions/workflows/build.yml)
[![License](https://img.shields.io/github/license/cisagov/gatherer)](https://spdx.org/licenses/)
[![CodeQL](https://github.com/cisagov/gatherer/workflows/CodeQL/badge.svg)](https://github.com/cisagov/gatherer/actions/workflows/codeql-analysis.yml)

## Docker Image ##

[![Docker Pulls](https://img.shields.io/docker/pulls/cisagov/gatherer)](https://hub.docker.com/r/cisagov/gatherer)
[![Docker Image Size (latest by date)](https://img.shields.io/docker/image-size/cisagov/gatherer)](https://hub.docker.com/r/cisagov/gatherer)
[![Platforms](https://img.shields.io/badge/platforms-386%20%7C%20amd64%20%7C%20arm%2Fv6%20%7C%20arm%2Fv7%20%7C%20arm64%20%7C%20ppc64le%20%7C%20s390x-blue)](https://hub.docker.com/r/cisagov/gatherer/tags)

This is a Docker container that uses [the `cisagov`
fork](https://github.com/cisagov/domain-scan) of 18F's
[`domain-scan`](https://github.com/18F/domain-scan) to gather domains
as a precursor to scanning by
[pshtt](https://github.com/cisagov/pshtt),
[trustymail](https://github.com/cisagov/trustymail), and
[sslyze](https://github.com/nabla-c0d3/sslyze).

This Docker container is intended to be run via
[orchestrator](https://github.com/cisagov/orchestrator).

**N.B.:** The secrets in the `src/secrets` directory are only used
when testing via the `compose.yml` composition.  Normally this
Docker container is run via the Docker composition in
[cisagov/orchestrator](https://github.com/cisagov/orchestrator), which
expects the secrets in a different location.

## Running ##

### Running with Docker ###

To run the `cisagov/gatherer` image via Docker:

```console
docker run cisagov/gatherer:2.0.5
```

### Running with Docker Compose ###

1. Create a `compose.yml` file similar to the one below to use [Docker Compose](https://docs.docker.com/compose/).

```yaml
    ---
    name: gatherer

services:
      gatherer:
        image: cisagov/gatherer:2.0.5
        volumes:
          - source: <your_output_dir>
            target: /home/cisa/shared
            type: bind
    ```

1. Start the container and detach:

```console
    docker compose up --detach
    ```

## Using secrets with your container ##

This container also supports passing sensitive values via [Docker
secrets](https://docs.docker.com/engine/swarm/secrets/).  Passing sensitive
values like your credentials can be more secure using secrets than using
environment variables.  See the
[secrets](#secrets) section below for a table of all supported secret files.

1. To use secrets, create a `database_creds.yml` file in [this
   format](https://github.com/cisagov/mongo-db-from-config#usage):

```yml
    ---
    version: '1'

database:
      name: cyhy
      uri: mongodb://readonly:the_password@cyhy.example.com:27017/cyhy

```

1. Then add the secret to your `compose.yml` file:

```yaml
    ---
    name: gatherer

secrets:
      database_creds:
        file: database_creds.yml

services:
      gatherer:
        image: cisagov/gatherer:2.0.5
        secrets:
          - source: database_creds
            target: database_creds.yml
        volumes:
          - source: <your_output_dir>
            target: /home/cisa/shared
            type: bind
    ```

## Updating your container ##

### Docker Compose ###

1. Pull the new image from Docker Hub:

```console
    docker compose pull
    ```

1. Recreate the running container by following the [previous instructions](#running-with-docker-compose):

```console
    docker compose up --detach
    ```

### Docker ###

1. Stop the running container:

```console
    docker stop <container_id>
    ```

1. Pull the new image:

```console
    docker pull cisagov/gatherer:2.0.5
    ```

1. Recreate and run the container by following the [previous instructions](#running-with-docker).

## Updating Python dependencies ##

This image uses [Pipenv] to manage Python dependencies using a [Pipfile](https://github.com/pypa/pipfile).
Both updating dependencies and changing the [Pipenv] configuration in `src/Pipfile`
will result in a modified `src/Pipfile.lock` file that should be committed to the
repository.

### Updating dependencies ###

If you want to update existing dependencies you would run the following command
in the `src/` subdirectory:

```console
pipenv lock
```

### Modifying dependencies ###

If you want to add or remove dependencies you would update the `src/Pipfile` file
and then update dependencies as you would above.

> [!NOTE]
> You should only specify packages that are direct requirements of
> your Docker configuration. Allow [Pipenv] to manage the dependencies
> of the specified packages.

## Image tags ##

The images of this container are tagged with [semantic
versions](https://semver.org) of the underlying gatherer project that they
containerize.  It is recommended that most users use a version tag (e.g.
`:2.0.5`).

| Image:tag | Description |
| --------- | ----------- |
| `cisagov/gatherer:2.0.5` | An exact release version. |
| `cisagov/gatherer:2.0` | The most recent release matching the major and minor version numbers. |
| `cisagov/gatherer:2` | The most recent release matching the major version number. |
| `cisagov/gatherer:edge` | The most recent image built from a merge into the `develop` branch of this repository. |
| `cisagov/gatherer:nightly` | A nightly build of the `develop` branch of this repository. |
| `cisagov/gatherer:latest` | The most recent release image pushed to a container registry.  Pulling an image using the `:latest` tag [should be avoided.](https://vsupalov.com/docker-latest-tag/) |

See the [tags tab](https://hub.docker.com/r/cisagov/gatherer/tags) on Docker
Hub for a list of all the supported tags.

## Volumes ##

| Mount point | Purpose |
| ----------- | ------- |
| /home/cisa/shared | Output |

## Ports ##

There are no ports exposed by this container.

## Environment variables ##

### Required ###

There are no required environment variables.

### Optional ###

There are no optional environment variables.

## Secrets ##

| Filename | Purpose |
| -------- | ------- |
| database_creds.yml | Cyber Hygiene database credentials in [this format](https://github.com/cisagov/mongo-db-from-config#usage) |

## Building from source ##

Build the image locally using this git repository as the [build context](https://docs.docker.com/engine/reference/commandline/build/#git-repositories):

```console
docker build \
  --tag cisagov/gatherer:2.0.5 \
  https://github.com/cisagov/gatherer.git#develop
```

## Cross-platform builds ##

To create images that are compatible with other platforms, you can use the
[`buildx`](https://docs.docker.com/buildx/working-with-buildx/) feature of
Docker:

1. Copy the project to your machine using the `Code` button above
   or the command line:

```console
    git clone https://github.com/cisagov/gatherer.git
    cd gatherer
    ```

1. Create the `Dockerfile-x` file with `buildx` platform support:

```console
    ./buildx-dockerfile.sh
    ```

1. Build the image using `buildx`:

```console
    docker buildx build \
      --file Dockerfile-x \
      --platform linux/amd64 \
      --output type=docker \
      --tag cisagov/gatherer:2.0.5 .
    ```

## Contributing ##

We welcome contributions!  Please see [`CONTRIBUTING.md`](CONTRIBUTING.md) for
details.

## License ##

This project is in the worldwide [public domain](LICENSE).

This project is in the public domain within the United States, and
copyright and related rights in the work worldwide are waived through
the [CC0 1.0 Universal public domain
dedication](https://creativecommons.org/publicdomain/zero/1.0/).

All contributions to this project will be released under the CC0
dedication. By submitting a pull request, you are agreeing to comply
with this waiver of copyright interest.

[Pipenv]: https://pypi.org/project/pipenv/

gatherer

gatherer Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

gatherer

gatherer Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README