terraform-gcp-datadog-integration

KO | EN

gitlite — search

#javascript #hacktoberfest #react #ai #typescript #golang #machine-learning #rust #deep-learning #macos #java #cli #kubernetes #mcp

terraform-gcp-datadog-integration

(★ 18)

Terraform code to make the Google Cloud Platform to Datadog log collection integration easier.

terraform-gcp-datadog-integration Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Log Collection Integration - Google Cloud Platform to Datadog

This Terraform module automates the integration between **Google Cloud Platform and Datadog for Log collection**, making the process faster and more efficient.  It builds upon the foundational overview provided in the official [Datadog guide](https://docs.datadoghq.com/integrations/google_cloud_platform/#log-collection). The module simplifies integration, accelerates implementation, and addresses essential security considerations for a successful observability strategy.

While this module provides security foundational principles, it's essential to note that in highly sensitive or production Google Cloud environments, additional layers of security and design principles should be thoughtfully analyzed and applied to uphold the highest standards of data protection and security principles (e.g. Utilize a bucket for TF state backup and encryption, egress traffic flow analysis, apply the module across various folders, disruption analysis, etc).

These deployment scripts are provided 'as is', without warranty. See [Copyright & License](https://github.com/googlecloudplatform/terraform-gcp-datadog-integration/blob/main/LICENSE).

## Solution diagram

![Image alt text](https://raw.githubusercontent.com/GoogleCloudPlatform/terraform-gcp-datadog-integration/main/gcp-to-datadog-diagram.png)

## Resources created

* `google_logging_folder_sink` OR `google_logging_project_sink`: Logs forwarder from Google Cloud Logging to Pub/Sub.
* `google_pubsub_topic` & `google_pubsub_subscription`: Service that handles the Logs sent by Cloud Logging and delivers it to Dataflow.
* `google_secret_manager_secret` & `google_secret_manager_secret_version`: Used to store the Datadog API Key in a secure way.
* `google_dataflow_job`: Create the Dataflow worker machines (Compute Engine) and generate a Dataflow job to pull logs from Pub/Sub subscription and export it to Datadog.
* `google_storage_bucket`: Used to store Dataflow temporary files.
* `google_compute_region_network_firewall_policy_rule`: Used to allow traffic from Dataflow workers private IP's to Datadog.
* `google_compute_firewall`: Used to allow internal traffic between Dataflow worker machines.
* `google_compute_router`: Used to serve as the control plane for network packets and to be attached to Cloud NAT.
* `google_compute_router_nat`: Required for outbound connections to the internet for the Dataflow private IP's workers - **IMPORTANT**: Take into account that if you already have virtual machines (VMs) in the same subnet as the one that Cloud NAT will use, those VMs will have outbound connectivity too.

**Note**: It's recommended to utilize a distinct project specifically for the deployment of all resources related to this integration.

## Prerequisites

* Enable the `serviceusage.googleapis.com`, `cloudresourcemanager.googleapis.com` & `pubsub.googleapis.com` APIs from the beginning to avoid any kind of errors. 
* Download `gcloud` CLI.
* Download `terraform` CLI.
* Ensure user's IAM policies have `secretmanager.versions.access` permissions applied.
* Perform `gcloud auth login` before performing terraform commands.
* A Virtual Private Cloud (VPC) - Required in the '**`vpc_name`**' input variable.
* A network subnet attached to the VPC with [Private Google Access](https://cloud.google.com/vpc/docs/private-google-access) enabled (Resources will be created in the subnet's region) - Required in the '**`subnet_name`**' input variable.
* Datadog API Key - Required in the '**`datadog_api_key`**' input variable
* Your Datadog Logs API URL (You can find it [here](https://docs.datadoghq.com/integrations/google_cloud_platform/#4-create-and-run-the-dataflow-job), **ensure** the site selector on the right part of the page is the correct for your Datadog site) - used in the '**`datadog_site_url`**' input variable
* A pre-selected inclusion filter for the logs you want to be sent to Datadog - **IMPORTANT**: To make the filter works using Terraform you must use *unquoted* filters.

Instead of **this**: 
  ```python
  resource.type="gce_instance" AND protoPayload.methodName="v1.compute.instances.stop"
  ```
  Use **this**:
  ```python
  resource.type=gce_instance AND protoPayload.methodName=v1.compute.instances.stop
  ```

If your desired filter is too complicated, start with a basic one or an empty one (*inclusion_filter = ""*). After running 'Terraform apply', you can easily adjust the filter in the sink configs manually.

## Usage

Fill the `main.tf` file with your input variables and from the root folder of this repo (where the `main.tf` file exists) run the **terraform init, plan, and apply** commands.

### Use a Log sink at the folder level

```hcl
provider "google" {
  project = "my-gcp-project-id"
  region  = "us-east1"
}

provider "google-beta" {
  project = "my-gcp-project-id"
  region  = "us-east1"
}

module "datadog-integration" {
  source                    = "./gcp-datadog-module"
  project_id                = "my-gcp-project-id"
  dataflow_job_name         = "datadog-export-job"
  dataflow_temp_bucket_name = "my-temp-bucket"
  topic_name                = "datadog-export-topic"
  subscription_name         = "datadog-export-sub"
  vpc_name                  = "vpc-name"
  subnet_name               = "subnet-name"
  subnet_region             = "us-east1"
  datadog_api_key           = "ab1c23d4ef56789a0bc1d23ef45ab6789"
  datadog_site_url          = "https://http-intake.logs.us5.datadoghq.com"
  log_sink_in_folder        = true
  folder_id                 = "123456789012"
  inclusion_filter          = "resource.type=gce_instance AND protoPayload.methodName=v1.compute.instances.stop"
}
```

### Use a Log sink at the project level

```hcl
provider "google" {
  project = "my-gcp-project-id"
  region  = "us-east1"
}

provider "google-beta" {
  project = "my-gcp-project-id"
  region  = "us-east1"
}

## Variables

## Outputs

| Output Name                  | Description                                          | Value                                                        |
| ---------------------------- | ---------------------------------------------------- | ------------------------------------------------------------ |
| dataflow_job_name            | The name of the created Dataflow job.                | `google_dataflow_job.pubsub_stream_to_datadog.name`         |
| temp_files_bucket_name       | The name of the created temporary files bucket.      | `google_storage_bucket.temp_files_bucket.name`              |
| datadog_topic_name           | The name of the created Pub/Sub topic.               | `google_pubsub_topic.datadog_topic.name`                    |
| datadog_subscription_name    | The name of the created Pub/Sub subscription.        | `google_pubsub_subscription.datadog_topic_sub.name`         |

## Additional Considerations

* Customization: The module is flexible and allows you to customize various aspects of the integration. Your specific Google Cloud organization might have unique requirements, so you may need to adjust the code to suit your particular situation (e.g., deploying the log sink across multiple folders or at the organization level, use a different solution for API key store, etc.)

## Authors

* **Diego González** - [diegonz2](https://github.com/diegonz2)

## Troubleshooting

* If the Dataflow job fails with a *"Workflow failed. Causes: There was a problem refreshing your credentials"* error, rerun "terraform apply"
* If you receive errors about API's not being used/enabled in your project, wait some minutes and rerun "terraform apply"

terraform-gcp-datadog-integration

terraform-gcp-datadog-integration Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

terraform-gcp-datadog-integration

terraform-gcp-datadog-integration Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README