dbt-databricks - GitLite

KO | EN

gitlite — search

#javascript #python #hacktoberfest #ai #llm #golang #android #machine-learning #rust #docker #cli #vue #pytorch #open-source

dbt-databricks

(★ 375)

A dbt adapter for Databricks.

파일 탐색기

최종 버전 다운로드 (.zip)

- - bug_report.md
  - feature_request.md
  - release.md
- - build_cluster_http_path.py
  - ci-pr-linting.yml
  - cleanup-python-model-dirs.yml
  - coverage.yml
  - integration-kernel.yml
  - integration-min-deps.yml
  - integration-spog.yml
  - integration-trigger.yml
  - integration.yml
  - main-min-deps.yml
  - main.yml
  - min-deps-lock-drift.yml
  - refresh-test-timings.yml
  - stale.yml
  - warmDepsCache.yml
- CODEOWNERS
- dependabot.yml
- pull_request_template.md
- test_timings.json
- __init__.py
- - incremental_flow.md
  - materialized_view_flow.md
  - README.md
  - replace_flow.md
  - seed_flow.md
  - snapshot_flow.md
  - streaming_table_flow.md
  - table_flow.md
  - view_flow.md
- - databricks-copy-into-macro-aws.md
  - databricks-jobs.md
  - uc.md
  - workflow-job-submission.md
- - ci-cd-overview.png
  - cluster-connection-details.png
  - dbt-task-type.png
  - sql-endpoint-connection-details.png
- dbr-capability-system.md
- dbt-databricks-dev.md
- README.md
- testing.md
- check_uv_lock_public_urls.py
- cleanup_python_model_dirs.py
- regenerate_min_deps_lock.sh
- regenerate_timings.py
- shard_assign.py
- shard_verify.py
- - .gitignore
  - dbt_project.yml
  - profiles.yml
- - __init__.py
  - conftest.py
  - fixtures.py
  - test_adapter.py
  - test_adapter_capabilities.py
  - test_auth.py
  - test_catalogs_v2.py
  - test_cleanup_python_model_dirs.py
  - test_column.py
  - test_connection_manager.py
  - test_constraints.py
  - test_credentials_cluster_id_regex.py
  - test_dbr_capabilities.py
  - test_describe_json_metadata.py
  - test_handle.py
  - test_integration_workflow_dispatch.py
  - test_parse_model.py
  - test_query_comment.py
  - test_query_config.py
  - test_query_tags.py
  - test_regenerate_timings.py
  - test_relation.py
  - test_utils.py
  - utils.py
- __init__.py
- conftest.py
- profiles.py
.gitignore
.pre-commit-config.yaml
AGENTS.md
CHANGELOG.md
CLAUDE.md
CONTRIBUTING.MD
License.md
pyproject.toml
README.md
requirements.lowest-direct.txt
test.env.example
uv.lock

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<p align="center">
  <img src="https://bynder-public-us-west-2.s3.amazonaws.com/styleguide/ABB317701CA31CB7F29268E32B303CAE-pdf-column-1.png" alt="databricks logo" width="50%" />
  <img src="https://raw.githubusercontent.com/dbt-labs/dbt/ec7dee39f793aa4f7dd3dae37282cc87664813e4/etc/dbt-logo-full.svg" alt="dbt logo" width="250"/>
</p>
<p align="center">
  <a href="https://github.com/databricks/dbt-databricks/actions/workflows/main.yml">
    <img src="https://github.com/databricks/dbt-databricks/actions/workflows/main.yml/badge.svg?event=push" alt="Unit Tests Badge"/>
  </a>
  <a href="https://github.com/databricks/dbt-databricks/actions/workflows/integration.yml">
    <img src="https://github.com/databricks/dbt-databricks/actions/workflows/integration.yml/badge.svg?event=push" alt="Integration Tests Badge"/>
  </a>
</p>

**[dbt](https://www.getdbt.com/)** enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

The **[Databricks Lakehouse](https://www.databricks.com/)** provides one simple platform to unify all your data, analytics and AI workloads.

# dbt-databricks

The `dbt-databricks` adapter contains all of the code enabling dbt to work with Databricks. This adapter is based off the amazing work done in [dbt-spark](https://github.com/dbt-labs/dbt-spark). Some key features include:

- **Easy setup**. No need to install an ODBC driver as the adapter uses pure Python APIs.
- **Open by default**. For example, it uses the the open and performant [Delta](https://delta.io/) table format by default. This has many benefits, including letting you use `MERGE` as the the default incremental materialization strategy.
- **Support for Unity Catalog**. dbt-databricks supports the 3-level namespace of Unity Catalog (catalog / schema / relations) so you can organize and secure your data the way you like.
- **Performance**. The adapter generates SQL expressions that are automatically accelerated by the native, vectorized [Photon](https://databricks.com/product/photon) execution engine.

## Choosing between dbt-databricks and dbt-spark

If you are developing a dbt project on Databricks, we recommend using `dbt-databricks` for the reasons noted above.

`dbt-spark` is an actively developed adapter which works with Databricks as well as Apache Spark anywhere it is hosted e.g. on AWS EMR.

## Getting started

### Installation

Install using pip:

```nofmt
pip install dbt-databricks
```

Upgrade to the latest version

```nofmt
pip install --upgrade dbt-databricks
```

### Profile Setup

```nofmt
your_profile_name:
  target: dev
  outputs:
    dev:
      type: databricks
      catalog: [optional catalog name, if you are using Unity Catalog]
      schema: [database/schema name]
      host: [your.databrickshost.com]
      http_path: [/sql/your/http/path]
      token: [dapiXXXXXXXXXXXXXXXXXXXXXXX]
```

### Documentation

For documentation on Databricks-specific features, configurations, and capabilities:

- **[Databricks configurations](https://docs.getdbt.com/reference/resource-configs/databricks-configs)** - Complete reference for all Databricks-specific model configurations, materializations, and incremental strategies
- **[Connect to Databricks](https://docs.getdbt.com/docs/core/connect-data-platform/databricks-setup)** - Setup and authentication guide

### Quick Starts

These following quick starts will get you up and running with the `dbt-databricks` adapter:

- [Set up your dbt project with Databricks](https://docs.getdbt.com/guides/set-up-your-databricks-dbt-project)
- Using dbt Cloud with Databricks ([Azure](https://docs.microsoft.com/en-us/azure/databricks/integrations/prep/dbt-cloud) | [AWS](https://docs.databricks.com/integrations/prep/dbt-cloud.html))
- [Submitting Python models as Databricks Workflows](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/workflow-job-submission.md)
- [Running a dbt project as a Databricks job](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/databricks-jobs.md)
- [Using Unity Catalog with dbt-databricks](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/uc.md)
- [Continuous integration in dbt](https://docs.getdbt.com/docs/deploy/continuous-integration)
- [Loading data from S3 into Delta using the databricks_copy_into macro](https://github.com/databricks/dbt-databricks/blob/main/docs/guides/databricks-copy-into-macro-aws.md)
- [Contribute to this repository](CONTRIBUTING.MD)

### Compatibility

The `dbt-databricks` adapter has been tested:

- with Python 3.7 or above.
- against `Databricks SQL` and `Databricks runtime releases 9.1 LTS` and later.

### Tips and Tricks

## Choosing compute for a Python model

You can override the compute used for a specific Python model by setting the `http_path` property in model configuration. This can be useful if, for example, you want to run a Python model on an All Purpose cluster, while running SQL models on a SQL Warehouse. Note that this capability is only available for Python models.

```
def model(dbt, session):
    dbt.config(
      http_path="sql/protocolv1/..."
    )
```

## Python models and ANSI mode

When ANSI mode is enabled (`spark.sql.ansi.enabled=true`), there are limitations when using pandas DataFrames in Python models:

1. **Regular pandas DataFrames**: dbt-databricks will automatically handle conversion even when ANSI mode is enabled, falling back to `spark.createDataFrame()` if needed.

2. **pandas-on-Spark DataFrames**: If you create pandas-on-Spark DataFrames directly in your model (using `pyspark.pandas` or `databricks.koalas`), you may encounter errors with ANSI mode enabled. In this case, you have two options:
   - Disable ANSI mode for your session: Set `spark.sql.ansi.enabled=false` in your cluster or SQL warehouse configuration
   - Set the pandas-on-Spark option in your model code:
     ```python
     import pyspark.pandas as ps
     ps.set_option('compute.fail_on_ansi_mode', False)
     ```
     Note: This may cause unexpected behavior as pandas-on-Spark follows pandas semantics (returning null/NaN for invalid operations) rather than ANSI SQL semantics (raising errors).

For more information about ANSI mode and its implications, see the [Spark documentation on ANSI compliance](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html).