KO
|
EN
gitlite — search
Search
#dotfiles
#vim
#ruby
#python
#java
#plugin
#mantisbt-plugin
#javascript
#php
#ai-agent
#nodejs
#adopt-this-plugin
kedro-databricks
★ 26
Open GitHub ↗
A Kedro Plugin for Databricks
Download README (.md)
Explore Similar Repositories
kedro-pandera
:
A kedro plugin to use pandera in your kedro projects
kedro-static-viz
:
kedro cli plugin for generating a static kedro viz site (html, css, js) that can be deployed on many serverless tools.
kedro-airflow-k8s
:
Kedro Plugin to support running pipelines on Kubernetes using Airflow.
kedro-sagemaker
:
Kedro Plugin to support running pipelines on AWS SageMaker.
kedro-dagster
:
Kedro plugin to support running pipelines on Dagster
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
kedro-databricks
?
Download (.md)
# kedro-databricks [](https://github.com/astral-sh/uv) [](https://github.com/astral-sh/ruff) [](https://opensource.org/licenses/MIT) [](https://codecov.io/gh/JenspederM/kedro-databricks) [](https://pypi.org/project/kedro-databricks/) [](https://pypi.org/project/kedro-databricks/) [](https://pypi.org/project/kedro-databricks/) [](https://kedro-databricks.readthedocs.io/) Kedro plugin to develop Kedro pipelines for Databricks. This plugin strives to provide an excellent developer experience when using Kedro on Databricks. ## Key Features 1. **Initialization**: Transform your local Kedro project into a Databricks Asset Bundle. 2. **Generation**: Generate Asset Bundle resource definitions from your Kedro pipelines. 3. **Deployment**: Deploy your Kedro pipelines to Databricks as Jobs. 4. **Execution**: Run your Kedro pipelines on Databricks straight from the command line. 5. **Cleanup**: Remove all Databricks resources created by the plugin. ## Additional Capabilities - Generate resources per node or per pipeline (`--resource-generator`). - Extend generation with custom resource generator classes. - Apply default, targeted, and regex-based overrides in `conf/<env>/databricks.yml`. - Include non-job Databricks resources (for example volumes) in generated bundle resources. - Forward raw Databricks CLI args via `-- ...` for advanced workflows. - Automatically upload local `data/` during deploy when target catalog `_file_path` is configured. ## Documentation & Contributing To learn more about the plugin, please refer to the [documentation](https://kedro-databricks.readthedocs.io/). Interested in contributing? Check out our [contribution guidelines](docs/contributing.md) to get started! ## Breaking Changes ### Version `0.14.0` To accommodate using Databricks Free Edition, we had to change the structure of overrides defined in `conf/<env>/databricks.yml`. Before: ``` default: environments: - environment_key: default spec: environment_version: '4' dependencies: - ../dist/*.whl tasks: - task_key: default environment_key: default ``` After: ``` resources: jobs: default: environments: - environment_key: default spec: environment_version: '4' dependencies: - ../dist/*.whl tasks: - task_key: default environment_key: default ``` This was done so that we could default to creating a volume in a newly initialized `kedro-databricks` project. While this requires users to migrate their databricks configuration, it also extends the ability of `kedro-databricks` beyond that of applying overrides to specific jobs. Now, you can add any type of resource in your `conf/<env>/databricks.yml` and those will be generated as well. > NOTE: Merges are only applied for `jobs` currently, so any other defined will be generated as defined in the configuration. In addition to the changes to the structure of `conf/<env>/databricks.yml`, we now also tag the generated resources with their resource type and target environment, meaning that newly generated resources will be named like `target.<env>.<resource-type>.<resouce-name>.yml`.
