okdp-sandbox

KO | EN

gitlite — search

#javascript #ai #machine-learning #rust #deep-learning #docker #java #awesome #awesome-list #pytorch #security #ios #openai #open-source

(★ 9)

Hands-on sandbox for deploying, testing, and exploring the OKDP (Open Kubernetes Data Platform) ecosystem.

okdp-sandbox 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![Flux](https://img.shields.io/badge/flux-latest-purple.svg)](https://fluxcd.io/)
[![KuboCD](https://img.shields.io/badge/kubocd-v0.2.1-green.svg)](https://github.com/kubocd/kubocd)&ensp;&ensp;
[![Kubernetes](https://img.shields.io/badge/kubernetes-1.28+-blue.svg)](https://kubernetes.io/)
[![Kind](https://img.shields.io/badge/kind-latest-orange.svg)](https://kind.sigs.k8s.io/)&ensp;&ensp;
[![License Apache2](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0)

OKDP Sandbox is a hands-on environment for deploying, testing, and exploring the [OKDP](https://okdp.io) ecosystem on a local Kubernetes cluster.

It provides a ready-to-use data platform environment covering identity, object storage, Spark processing, workflow orchestration, interactive notebooks, SQL querying, and visualization, running end to end on a local cluster. See [What is included](#what-is-included-in-the-sandbox) for the full component list.

## What is included in the sandbox?

The default OKDP sandbox deploys a complete local data-platform environment with:

- Keycloak for identity and access management
- SeaweedFS for object storage
- Spark Operator and Spark History Server for Spark workloads and monitoring
- Apache Airflow for workflow orchestration
- JupyterHub for interactive data-science workspaces
- Trino and Hive Metastore for SQL query and metadata services
- Apache Superset for dashboards and visual analytics
- OKDP Server and OKDP UI for platform management

## Repository scope

This repository owns the **single-cluster sandbox deployment**. It describes how to deploy the OKDP platform onto a local Kubernetes cluster and contains the deployment assets only:

- `clusters/sandbox/flux/` : Flux bootstrap of the KuboCD controller (`kubocd.yaml`)
- `clusters/sandbox/releases/` : KuboCD `Release` manifests (what gets installed, which package tag, which parameters)
- `clusters/sandbox/contexts/` : the layered KuboCD `Context` files (`10-platform`, `20-provider`, `30-service`, `99-examples`)
- `docs/` : deployment guides (DNS, certificates)

The **packages themselves** (the KuboCD packages bundled as OCI artifacts) live in dedicated repositories, split by ownership, OKDP core versus the third-party dependencies OKDP does not own and are consumed here from the registry. This repository never builds packages, it only deploys published ones.

| Concern | Owner |
|---|---|
| KuboCD packages for core services and control plane | [`OKDP/platform-packages`](https://github.com/OKDP/platform-packages) |
| KuboCD packages for third-party and bootstrap dependencies | [`OKDP/sandbox-dependencies`](https://github.com/OKDP/sandbox-dependencies) |
| Reusable utility Helm charts | [`OKDP/helm-charts-utilities`](https://github.com/OKDP/helm-charts-utilities) |
| Notebooks, DAGs, and runnable examples | [`OKDP/okdp-examples`](https://github.com/OKDP/okdp-examples) |
| OKDP web UI | [`OKDP/okdp-ui`](https://github.com/OKDP/okdp-ui) |
| OKDP backend server (API) | [`OKDP/okdp-server`](https://github.com/OKDP/okdp-server) |
| Single-cluster sandbox deployment (this repository) | `OKDP/okdp-sandbox` |

## Prerequisites

### System requirements

- Minimum: 16 GB RAM and 4 CPUs
- Docker or Podman allocation: at least 8 GB RAM and 2 CPUs

### Software dependencies

- [Docker](https://docs.docker.com/get-docker/) or a compatible container runtime
- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation)
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
- [Flux CLI v2.7.5](https://fluxcd.io/flux/installation/)

## Quick start

The sandbox deployment files live in this repository under `clusters/sandbox/`.

### 1. Clone this repository

```sh
git clone https://github.com/OKDP/okdp-sandbox.git
cd okdp-sandbox
```

### 2. Create Kubernetes Kind Cluster

Create a Kind cluster configuration file and deploy the cluster:

> ℹ️ [Kind](https://kind.sigs.k8s.io/) is a tool for running local Kubernetes clusters using Docker.  
> It’s ideal for **development**, **testing**, and **sandbox reproducible environments**.  
> Kind follows a **manifest-first** (infrastructure-as-code) approach, while **Minikube** is a **command-line-first** approach.

```sh
# Create cluster configuration
cat > /tmp/okdp-sandbox-config.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: okdp-sandbox
nodes:
- role: control-plane
 extraPortMappings:
 - containerPort: 30080
 hostPort: 80
 - containerPort: 30443
 hostPort: 443
 - containerPort: 30053
 hostPort: 30053
 protocol: UDP
EOF

# Create the cluster
kind create cluster --config /tmp/okdp-sandbox-config.yaml
```

<details>
<summary>PowerShell</summary>

```powershell
# Create cluster configuration
@"
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: okdp-sandbox
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30080
    hostPort: 80
  - containerPort: 30443
    hostPort: 443
  - containerPort: 30053
    hostPort: 53
    protocol: UDP
"@ | Out-File -FilePath "$env:TEMP\okdp-sandbox-config.yaml" -Encoding UTF8

# Create the cluster
kind create cluster --config "$env:TEMP\okdp-sandbox-config.yaml"
```

</details>

### 3. Install Platform Components
#### Install Flux (GitOps engine)

> ℹ️ **Note**  
> This step is only required for a fresh installation. It is **not required for upgrades** if Flux is already installed and running.  
> For upgrades, jump to [Deploy/Upgrade OKDP platform components](#deployupgrade-okdp-platform-components).

> ℹ️ **[Flux](https://fluxcd.io/flux/concepts/)** is the GitOps controller that continuously reconciles your cluster state with what’s defined in Git.  
> The following command installs all Flux core components:
> - **source-controller**: fetches sources such as Git repositories and Helm charts  
> - **kustomize-controller**: applies Kubernetes manifests using Kustomize  
> - **helm-controller**: manages Helm releases declaratively  
> - **notification-controller**: handles alerts and automation triggers  
>
> 💡 In this setup, Flux controllers manage resources locally and are **not connected to a Git repository**.  
> Manifests are applied manually with `kubectl`, so **no Git access is required**.

```sh
flux install
```

#### Install metrics-server

```sh
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl patch deployment metrics-server -n kube-system --type=json \
  -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}]'
```

Verify it's working:

```sh
kubectl top nodes
```

#### Configure proxy settings for Flux controllers (Optional)

If your environment requires a proxy to reach external sources (container registries), the following command sets the proxy configuration variables to all Flux controllers (source, kustomize, helm, notification):

```sh
[ -n "${https_proxy}${HTTPS_PROXY}" ] && kubectl -n flux-system set env deploy -l app.kubernetes.io/part-of=flux \
        HTTPS_PROXY="${HTTPS_PROXY:-${https_proxy}}" \
        HTTP_PROXY="${HTTP_PROXY:-${http_proxy}}" \
        NO_PROXY="${NO_PROXY:-${no_proxy}}"
```

<details>
<summary>PowerShell</summary>

```powershell
if ($env:HTTPS_PROXY -or $env:https_proxy) {
    kubectl -n flux-system set env deploy -l app.kubernetes.io/part-of=flux `
        HTTPS_PROXY=($env:HTTPS_PROXY ?? $env:https_proxy) `
        HTTP_PROXY=($env:HTTP_PROXY ?? $env:http_proxy) `
        NO_PROXY=($env:NO_PROXY ?? $env:no_proxy)
}
```

</details>

Verify the proxy environment variables are correctly set for all Flux controllers:

> 💡 You may see the same variable (e.g., `HTTPS_PROXY`) repeated multiple times,
> one for each controller (**source**, **kustomize**, **helm**, **notification**).  
> This is expected and confirms that the variables were applied consistently.

```sh
kubectl -n flux-system set env deploy -l app.kubernetes.io/part-of=flux --list \
                                         | grep PROXY
```

> 💡 **Use the following command if you want to remove the proxy configuration from Flux controllers:**  
> After removing the proxy, Flux will **no longer be able to pull images or manifests from external registries** that require proxy access.
>
> ```sh
> kubectl -n flux-system set env deploy -l app.kubernetes.io/part-of=flux \
>    HTTPS_PROXY- \
>    NO_PROXY-
> ```

#### Wait for Flux controllers to be ready

Ensures all Flux controllers (source-controller, kustomize-controller, helm-controller, notification-controller) are fully running before proceeding to the next step:

```sh
kubectl wait --for=condition=ready pod -l app=source-controller -n flux-system --timeout=300s
```

#### Install KuboCD (Flux extension)

> ℹ️ [KuboCD](https://www.kubocd.io/) is the continuous delivery layer built on top of **Flux**.  
> It manages platform components and applications **declaratively**, providing a higher-level CD abstraction for GitOps workflows.

```sh
kubectl apply -f clusters/sandbox/flux/kubocd.yaml
```

#### Deploy/Upgrade OKDP platform components

> ℹ️ **Note**  
> To upgrade the OKDP platform components, run:
>
> ```bash
> kubectl delete $(kubectl get release -n kubocd-system -o name) -n kubocd-system
> ```
>
> This will delete all KuboCD `Release` resources in the `kubocd-system` namespace.
>
> During upgrade command, you may see errors like:
>
> ```
> Error from server (Forbidden): admission webhook "vrelease-v1alpha1.kb.io" denied the request: release cert-manager is protected
> Error from server (Forbidden): admission webhook "vrelease-v1alpha1.kb.io" denied the request: release kubocd-webhooks is protected
> ```
>
> These errors can be safely ignored. The affected releases are **system-protected components** managed by the platform and have a **separate upgrade lifecycle**.
>
> Pull the latest updates locally before starting the upgrade.
> 
> ```bash
> git pull --rebase
> ```
>

Deploy/Upgrade the sandbox default context:

> 💡 **The KuboCD Context** is a centralized, reusable, declarative and environment-aware configuration layer that provides user defined shared parameters (ingress suffixes, storage classes, certificate issuers, catalogs, and authentication settings, etc) to all the components, ensuring consistent deployment.
>
> During deployment, KuboCD automatically resolves and injects these context variables into the target Kubernetes components across the cluster (cluster-wide), ensuring that every component is deployed with a consistent configuration.
>
> During a Context update, changes are automatically propagated only to the affected components, which are then reconciled to align with the desired configuration.
> 
> For example, the **Context** enables defining **different configurations for different environments**:
> - `sandbox` for experimentation
> - `dev` for internal testing  
> - `prod` for stable production environments 
> - `org` (or `global`) for the organization-wide configuration that provides defaults to other environments.
>
> Each environment can **define, override or extend** one or more contexts while preserving a unified, declarative deployment model.

```sh
kubectl apply -f clusters/sandbox/contexts/10-platform-context.yaml
kubectl apply -f clusters/sandbox/contexts/20-provider-context.yaml
kubectl apply -f clusters/sandbox/contexts/30-service-context.yaml
kubectl apply -f clusters/sandbox/contexts/99-examples-context.yaml
```

> 💡 By default, the **default Context** uses **okdp.sandbox** as the ingress domain suffix. 
> This domain may be blocked if it does not comply with your organization’s allowed domain policy. 
>
> Use the following command to update the domain suffix to match your organization’s domain (replace **<CUSTOM_DOMAIN>** with your actual domain name):
>
> ```sh
> kubectl -n kubocd-system patch context platform \
> -p '{"spec":{"context":{"platform":{"ingress":{"suffix":"<CUSTOM_DOMAIN>"}}}}}' \
> --type=merge
> ```

Configure proxy settings for OKDP Services (Optional)

If your environment requires a proxy to reach external datasets (Superset examples, okdp examples, quay.io KuboCD packages), the following command sets the proxy configuration variables to the required OKDP services:

```sh
kubectl -n kubocd-system patch context platform --type merge -p "$(cat <<EOF
spec:
 context:
 platform:
 proxy:
 httpProxy: "${HTTP_PROXY:-${http_proxy}}"
 httpsProxy: "${HTTPS_PROXY:-${https_proxy}}"
 noProxy: "${NO_PROXY:-${no_proxy}}"
EOF
)"
```

<details>
<summary>PowerShell</summary>

```powershell
kubectl -n kubocd-system patch context platform --type merge -p @"
spec:
 context:
 platform:
 proxy:
 httpProxy: "$($env:HTTP_PROXY ?? $env:http_proxy)"
 httpsProxy: "$($env:HTTPS_PROXY ?? $env:https_proxy)"
 noProxy: "$($env:NO_PROXY ?? $env:no_proxy)"
"@
```
</details>

Deploy/Upgrade OKDP components:

```sh
kubectl apply -f clusters/sandbox/releases/
```

#### Verify and monitor release deployment status

Watch releases as they are deployed until all the components become ready.

```sh
kubectl get releases -A --watch
# Wait until all releases show STATUS=READY (press Ctrl+C to exit watch)
# Alternative: kubectl wait --for=condition=ready release --all --all-namespaces --timeout=600s
```

### 4. DNS Setup

Enable access to OKDP services through DNS resolution for the `okdp.sandbox` or your custom domain `<CUSTOM_DOMAIN>`:

- **Option 1 (Recommended)**: Local DNS server configuration (recommended, automatic for all services)
- **Option 2**: Manual `/etc/hosts` configuration (simple but requires manual updates)

📋 **See [dns-configuration.md](docs/dns-configuration.md) for detailed setup instructions for your operating system.**

### 5. SSL Certificate

For HTTPS access without warnings, two options:

**Option 1**: Install the CA certificate

The sandbox uses a local certificate authority.

To avoid browser warnings, export the generated CA certificate and import it into your system or browser trust store:

```sh
kubectl get secret default-issuer -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d > okdp-sandbox-ca.crt
```

<details>
<summary>PowerShell</summary>

```powershell
# Import okdp-sandbox-ca.crt into your system's or browser's certificate store
kubectl get secret default-issuer -n cert-manager -o jsonpath='{.data.ca\.crt}' | ForEach-Object { [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($_)) } | Out-File -FilePath "okdp-sandbox-ca.crt" -Encoding ASCII
```

</details>

**Option 2**: Ignore certificate warnings
- **First, connect to Keycloak** (https://keycloak.okdp.sandbox or https://keycloak.<CUSTOM_DOMAIN>) and accept the self-signed certificate in your browser.
- This step is **mandatory** for all OKDP services (UI, Seaweedfs, etc.) to communicate properly with Keycloak.

## Quick Start Guide

1. **Access OKDP UI**: https://okdp-ui.okdp.sandbox or https://okdp-ui.<CUSTOM_DOMAIN>
2. **Login credentials**: Default authentication via Keycloak (login/password: adm/adm)
3. **Run the examples**: https://github.com/OKDP/okdp-examples

## Cleanup

```bash
kind delete cluster --name okdp-sandbox
rm /tmp/okdp-sandbox-config.yaml
```

<details>
<summary>PowerShell</summary>

```powershell
kind delete cluster --name okdp-sandbox
Remove-Item "$env:TEMP\okdp-sandbox-config.yaml" -Force
```

</details>

## License

This project is licensed under the [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0).

---

**Built 🚀 for the OKDP Community**
<a href="https://okdp.io">
 <img src="https://okdp.io/logos/okdp-notext.svg" height="20px" style="margin: 0 2px;" />
</a>

okdp-sandbox

okdp-sandbox 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

okdp-sandbox

okdp-sandbox 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README