trino-on-kubernetes

KO | EN

gitlite — search

#javascript #llm #golang #android #machine-learning #rust #linux #windows #cli #kubernetes #vue #awesome-list #ios #open-source

trino-on-kubernetes

(★ 10)

No description available.

trino-on-kubernetes Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# trino-on-kubernetes

Trino is a federated query engine, the connector-based architecture makes it easy to integrate with other systems.

In this repo, we are using the [Kubernetes](https://kubernetes.io/) to deploy the Trino service and other systems.

- [prerequisites](#prerequisites)
- [setup](#setup)
  - [namespace](#namespace)
  - [postgresql](#postgresql)
  - [minio](#minio)
  - [hive-metastore](#hive-metastore)
    - [hive-metastore-postgresql](#hive-metastore-postgresql)
    - [hive-metastore](#hive-metastore-1)
  - [clickhouse](#clickhouse)
  - [redis](#redis)
  - [trino](#trino)
- [playground](#playground)
  - [blackhole connector](#blackhole-connector)
  - [postgresql connector](#postgresql-connector)
  - [hive connector](#hive-connector)
  - [clickhouse connector](#clickhouse-connector)
  - [redis connector](#redis-connector)
- [cleanup](#cleanup)
- [references](#references)

## prerequisites
- [Rancher Desktop](https://github.com/rancher-sandbox/rancher-desktop): `1.2.1`
- Kubernetes: `v1.22.6`
- kubectl `v1.23.3`
- Helm: `v3.7.2`

## setup

tl;dr: `bash scripts/up.sh`

### namespace

```sh
kubectl create namespace trino --dry-run=client -o yaml | kubectl apply -f -
```

### postgresql

follow the [bitnami postgresql chart](https://github.com/bitnami/charts/tree/master/bitnami/postgresql) to install postgresql

```sh
helm repo add bitnami https://charts.bitnami.com/bitnami
```

```sh
helm upgrade --install my-postgresql bitnami/postgresql -n trino -f postgresql/values.yaml
```

verify the installation

```sh
kubectl run my-postgresql-client --rm --tty -i --restart='Never' --namespace trino --image docker.io/bitnami/postgresql:14.1.0-debian-10-r80 --env="PGPASSWORD=trino_demo_password" -- psql --host my-postgresql -U trino_demo_user -d trino_demo -p 5432 -c "\dt"
```

### minio

follow the [bitnami minio chart](https://github.com/bitnami/charts/tree/master/bitnami/minio) to install minio

```sh
helm repo add bitnami https://charts.bitnami.com/bitnami
```

```sh
helm upgrade --install my-minio bitnami/minio -n trino -f minio/values.yaml
```

verify the installation

```sh
kubectl run --namespace trino my-minio-client \
     --rm --tty -i --restart='Never' \
     --env MINIO_SERVER_ROOT_USER=minio_access_key \
     --env MINIO_SERVER_ROOT_PASSWORD=minio_secret_key \
     --env MINIO_SERVER_HOST=my-minio \
     --image docker.io/bitnami/minio-client:2022.2.7-debian-10-r0 -- admin info minio
```

```sh
●  my-minio:9000
   Uptime: 7 minutes
   Version: 2022-02-07T08:17:33Z
   Network: 1/1 OK

0 B Used, 2 Buckets, 0 Objects
```

### hive-metastore

#### hive-metastore-postgresql

follow the [bitnami postgresql chart](https://github.com/bitnami/charts/tree/master/bitnami/postgresql) to install postgresql for hive-metastore

```sh
helm repo add bitnami https://charts.bitnami.com/bitnami
```

```sh
helm upgrade --install hive-metastore-postgresql bitnami/postgresql -n trino -f hive-metastore-postgresql/values.yaml
```

verify the installation

```sh
kubectl run hive-metastore-postgresql-client --rm --tty -i --restart='Never' --namespace trino --image docker.io/bitnami/postgresql:14.1.0-debian-10-r80 --env="PGPASSWORD=admin" -- psql --host hive-metastore-postgresql -U admin -d metastore_db -p 5432 -c "\du"
```

```sh
                                   List of roles
 Role name |                         Attributes                         | Member of
-----------+------------------------------------------------------------+-----------
 admin     | Create DB                                                  | {}
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
```

#### hive-metastore

using the helm chart (based on the [gradiant hive-metastore chart](https://github.com/Gradiant/bigdata-charts/tree/master/charts/hive-metastore)) to install hive-metastore

```sh
helm upgrade --install my-hive-metastore -n trino -f hive-metastore/values.yaml ./charts/hive-metastore
```

verify the installation

### clickhouse

follow the [altinity clickhouse operator](https://github.com/Altinity/clickhouse-operator) to install clickhouse

install the Custom Resource Definition(crd)

```sh
kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml
```

the following results are expected

```sh
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallations.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallationtemplates.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseoperatorconfigurations.clickhouse.altinity.com created
serviceaccount/clickhouse-operator created
clusterrole.rbac.authorization.k8s.io/clickhouse-operator-kube-system created
clusterrolebinding.rbac.authorization.k8s.io/clickhouse-operator-kube-system created
configmap/etc-clickhouse-operator-files created
configmap/etc-clickhouse-operator-confd-files created
configmap/etc-clickhouse-operator-configd-files created
configmap/etc-clickhouse-operator-templatesd-files created
configmap/etc-clickhouse-operator-usersd-files created
deployment.apps/clickhouse-operator created
service/clickhouse-operator-metrics created
```

verify the installation

```sh
kubectl get pods --namespace kube-system
```

```sh
NAME                                      READY   STATUS      RESTARTS        AGE
...
clickhouse-operator-78f59f855b-pqntt      2/2     Running     0               57s
...
```

install the clickhouse via the operator

```sh
kubectl apply -f clickhouse/ -n trino
```

verify the installation

```sh
kubectl exec chi-bootstrap-bootstrap-0-0-0 -n trino -- clickhouse-client -u analytics --password admin --query="SHOW DATABASES"
```

```sh
INFORMATION_SCHEMA
default
information_schema
psql
system
```

### redis

follow the [bitnami redis chart](https://github.com/bitnami/charts/tree/master/bitnami/redis) to install redis

```sh
helm repo add bitnami https://charts.bitnami.com/bitnami
```

```sh
helm upgrade --install my-redis bitnami/redis -n trino -f redis/values.yaml
```

verify the installation

```sh
kubectl exec -it svc/my-redis-master -n trino -- redis-cli -h my-redis-master -a trino_demo_password scan 0
```

```sh
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
1) "0"
2) (empty array)
```

create the redis table definition config-map

```sh
kubectl create secret generic redis-table-definition --from-file=redis/test.json -n trino
```

### trino

follow the [trino official chart](https://github.com/trinodb/charts/tree/main) to install trino

```sh
helm repo add trino https://trinodb.github.io/charts/
```

```sh
helm upgrade --install my-trino trino/trino --version 0.7.0 --namespace trino -f trino/values.yaml
```

verify the installation

```sh
kubectl exec -it deploy/my-trino-coordinator -n trino -- trino
```

## playground

all of the sql is running in the trino shell

```sh
kubectl exec -it deploy/my-trino-coordinator -n trino -- trino
```

```sh
SHOW CATALOGS;

Catalog   
------------
 blackhole  
 clickhouse 
 minio      
 postgresql 
 redis      
 system     
 tpcds      
 tpch       
(8 rows)
```

### blackhole connector

```sh
CREATE SCHEMA blackhole.test;
CREATE TABLE blackhole.test.orders AS SELECT * FROM tpch.tiny.orders;
INSERT INTO blackhole.test.orders SELECT * FROM tpch.sf3.orders;
```

### postgresql connector

```sh
SHOW SCHEMAS IN postgresql;

Schema
--------------------
 information_schema
 pg_catalog
 public
(3 rows)
```

Using the PostgreSQL connector, Trino is able to retrieve the data for processing, returning the results to the user

```sh
SELECT * FROM postgresql.public.users LIMIT 10;

### hive connector

FIXME: for some unknown timing issue, the minio connector cannot establish every time properly, deleting and recreating `hive-metastore-postgresql` can mitigate the issue

follow [Hive connector over MinIO file storage tutorial](https://github.com/bitsondatadev/trino-getting-started/tree/main/hive/trino-minio) to test the hive connector

```sh
SHOW SCHEMAS IN minio;
```

```sh
       Schema
--------------------
 default
 information_schema
```

```sh
CREATE SCHEMA minio.tiny
WITH (location = 's3a://tiny/');
```

```sh
CREATE TABLE minio.tiny.customer
WITH (
    format = 'ORC',
    external_location = 's3a://tiny/customer/'
) 
AS SELECT * FROM tpch.tiny.customer;
```

```sh
SELECT * FROM minio.tiny.customer LIMIT 50;
```

```sh
SHOW SCHEMAS IN minio;
```

```sh
       Schema
--------------------
 default
 information_schema
 tiny
(3 rows)
```

### clickhouse connector

create database and table

```sh
kubectl exec chi-bootstrap-bootstrap-0-0-0 -n trino -c clickhouse -- clickhouse-client -u analytics --password admin --query="CREATE DATABASE IF NOT EXISTS psql;"
```

```sh
kubectl exec chi-bootstrap-bootstrap-0-0-0 -n trino -c clickhouse -- clickhouse-client -u analytics --password admin --query="CREATE TABLE IF NOT EXISTS psql.users
(
  hash_firstname String,
  hash_lastname String,
  gender String
) ENGINE = TinyLog;"
```

```sh
SHOW SCHEMAS IN clickhouse;
```

```sh
       Schema
--------------------
 default
 information_schema
 psql
 system
(4 rows)
```

no data in the clickhouse

```sh
SELECT * FROM clickhouse.psql.users limit 10;
```

```sh
 hash_firstname | hash_lastname | gender
----------------+---------------+--------
(0 rows)
```

insert from postgresql.public.users

```sh
INSERT INTO clickhouse.psql.users
SELECT
  to_utf8(hash_firstname),
  to_utf8(hash_lastname),
  to_utf8(gender)
FROM postgresql.public.users;
```

```sh
SELECT
  from_utf8(hash_firstname) as hash_firstname,
  from_utf8(hash_lastname) as hash_lastname,
  from_utf8(gender) as gender
FROM clickhouse.psql.users limit 10;
```

### redis connector

```sh
SHOW SCHEMAS IN redis;
```

```sh
       Schema       
--------------------
 information_schema 
 trino              
(2 rows)
```

```sh
SHOW COLUMNS IN redis.default.test;
```

save some test data in the redis

```sh
kubectl exec -it svc/my-redis-master -n trino -- redis-cli -h my-redis-master -a trino_demo_password set test '{"number": 20}'
```

read from trino

```sh
SELECT * FROM redis.default.test LIMIT 10;
```

```sh
 number 
--------
     20 
(1 row)
```

## cleanup

tl;dr: `bash scripts/down.sh`

```sh
helm uninstall my-trino -n trino
helm uninstall my-redis -n trino
helm uninstall my-postgresql -n trino
helm uninstall my-hive-metastore -n trino
helm uninstall hive-metastore-postgresql -n trino
helm uninstall my-minio -n trino
kubectl delete -f clickhouse/ -n trino
kubectl delete -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml
kubectl delete pvc --all -n trino
kubectl delete namespace trino
```

## references
- [realtimedatalake/hive-metastore-docker: Containerized Apache Hive Metastore for horizontally scalable Hive Metastore deployments](https://github.com/realtimedatalake/hive-metastore-docker)
- [liangjingyang/hive-metastore](https://github.com/liangjingyang/hive-metastore/commit/112ce99241f5b68d9f45b775ce49e0550634c194)
- [Gradiant/bigdata-charts: charts/hive-metastore](https://github.com/Gradiant/bigdata-charts/tree/master/charts/hive-metastore)
- [bitsondatadev/trino-getting-started: Hive connector over MinIO file storage](https://github.com/bitsondatadev/trino-getting-started/tree/main/hive/trino-minio)
- [bitsondatadev/hive-metastore: entrypoint.sh](https://github.com/bitsondatadev/hive-metastore/blob/master/scripts/entrypoint.sh)

trino-on-kubernetes

trino-on-kubernetes Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

trino-on-kubernetes

trino-on-kubernetes Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README