trino-storage - GitLite

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #deep-learning #linux #nodejs #macos #java #awesome-list #pytorch #open-source

trino-storage

(★ 121)

Storage connector for Trino

파일 탐색기

최종 버전 다운로드 (.zip)

- - main.yml
  - release.yml
- dependabot.yml
- problem-matcher.json
- - violations.xml
- - maven-wrapper.properties
- jvm.config
- docker-compose.yml
.gitignore
LICENSE
mvnw
pom.xml
README.md

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/ebyhr/trino-storage

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd trino-storage

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Docker

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Docker Desktop 컨테이너를 빌드하고 실행하려면 필요합니다. 설치 후 실행해서 백그라운드에 켜두세요.

docker compose -f docker/docker-compose.yml up -d --build

compose 설정 파일에 정의된 서비스들을 대상으로 명령을 실행합니다.

✅ 터미널에 docker compose ps 를 입력해 컨테이너들이 Up 상태인지 확인하세요. README에 포트 번호가 적혀있다면 브라우저에서 http://localhost:포트번호 로 접속해보세요.

3. Maven (Java)

보통

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
JDK (Java) Java 프로젝트를 빌드/실행하려면 필요합니다.
Maven mvn 명령어를 쓰기 위한 빌드 도구입니다.

mvn clean install

Maven으로 의존성 설치와 빌드를 진행합니다.

✅ BUILD SUCCESS 메시지가 뜨면 성공입니다. target/ 폴더에 결과물이 생성됩니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Trino Storage Connector [![Build Status](https://github.com/snowlift/trino-storage/workflows/CI/badge.svg)](https://github.com/snowlift/trino-storage/actions?query=workflow%3ACI+event%3Apush+branch%3Amaster)
This is a [Trino](http://trino.io/) connector to access single file (e.g. csv, tsv). Please keep in mind that this is not production ready and it was created for tests.

# Supported scheme
- hdfs
- s3a
- file
- http
- https

> Note: reading local files (with the `file` or no schema) can be disabled by setting `allow-local-files=false` in the catalog configuration.

# Query
You need to specify file type by schema name and use absolute path.
```sql
SELECT * FROM
storage.csv."file:///tmp/numbers-2.csv";

SELECT * FROM
storage.csv."https://raw.githubusercontent.com/snowlift/trino-storage/master/src/test/resources/example-data/numbers-2.csv";
```

Supported schemas are below.
- `tsv`
- `csv`
- `ssv`
- `txt`
- `raw`
- `excel`
- `orc`
- `json`

`csv` plugin extracts each line, with columns separated by `,`. Currently the first line is used as column names.
```sql
SELECT * FROM
storage.csv."https://raw.githubusercontent.com/snowlift/trino-storage/master/src/test/resources/example-data/numbers-2.csv";
```
```
  ten   | 10 
--------+----
 eleven | 11 
 twelve | 12
(2 rows)
```

Tab (`\t`) and semicolon (`;`) delimiters are also supported, using the `tsv` and `ssv` plugins, respectively.

`txt` plugin doesn't extract each line. Currently column name is always `value`.
```sql
SELECT * FROM
storage.txt."https://raw.githubusercontent.com/snowlift/trino-storage/master/src/test/resources/example-data/numbers.tsv";
``` 
```
 value  
--------
 one    1   
 two    2   
 three  3
(3 rows)
```

`raw` plugin doesn't extract each line. Currently column name is always `data`. This connector is similar to `txt` plugin. 
The main difference is `txt` plugin may return multiple rows, but `raw` plugin always return only one row.
```sql
SELECT * FROM
storage.raw."https://raw.githubusercontent.com/snowlift/trino-storage/master/src/test/resources/example-data/numbers.tsv";
``` 
```
  data  
--------
 one    1   
 two    2   
 three  3 
(1 row)
```

`excel` plugin currently read first sheet.
```sql
SELECT * FROM
storage.excel."https://raw.githubusercontent.com/snowlift/trino-storage/master/src/test/resources/example-data/sample.xlsx";
``` 
```
  data  
--------
 one    1   
 two    2   
 three  3 
(1 row)
```

# Table functions

The connector provides specific table functions to list directory status and read files.
```sql
SELECT * FROM TABLE(storage.system.list('/tmp/trino-storage'));
```
```
     file_modified_time      | size |            name
-----------------------------+------+-----------------------------
 2023-05-03 12:14:22.107 UTC |   12 | /tmp/trino-storage/test.txt
```

```sql
SELECT * FROM TABLE(storage.system.read_file('csv', '/tmp/trino-storage/test.txt'));
```
```
     col
-------------
 hello world
```

# Build
Run all the unit test classes.
```
./mvnw test
```

Build without running tests
```
./mvnw clean install -DskipTests
```

> Note: tests include intergration tests, that will run Minio and HDFS as Docker containers. They need to pull their images,
> which can take a while. If you see the tests getting stuck, try pulling these images before starting tests, to see the progress.
> Look for image names and versions in `TestingMinioServer` and `TestingHadoopServer` test classes.
> It is also required to set the `HADOOP_USER_NAME` environmental variable to `hive`.

# Deploy
Unarchive trino-storage-{version}.zip and copy jar files in target directory to use storage connector in your Trino cluster.