s3-tables-catalog - GitLite

KO | EN

gitlite — search

#react #typescript #llm #golang #android #machine-learning #nodejs #java #windows #awesome #cli #awesome-list #mcp #open-source

s3-tables-catalog

(★ 154)

The Amazon S3 Tables catalog is a client library that bridges control plane operations provided by S3 Tables to engines like Apache Spark, Flink and others, when used with the Iceberg Table format

파일 탐색기

최종 버전 다운로드 (.zip)

- - gradle-publish.yml
  - gradle.yml
- - gradle-wrapper.jar
  - gradle-wrapper.properties
- libs.versions.toml
- - THIRD-PARTY
- build.gradle
.gitignore
build.gradle
CODE_OF_CONDUCT.md
CONTRIBUTING.md
gradlew
gradlew.bat
LICENSE
NOTICE
README.md
settings.gradle

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/awslabs/s3-tables-catalog

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd s3-tables-catalog

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Gradle (Java/Kotlin)

보통 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
JDK (Java) Java/Kotlin 프로젝트를 빌드/실행하려면 필요합니다.
Gradle 레포에 포함된 gradlew(Gradle Wrapper)를 쓰면 Gradle을 따로 설치할 필요가 없습니다.

./gradlew build

Gradle로 빌드를 진행합니다.

✅ BUILD SUCCESSFUL 메시지가 뜨면 성공입니다. build/ 폴더에 결과물이 생성됩니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Amazon S3 Tables Catalog for Apache Iceberg

The Amazon S3 Tables Catalog for Apache Iceberg is an open-source library that bridges [S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html) operations to engines like [Apache Spark](https://spark.apache.org/), when used with the [Apache Iceberg](https://iceberg.apache.org/) Open Table Format.

This library can: 
* Translate [Apache Iceberg](https://iceberg.apache.org/) operations such as table discovery, metadata reads, and updates
* Add and removes tables in Amazon S3 Tables

### What are Amazon S3 Tables and table buckets ?

Amazon S3 Tables are built for storing tabular data, such as daily purchase transactions, streaming sensor data, or ad impressions. Tabular data represents data in columns and rows, like in a database table. Tabular data is most commonly stored in the [Apache Parquet](https://parquet.apache.org/) format.

The tabular data in Amazon S3 Tables is stored in a new S3 bucket type: a **table bucket**, which stores tables as subresources. S3 Tables has built-in support for tables in the [Apache Iceberg](https://iceberg.apache.org/) format. Using standard SQL statements, you can query your tables with query engines that support Apache Iceberg, such as [Amazon Athena](https://aws.amazon.com/athena/), [Amazon Redshift](https://aws.amazon.com/pm/redshift/), and [Apache Spark](https://spark.apache.org/).

## Current Status

Amazon S3 Tables Catalog for Apache Iceberg is generally available. We're always interested in feedback on features, performance, and compatibility. Please send feedback by opening a [GitHub issue](https://github.com/awslabs/s3-tables-catalog/issues/new/).

If you discover a potential security issue in this project we ask that you notify Amazon Web Services (AWS) Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do not create a public GitHub issue.

## Getting Started

To get started with Amazon S3 Tables, see [Tutorial: Getting started with S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-getting-started.html) in the *Amazon S3 User Guide*.

### Configuration

- <catalog_name> is your Iceberg Spark session catalog name. Replace it with the name of
your catalog, and remember to change the references throughout all configurations that
are associated with this catalog. In your code, you should then refer to your Iceberg tables
with the fully qualified table name, including the Spark session catalog name, as follows:
<catalog_name>.<database_name>.<table_name>.

- <catalog_name>.warehouse points to the Amazon S3 Tables path
- <catalog_name>.catalog-impl = "software.amazon.s3tables.iceberg.S3TablesCatalog" This key is required to point to an
implementation class for any custom catalog implementation.

### Java Spark app Example

Add the lines below to your pom.xml:
```
<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3tables</artifactId>
    <version>2.29.26</version>
</dependency>
<dependency>
    <groupId>software.amazon.s3tables</groupId>
    <artifactId>s3-tables-catalog-for-iceberg</artifactId>
    <version>0.1.8</version>
</dependency>
```
Or if you using a [BOM](https://aws.amazon.com/blogs/developer/managing-dependencies-with-aws-sdk-for-java-bill-of-materials-module-bom/) just add a dependency on the s3 tables sdk:
```
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>bom</artifactId>
            <version>2.29.26</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
```

Or for Gradle:

```
dependencies {
    implementation 'software.amazon.awssdk:s3tables:2.29.26'
    implementation 'software.amazon.s3tables:s3-tables-catalog-for-iceberg:0.1.8'
}
```

And finally start a spark session:

```
spark = SparkSession.builder()
            .config("spark.sql.catalog.<catalog_name>", "org.apache.iceberg.spark.SparkCatalog")
            .config("spark.sql.catalog.<catalog_name>.catalog-impl","software.amazon.s3tables.iceberg.S3TablesCatalog")
            .config("spark.sql.catalog.<catalog_name>.warehouse", <TABLE_BUCKET_ARN>)
            .getOrCreate();
```

## Contributions

We welcome contributions to Amazon S3 Tables Catalog for Apache Iceberg! Please see the [contributing guidelines](CONTRIBUTING.md) for more information on how to report bugs, build from source code, or submit pull requests.

## Security

If you discover a potential security issue in this project we ask that you notify Amazon Web Services (AWS) Security via our [vulnerability reporting](http://aws.amazon.com/security/vulnerability-reporting/) page. Please do not create a public GitHub issue.

## License

This project is licensed under the [Apache-2.0 License](LICENSE).