setl
A simple Spark-powered ETL framework that just works ๐บ
ํ์ผ ํ์๊ธฐ
์ต์ข ๋ฒ์ ๋ค์ด๋ก๋ (.zip)- bug_report.md
- feature_request.md
- release.yml
- snapshot.yml
- test.yml
- dependabot.yml
- stale.yml
- change-scala-version.sh
- deploy-release.sh
- deploy-snapshot.sh
- docker-compose.yml
- test.sh
- configuration_example.md
- Connector.md
- ConnectorBuilder.md
- CustomConnector.md
- Repository.md
- SparkRepositoryAdapter.md
- SparkRepositoryBuilder.md
- Structured-Streaming-Connector.md
- logo_setl.png
- logo_setl_1280_640.png
- logo_setl.png
- logo_setl_1280_640.png
- Compressor_Archiver.md
- _config.yml
- Annotations.md
- Architecture.md
- Condition.md
- Conf.md
- ConfigLoader.md
- Deliverable.md
- Factory.md
- index.md
- Logging.md
- Pipeline.md
- Quick-Start.md
- SchemaConverter.md
- Setl.md
- SparkRepository-caching.md
- SparkSessionBuilder.md
- Stage.md
- StructAnalyser.md
- Transformer.md
- vocabulary.md
- Benchmark.java
- Compress.java
- Delivery.java
- Experimental.java
- InterfaceStability.java
- PathFormat.java
- Storage.java
- ValueType.java
- AlreadyExistsException.java
- BaseException.java
- ConfException.java
- ConnectorException.java
- InvalidConnectorException.java
- InvalidDeliveryException.java
- InvalidSchemaException.java
- RepositoryException.java
- UnknownException.java
- BenchmarkInvocationHandler.java
- GZIPCompressor.java
- SnappyCompressor.java
- XZCompressor.java
- ColumnName.scala
- CompoundKey.scala
- Conf.scala
- ConfigLoader.scala
- ConnectorConf.scala
- DeltaConnectorConf.scala
- DynamoDBConnectorConf.scala
- FileConnectorConf.scala
- HudiConnectorConf.scala
- JDBCConnectorConf.scala
- StructuredStreamingConnectorConf.scala
- CanCreate.scala
- CanDelete.scala
- CanDrop.scala
- CanPartition.scala
- CanUpdate.scala
- CanVacuum.scala
- CanWait.scala
- Configurable.scala
- HasBenchmark.scala
- HasDescription.scala
- HasDiagram.scala
- HasReader.scala
- HasReaderWriter.scala
- HasRegistry.scala
- HasType.scala
- HasWriter.scala
- Identifiable.scala
- Logging.scala
- SchemaConverter.scala
- StructAnalyser.scala
- Writable.scala
- ACIDConnector.scala
- CassandraConnector.scala
- Connector.scala
- ConnectorInterface.scala
- CSVConnector.scala
- DBConnector.scala
- DeltaConnector.scala
- DynamoDBConnector.scala
- ExcelConnector.scala
- FileConnector.scala
- HudiConnector.scala
- JDBCConnector.scala
- JSONConnector.scala
- ParquetConnector.scala
- SparkSQLConnector.scala
- StreamingConnector.scala
- StructuredStreamingConnector.scala
- ImplicitRepositoryAdapter.scala
- Repository.scala
- RepositoryAdapter.scala
- SparkRepository.scala
- Archiver.scala
- Compressor.scala
- Condition.scala
- ConnectorBuilder.scala
- DatasetConverter.scala
- SparkRepositoryBuilder.scala
- ZipArchiver.scala
- AbstractFactory.scala
- Deliverable.scala
- Factory.scala
- FactoryDeliveryMetadata.scala
- FactoryInput.scala
- FactoryOutput.scala
- MLTransformer.scala
- Transformer.scala
- DateUtils.scala
- ExpectedDeliverable.scala
- FilterImplicits.scala
- HasSparkSession.scala
- MermaidUtils.scala
- ReflectUtils.scala
- SparkUtils.scala
- TypesafeConfigUtils.scala
- DAG.scala
- DeliverableDispatcher.scala
- External.scala
- Flow.scala
- Node.scala
- Pipeline.scala
- PipelineInspector.scala
- PipelineOptimizer.scala
- SimplePipelineOptimizer.scala
- Stage.scala
- BenchmarkResult.scala
- Builder.scala
- Converter.scala
- Setl.scala
- SparkSessionBuilder.scala
- text.txt
- input2.csv
- streaming.conf
- file1-1-1.csv
- file1-2-1.csv
- file1-2-2.csv
- file2-1.csv
- test-input-file.txt
- file1-1-1.csv
- wrongfile1-1-1.csv
- file1-2-1.csv
- file1-2-2.csv
- file2-1.csv
- file1.csv
- file1-1-1.csv
- file1-2-1.csv
- file1-2-2.csv
- file2-1.csv
- application.conf
- dynamodb.conf
- local.conf
- log4j.properties
- myconf.conf
- test-json.json
- test_base_path.csv
- test_connector_builder.conf
- test_priority.conf
- test_schema_converter.csv
- ConfLoaderSuite.scala
- ConfSuite.scala
- DeltaConnectorConfSuite.scala
- DynamoDBConnectorConfSuite.scala
- FileConnectorConfSuite.scala
- HudiConnectorConfSuite.scala
- JDBCConnectorConfSuite.scala
- Properties.scala
- PropertiesSuite.scala
- StructuredStreamingConnectorConfSuite.scala
- FactoryDeliveryMetadataSuite.scala
- BenchmarkInvocationHandlerSuite.scala
- HasRegistrySuite.scala
- SchemaConverterSuite.scala
- StructAnalyserSuite.scala
- TestClasses.scala
- CassandraConnectorSuite.scala
- ConnectorSuite.scala
- CSVConnectorSuite.scala
- DeltaConnectorSuite.scala
- DynamoDBConnectorSuite.scala
- ExcelConnectorSuite.scala
- FileConnectorSuite.scala
- HudiConnectorSuite.scala
- JDBCConnectorSuite.scala
- JSONConnectorSuite.scala
- ParquetConnectorSuite.scala
- SparkSQLConnectorSuite.scala
- StructuredStreamingConnectorSuite.scala
- StreamingRepositorySuite.scala
- package.scala
- RepositoryAdapterSuite.scala
- SparkRepositorySuite.scala
- ConditionSuite.scala
- ConnectorBuilderSuite.scala
- GZIPCompressorSuite.scala
- package.scala
- SnappyCompressorSuite.scala
- SparkRepositoryBuilderSuite.scala
- XZCompressorSuite.scala
- ZipArchiverSuite.scala
- DeliverableSuite.scala
- DateUtilsSuite.scala
- FilterImplicitsSuite.scala
- IOUtils.scala
- MermaidUtilsSuite.scala
- TypesafeConfigUtilsSuite.scala
- DeliverableDispatcherSuite.scala
- FlowSuite.scala
- NodeSuite.scala
- package.scala
- PipelineInspectorSuite.scala
- PipelineSuite.scala
- SimplePipelineOptimizerSuite.scala
- StageSuite.scala
- MockCassandra.scala
- SetlSuite.scala
- SparkSessionBuilderSuite.scala
- SparkTestUtils.scala
- TestObject.scala
- .gitignore
- CHANGELOG.md
- CODE_OF_CONDUCT.md
- CONTRIBUTING.md
- LICENSE
- pom.xml
- README.md
# ์ค์น ๊ฐ์ด๋
1. ์ฝ๋ ๋ด๋ ค๋ฐ๊ธฐ
git clone https://github.com/SETL-Framework/setl
๊นํ๋ธ์์ ํ๋ก์ ํธ ์ฝ๋ ์ ์ฒด๋ฅผ ๋ด ์ปดํจํฐ๋ก ๋ด๋ ค๋ฐ์ต๋๋ค.
cd setl
๋ฐฉ๊ธ ๋ด๋ ค๋ฐ์ ํ๋ก์ ํธ ํด๋ ์์ผ๋ก ์ด๋ํฉ๋๋ค.
2. Docker
์ฌ์ ์ถ์ฒ์ฌ์ ์ค๋น๋ฌผ
- Git GitHub์์ ํ๋ก์ ํธ ์ฝ๋๋ฅผ ๋ด๋ ค๋ฐ์ผ๋ ค๋ฉด ํ์ํฉ๋๋ค.
- Docker Desktop ์ปจํ ์ด๋๋ฅผ ๋น๋ํ๊ณ ์คํํ๋ ค๋ฉด ํ์ํฉ๋๋ค. ์ค์น ํ ์คํํด์ ๋ฐฑ๊ทธ๋ผ์ด๋์ ์ผ๋์ธ์.
docker compose -f dev/docker-compose.yml up -d --build
compose ์ค์ ํ์ผ์ ์ ์๋ ์๋น์ค๋ค์ ๋์์ผ๋ก ๋ช ๋ น์ ์คํํฉ๋๋ค.
ํฐ๋ฏธ๋์ docker compose ps ๋ฅผ ์
๋ ฅํด ์ปจํ
์ด๋๋ค์ด Up ์ํ์ธ์ง ํ์ธํ์ธ์. README์ ํฌํธ ๋ฒํธ๊ฐ ์ ํ์๋ค๋ฉด ๋ธ๋ผ์ฐ์ ์์ http://localhost:ํฌํธ๋ฒํธ ๋ก ์ ์ํด๋ณด์ธ์.
3. Maven (Java)
๋ณดํต์ฌ์ ์ค๋น๋ฌผ
- Git GitHub์์ ํ๋ก์ ํธ ์ฝ๋๋ฅผ ๋ด๋ ค๋ฐ์ผ๋ ค๋ฉด ํ์ํฉ๋๋ค.
- JDK (Java) Java ํ๋ก์ ํธ๋ฅผ ๋น๋/์คํํ๋ ค๋ฉด ํ์ํฉ๋๋ค.
- Maven mvn ๋ช ๋ น์ด๋ฅผ ์ฐ๊ธฐ ์ํ ๋น๋ ๋๊ตฌ์ ๋๋ค.
mvn clean install
Maven์ผ๋ก ์์กด์ฑ ์ค์น์ ๋น๋๋ฅผ ์งํํฉ๋๋ค.
BUILD SUCCESS ๋ฉ์์ง๊ฐ ๋จ๋ฉด ์ฑ๊ณต์
๋๋ค. target/ ํด๋์ ๊ฒฐ๊ณผ๋ฌผ์ด ์์ฑ๋ฉ๋๋ค.
// repository documentation
Was this content helpful?
(0 ratings)