llm-twin-course
๐ค ๐๐ฒ๐ฎ๐ฟ๐ป for ๐ณ๐ฟ๐ฒ๐ฒ how to ๐ฏ๐๐ถ๐น๐ฑ an end-to-end ๐ฝ๐ฟ๐ผ๐ฑ๐๐ฐ๐๐ถ๐ผ๐ป-๐ฟ๐ฒ๐ฎ๐ฑ๐ ๐๐๐ & ๐ฅ๐๐ ๐๐๐๐๐ฒ๐บ using ๐๐๐ ๐ข๐ฝ๐ best practices: ~ ๐ด๐ฐ๐ถ๐ณ๐ค๐ฆ ๐ค๐ฐ๐ฅ๐ฆ + 12 ๐ฉ๐ข๐ฏ๐ฅ๐ด-๐ฐ๐ฏ ๐ญ๐ฆ๐ด๐ด๐ฐ๐ฏ๐ด
ํ์ผ ํ์๊ธฐ
์ต์ข ๋ฒ์ ๋ค์ด๋ก๋ (.zip)- .dockerignore
- Dockerfile.data_cdc
- Dockerfile.data_crawlers
- Dockerfile.feature_pipeline
- Dockerfile.feature_pipeline.superlinked
- clarify-concept.md
- questions-about-the-course-material.md
- technical-troubleshooting-or-bugs.md
- crawler.yml
- links.txt
- bytewax.png
- comet.png
- opik.svg
- qdrant.svg
- qwak.png
- superlinked.svg
- architecture.png
- cover.png
- fine-tuning-workflow.png
- llm_engineers_handbook_cover.png
- qdrant-example.png
- ui-example.png
- __init__.py
- stream_input.py
- stream_output.py
- __init__.py
- cleaning_data_handlers.py
- dispatchers.py
- splitters.py
- __init__.py
- chain.py
- prompt_templates.py
- __init__.py
- documents.py
- raw.py
- utils.py
- __init__.py
- query_expanison.py
- reranking.py
- retriever.py
- self_query.py
- bytewax_entrypoint.sh
- aws_credentials.json
- config.yaml
- gcp_credentials.json
- amazon_app.py
- app.py
- app_with_mongodb.py
- mongodb.md
- app_with_redis.py
- redis.md
- api.md
- app.md
- bucket.md
- dummy_app.py
- vector_databases.md
- vm.md
- __init__.py
- app_config.py
- __init__.py
- exception.py
- exception_handler.py
- __init__.py
- lifespan_event.py
- __init__.py
- management_router.py
- __init__.py
- data_loader.py
- file_handler_service.py
- file_object_serializer.py
- persistence_service.py
- supervisor_service.py
- __init__.py
- fast_api_handler.py
- open_api_description_util.py
- registry_loader.py
- __init__.py
- dependency_register.py
- main.py
- static_endpoint_descriptor.json
- __init__.py
- Dockerfile
- supervisord.conf
- __init__.py
- app_location_parser.py
- __init__.py
- poller_config.py
- __init__.py
- poller.py
- __init__.py
- gcs_resource_handler.py
- __init__.py
- local_resource_handler.py
- __init__.py
- s3_resource_handler.py
- __init__.py
- resource_handler.py
- resource_handler_factory.py
- __init__.py
- main.py
- __init__.py
- Dockerfile
- logging_config.ini
- poller_config.ini
- .python-version
- poetry.lock
- pyproject.toml
- app.py
- deploy.py
- init-venv.sh
- .python-version
- compose.yaml
- LICENSE
- NOTICE
- README.md
- __init__.py
- cleaning.py
- logging.py
- config.py
- local_test.py
- main.py
- mq.py
- README.md
- retriever.py
- singleton.py
- superlinked_client.py
- __init__.py
- create_execution_role.py
- create_sagemaker_role.py
- __init__.py
- documents.py
- mongo.py
- qdrant.py
- __init__.py
- prompt_templates.py
- query_expanison.py
- reranking.py
- retriever.py
- self_query.py
- __init__.py
- config.py
- errors.py
- lib.py
- logger_utils.py
- mq.py
- opik_utils.py
- cdc.py
- config.py
- test_cdc.py
- __init__.py
- base.py
- custom_article.py
- github.py
- linkedin.py
- medium.py
- config.py
- dispatcher.py
- main.py
- utils.py
- __init__.py
- stream_input.py
- stream_output.py
- __init__.py
- chunking_data_handlers.py
- cleaning_data_handlers.py
- dispatchers.py
- embedding_data_handlers.py
- __init__.py
- chunk_documents.py
- exceptions.py
- file_handler.py
- generate.py
- llm_communication.py
- __init__.py
- base.py
- chunk.py
- clean.py
- embedded_chunk.py
- raw.py
- bytewax_entrypoint.sh
- __init__.py
- chunking.py
- cleaning.py
- embeddings.py
- config.py
- main.py
- retriever.py
- __init__.py
- delete_sagemaker_endpoint.py
- deploy_sagemaker_endpoint.py
- __init__.py
- evaluate.py
- evaluate_monitoring.py
- evaluate_rag.py
- style.py
- config.py
- llm_twin.py
- main.py
- prompt_templates.py
- ui.py
- utils.py
- config.py
- download_dataset.py
- finetune.py
- requirements.txt
- run_on_sagemaker.py
- .env.example
- .gitignore
- .python-version
- docker-compose-superlinked.yml
- docker-compose.yml
- INSTALL_AND_USAGE.md
- LICENSE
- Makefile
- poetry.lock
- pyproject.toml
- README.md
# ์ค์น ๊ฐ์ด๋
1. ์ฝ๋ ๋ด๋ ค๋ฐ๊ธฐ
git clone https://github.com/decodingai-magazine/llm-twin-course
๊นํ๋ธ์์ ํ๋ก์ ํธ ์ฝ๋ ์ ์ฒด๋ฅผ ๋ด ์ปดํจํฐ๋ก ๋ด๋ ค๋ฐ์ต๋๋ค.
cd llm-twin-course
๋ฐฉ๊ธ ๋ด๋ ค๋ฐ์ ํ๋ก์ ํธ ํด๋ ์์ผ๋ก ์ด๋ํฉ๋๋ค.
2. Docker
์ฌ์ ์ถ์ฒ์ฌ์ ์ค๋น๋ฌผ
- Git GitHub์์ ํ๋ก์ ํธ ์ฝ๋๋ฅผ ๋ด๋ ค๋ฐ์ผ๋ ค๋ฉด ํ์ํฉ๋๋ค.
- Docker Desktop ์ปจํ ์ด๋๋ฅผ ๋น๋ํ๊ณ ์คํํ๋ ค๋ฉด ํ์ํฉ๋๋ค. ์ค์น ํ ์คํํด์ ๋ฐฑ๊ทธ๋ผ์ด๋์ ์ผ๋์ธ์.
docker compose up -d --build
์ ์๋ ๋ชจ๋ ์ปจํ ์ด๋(์๋ฒ, DB ๋ฑ)๋ฅผ ํ ๋ฒ์ ๋น๋ํ๊ณ ๋ฐฑ๊ทธ๋ผ์ด๋์์ ์คํํฉ๋๋ค.
ํฐ๋ฏธ๋์ docker compose ps ๋ฅผ ์
๋ ฅํด ์ปจํ
์ด๋๋ค์ด Up ์ํ์ธ์ง ํ์ธํ์ธ์. README์ ํฌํธ ๋ฒํธ๊ฐ ์ ํ์๋ค๋ฉด ๋ธ๋ผ์ฐ์ ์์ http://localhost:ํฌํธ๋ฒํธ ๋ก ์ ์ํด๋ณด์ธ์.
3. Python
์ฌ์์ฌ์ ์ค๋น๋ฌผ
pip install -r src/training_pipeline/requirements.txt
requirements.txt ๋ฑ์ ๋ช ์๋ ํ์ด์ฌ ๋ผ์ด๋ธ๋ฌ๋ฆฌ๋ฅผ ์ค์นํฉ๋๋ค.
python <์คํํ ํ์ผ๋ช
>.py # README์์ ์ ํํ ์คํ ํ์ผ๋ช
์ ํ์ธํ์ธ์
ํ์ด์ฌ ์คํฌ๋ฆฝํธ(๋๋ ๋ชจ๋)๋ฅผ ์คํํฉ๋๋ค.
์๋ฌ ๋ฉ์์ง ์์ด ์คํ๋๊ณ ํฐ๋ฏธ๋์ ์๋ด ๋ฌธ๊ตฌ๊ฐ ์ถ๋ ฅ๋๋ฉด ์ ์์
๋๋ค.
4. Make
๋ณดํต์ฌ์ ์ค๋น๋ฌผ
- Git GitHub์์ ํ๋ก์ ํธ ์ฝ๋๋ฅผ ๋ด๋ ค๋ฐ์ผ๋ ค๋ฉด ํ์ํฉ๋๋ค.
- Make Linux/macOS๋ ๋ณดํต ๊ธฐ๋ณธ ์ค์น๋์ด ์์ต๋๋ค. Windows๋ ๋ณ๋ ์ค์น(์: MSYS2, WSL)๊ฐ ํ์ํฉ๋๋ค.
make
์์ฑ๋ ๋น๋ ์ค์ ์ ๋ฐํ์ผ๋ก ์ค์ ์ปดํ์ผ์ ์งํํด ์คํ ํ์ผ์ ๋ง๋ญ๋๋ค.
์๋ฌ ์์ด ๋๋๋ฉด ์ฑ๊ณต์
๋๋ค. ์์ฑ๋ ์คํ ํ์ผ์ ์ง์ ์คํํด๋ณด์ธ์.
// repository documentation
Was this content helpful?
(0 ratings)