grain
Library for reading and processing ML training data.
파일 탐색기
최종 버전 다운로드 (.zip)- build_and_publish_template.yml
- preview_docs.yml
- publish_nightly.yml
- publish_release.yml
- tests.yml
- samplers.md
- transformations.md
- protocol.rst
- data_flow_multiple_workers.png
- data_flow_zero_workers.png
- grain_pipeline.svg
- arrayrecord_data_source_tutorial.ipynb
- arrayrecord_data_source_tutorial.md
- bagz_data_source_tutorial.ipynb
- bagz_data_source_tutorial.md
- huggingface_dataset_tutorial.ipynb
- huggingface_dataset_tutorial.md
- load_from_gcs_tutorial.ipynb
- load_from_gcs_tutorial.md
- load_from_s3_tutorial.ipynb
- load_from_s3_tutorial.md
- parquet_dataset_tutorial.ipynb
- parquet_dataset_tutorial.md
- pytorch_dataset_tutorial.ipynb
- pytorch_dataset_tutorial.md
- data_loader_tutorial.ipynb
- data_loader_tutorial.md
- dataset_advanced_tutorial.ipynb
- dataset_advanced_tutorial.md
- dataset_basic_tutorial.ipynb
- dataset_basic_tutorial.md
- dataset_debugging_tutorial.ipynb
- dataset_debugging_tutorial.md
- jax_training_tutorial.ipynb
- jax_training_tutorial.md
- performance_debugging.ipynb
- api_choice.md
- behind_the_scenes.md
- changelog.md
- conf.py
- CONTRIBUTING.md
- grain.checkpoint.rst
- grain.constants.rst
- grain.data_loader.rst
- grain.dataset.rst
- grain.experimental.rst
- grain.multiprocessing.rst
- grain.rst
- grain.samplers.rst
- grain.sharding.rst
- grain.sources.rst
- grain.transforms.rst
- index.md
- installation.md
- README.md
- requirements.txt
- BUILD
- config.py
- constants.py
- exceptions.py
- grain_random.py
- jax_import_smoke_test.py
- monitoring.py
- monitoring_base.py
- monitoring_test.py
- multiprocessing_test.py
- parallel.py
- parallel_test.py
- profiler.py
- profiler_test.py
- prometheus_monitoring.py
- prometheus_monitoring_test.py
- py_version_test.py
- pytest.py
- sharding.py
- sharding_test.py
- tf_import_smoke_test.py
- traceback_util.py
- traceback_util_test.py
- transforms.py
- transforms_test.py
- tree_lib.py
- tree_lib_jax_test.py
- tree_lib_test.py
- version.py
- base.py
- BUILD
- elastic_checkpoint.py
- elastic_checkpoint_test.py
- handler.py
- BUILD
- hf_source.py
- hf_source_test.py
- parquet_dataset.py
- parquet_dataset_test.py
- tfrecord_dataset.py
- tfrecord_dataset_test.py
- autotune.py
- autotune_test.py
- batch.py
- batch_test.py
- BUILD
- cache.py
- cache_test.py
- filter.py
- filter_test.py
- flatmap.py
- flatmap_test.py
- interleave.py
- interleave_test.py
- limit.py
- limit_test.py
- map.py
- map_test.py
- mix.py
- mix_test.py
- packing.py
- packing_concat_then_split.py
- packing_concat_then_split_test.py
- packing_packed_batch.py
- packing_test.py
- prefetch.py
- prefetch_autotune.py
- prefetch_autotune_test.py
- prefetch_test.py
- process_prefetch.py
- process_prefetch_test.py
- rebatch.py
- rebatch_test.py
- repeat.py
- repeat_test.py
- shuffle.py
- shuffle_test.py
- slice.py
- slice_test.py
- source.py
- source_test.py
- testing_util.py
- zip.py
- zip_test.py
- base.py
- base_test.py
- BUILD
- dataset.py
- dataset_test.py
- elastic_iterator.py
- elastic_iterator_test.py
- stats.py
- stats_test.py
- stats_utils.py
- stats_utils_test.py
- BUILD
- device_put.py
- device_put_test.py
- BUILD
- pack_and_batch_test.py
- packing.py
- BUILD
- index_shuffle_module.cc
- index_shuffle_test.py
- BUILD
- index_shuffle.cc
- index_shuffle.h
- BUILD
- queue.py
- queue_test.py
- shared_memory_array.py
- shared_memory_array_test.py
- variable_size_queue.py
- variable_size_queue_test.py
- BUILD
- digits.array_record-00000-of-00002
- digits.array_record-00001-of-00002
- morris_sequence_first_5.tfrecord
- __init__.py
- BUILD
- experimental.py
- BUILD
- data_loader.py
- data_loader_test.py
- data_sources.py
- data_sources_test.py
- grain_logging.py
- grain_logging_test.py
- load.py
- load_test.py
- operations.py
- operations_test.py
- options.py
- options_test.py
- record.py
- record_test.py
- samplers.py
- samplers_test.py
- __init__.py
- build_whl.sh
- Dockerfile
- runner_common.sh
- BUILD
- execution_summary.proto
- __init__.py
- experimental.py
- __init__.py
- BUILD
- checkpoint.py
- constants.py
- experimental.py
- multiprocessing.py
- samplers.py
- sharding.py
- sources.py
- transforms.py
- .gitignore
- .pre-commit-config.yaml
- .readthedocs.yaml
- BUILD
- CHANGELOG.md
- CONTRIBUTING.md
- LICENSE
- MODULE.bazel
- pyproject.toml
- README.md
- setup.py
- test_requirements.in
- test_requirements_lock_3_11.txt
- test_requirements_lock_3_12.txt
- test_requirements_lock_3_13.txt
- test_requirements_lock_3_14.txt
# 설치 가이드
1. 코드 내려받기
git clone https://github.com/google/grain
깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.
cd grain
방금 내려받은 프로젝트 폴더 안으로 이동합니다.
2. Docker
쉬움 추천사전 준비물
- Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
- Docker Desktop 컨테이너를 빌드하고 실행하려면 필요합니다. 설치 후 실행해서 백그라운드에 켜두세요.
⚠️ 이 프로젝트는 규모가 큰 저장소라, 이 방법이 실제 핵심 제품이 아니라 내부 하위 패키지를 가리키는 것일 수 있습니다. README 전체를 함께 확인해보세요.
docker build -f grain/oss/Dockerfile -t grain .
Dockerfile을 기반으로 실행 가능한 이미지를 빌드합니다.
docker run -p 8080:80 grain
빌드된 이미지를 실제 컨테이너로 실행합니다.
터미널에 docker compose ps 를 입력해 컨테이너들이 Up 상태인지 확인하세요. README에 포트 번호가 적혀있다면 브라우저에서 http://localhost:포트번호 로 접속해보세요.
// repository documentation
Was this content helpful?
(0 ratings)