data-juicer
Data processing for and with foundation models! ๐ ๐ ๐ฝ โก๏ธ โก๏ธ๐ธ ๐น ๐ท
File Explorer
Download Latest Version (.zip)- bug_report.yml
- custom.md
- feature_request.yml
- question.yml
- fetch_traffic_data.py
- docker-compose.yml
- deploy_sphinx_docs.yml
- perf-bench.yml
- pre-commit.yml
- publish-docker-oss.yml
- publish-docker.yml
- publish-pypi.yml
- stale.yml
- traffic-report.yml
- unit-test-partial.yml
- unit-test.yml
- build_op_doc.py
- tag_mappings.json
- __init__.py
- collector.py
- column_wise_analysis.py
- correlation_analysis.py
- diversity_analysis.py
- measure.py
- overall_analysis.py
- __init__.py
- config.py
- config_all.yaml
- config_min.yaml
- recipe_llm_semantic.yaml
- __init__.py
- config_validator.py
- data_validator.py
- dataset_builder.py
- dj_dataset.py
- load_strategy.py
- ray_dataset.py
- schema.py
- __init__.py
- base.py
- dag_execution_mixin.py
- dag_execution_strategies.py
- default_executor.py
- event_logging_mixin.py
- factory.py
- partition_size_optimizer.py
- pipeline_dag.py
- ray_executor.py
- ray_executor_partitioned.py
- __init__.py
- ray_tracer.py
- tracer.py
- __init__.py
- adapter.py
- analyzer.py
- exporter.py
- monitor.py
- ray_analyzer.py
- ray_exporter.py
- __init__.py
- arxiv.py
- commoncrawl.py
- downloader.py
- wikipedia.py
- __init__.py
- csv_formatter.py
- empty_formatter.py
- formatter.py
- json_formatter.py
- load.py
- parquet_formatter.py
- text_formatter.py
- tsv_formatter.py
- __init__.py
- entity_attribute_aggregator.py
- meta_tags_aggregator.py
- most_relevant_entities_aggregator.py
- nested_aggregator.py
- __init__.py
- dwpose_func.py
- hawor_constants.py
- hawor_func.py
- hawor_func_vit.py
- helper_func.py
- mano_func.py
- prompt2prompt_pipeline.py
- special_characters.py
- __init__.py
- document_deduplicator.py
- document_line_deduplicator.py
- document_minhash_deduplicator.py
- document_simhash_deduplicator.py
- image_deduplicator.py
- minhash.cpp
- ray_basic_deduplicator.py
- ray_bts_minhash_cpp_deduplicator.py
- ray_bts_minhash_deduplicator.py
- ray_document_deduplicator.py
- ray_image_deduplicator.py
- ray_video_deduplicator.py
- tokenize.pyx
- video_deduplicator.py
- __init__.py
- alphanumeric_filter.py
- audio_duration_filter.py
- audio_nmf_snr_filter.py
- audio_size_filter.py
- average_line_length_filter.py
- character_repetition_filter.py
- flagged_words_filter.py
- general_field_filter.py
- image_aesthetics_filter.py
- image_aspect_ratio_filter.py
- image_face_count_filter.py
- image_face_ratio_filter.py
- image_nsfw_filter.py
- image_pair_similarity_filter.py
- image_shape_filter.py
- image_size_filter.py
- image_subplot_filter.py
- image_text_matching_filter.py
- image_text_similarity_filter.py
- image_watermark_filter.py
- in_context_influence_filter.py
- instruction_following_difficulty_filter.py
- language_id_score_filter.py
- llm_analysis_filter.py
- llm_condition_filter.py
- llm_difficulty_score_filter.py
- llm_perplexity_filter.py
- llm_quality_score_filter.py
- llm_task_relevance_filter.py
- maximum_line_length_filter.py
- perplexity_filter.py
- phrase_grounding_recall_filter.py
- special_characters_filter.py
- specified_field_filter.py
- specified_numeric_field_filter.py
- stopwords_filter.py
- suffix_filter.py
- text_action_filter.py
- text_embd_similarity_filter.py
- text_entity_dependency_filter.py
- text_length_filter.py
- text_pair_similarity_filter.py
- token_num_filter.py
- video_aesthetics_filter.py
- video_aspect_ratio_filter.py
- video_duration_filter.py
- video_face_ratio_filter.py
- video_frames_text_similarity_filter.py
- video_motion_score_filter.py
- video_motion_score_ptlflow_filter.py
- video_motion_score_raft_filter.py
- video_nsfw_filter.py
- video_ocr_area_ratio_filter.py
- video_resolution_filter.py
- video_tagging_from_frames_filter.py
- video_watermark_filter.py
- word_repetition_filter.py
- words_num_filter.py
- __init__.py
- key_value_grouper.py
- naive_grouper.py
- naive_reverse_grouper.py
- __init__.py
- annotation_mapper.py
- human_preference_annotation_mapper.py
- __init__.py
- agent_bad_case_signal_mapper.py
- agent_dialog_normalize_mapper.py
- agent_insight_llm_mapper.py
- agent_skill_insight_mapper.py
- agent_tool_relevance_mapper.py
- agent_tool_type_mapper.py
- agent_trace_coherence_mapper.py
- audio_add_gaussian_noise_mapper.py
- audio_ffmpeg_wrapped_mapper.py
- calibrate_qa_mapper.py
- calibrate_query_mapper.py
- calibrate_response_mapper.py
- chinese_convert_mapper.py
- clean_copyright_mapper.py
- clean_email_mapper.py
- clean_html_mapper.py
- clean_ip_mapper.py
- clean_links_mapper.py
- detect_character_attributes_mapper.py
- detect_character_locations_mapper.py
- detect_main_character_mapper.py
- dialog_clarification_quality_mapper.py
- dialog_coreference_mapper.py
- dialog_error_recovery_mapper.py
- dialog_intent_detection_mapper.py
- dialog_llm_input_utils.py
- dialog_memory_consistency_mapper.py
- dialog_non_repetition_mapper.py
- dialog_proactivity_mapper.py
- dialog_quality_llm_base.py
- dialog_quality_llm_utils.py
- dialog_sentiment_detection_mapper.py
- dialog_sentiment_intensity_mapper.py
- dialog_topic_detection_mapper.py
- dialog_topic_shift_mapper.py
- download_file_mapper.py
- expand_macro_mapper.py
- export_to_lerobot_mapper.py
- extract_entity_attribute_mapper.py
- extract_entity_relation_mapper.py
- extract_event_mapper.py
- extract_keyword_mapper.py
- extract_nickname_mapper.py
- extract_support_text_mapper.py
- extract_tables_from_html_mapper.py
- fix_unicode_mapper.py
- generate_qa_from_examples_mapper.py
- generate_qa_from_text_mapper.py
- image_blur_mapper.py
- image_captioning_mapper.py
- image_detection_yolo_mapper.py
- image_diffusion_mapper.py
- image_face_blur_mapper.py
- image_mmpose_mapper.py
- image_remove_background_mapper.py
- image_sam_3d_body_mapper.py
- image_segment_mapper.py
- image_tagging_mapper.py
- image_tagging_vlm_mapper.py
- imgdiff_difference_area_generator_mapper.py
- imgdiff_difference_caption_generator_mapper.py
- latex_figure_context_extractor_mapper.py
- latex_merge_tex_mapper.py
- llm_extract_mapper.py
- mllm_mapper.py
- nlpaug_en_mapper.py
- nlpcda_zh_mapper.py
- optimize_prompt_mapper.py
- optimize_qa_mapper.py
- optimize_query_mapper.py
- optimize_response_mapper.py
- pair_preference_mapper.py
- pii_llm_suspect_mapper.py
- pii_redaction_mapper.py
- punctuation_normalization_mapper.py
- python_file_mapper.py
- python_lambda_mapper.py
- query_intent_detection_mapper.py
- query_sentiment_detection_mapper.py
- query_topic_detection_mapper.py
- relation_identity_mapper.py
- remove_bibliography_mapper.py
- remove_comments_mapper.py
- remove_header_mapper.py
- remove_long_words_mapper.py
- remove_non_chinese_character_mapper.py
- remove_repeat_sentences_mapper.py
- remove_specific_chars_mapper.py
- remove_table_text_mapper.py
- remove_words_with_incorrect_substrings_mapper.py
- replace_content_mapper.py
- s3_download_file_mapper.py
- s3_upload_file_mapper.py
- sdxl_prompt2prompt_mapper.py
- sentence_augmentation_mapper.py
- sentence_split_mapper.py
- text_chunk_mapper.py
- text_tagging_by_prompt_mapper.py
- tool_success_tagger_mapper.py
- usage_counter_mapper.py
- vggt_mapper.py
- video_active_speaker_detect_mapper.py
- video_atomic_action_segment_mapper.py
- video_audio_ASR_mapper.py
- video_audio_detect_age_gender_mapper.py
- video_audio_speech_emotion_mapper.py
- video_camera_calibration_deepcalib_mapper.py
- video_camera_calibration_droidcalib_mapper.py
- video_camera_calibration_moge_mapper.py
- video_camera_pose_megasam_mapper.py
- video_captioning_face_attribute_emotion_mapper.py
- video_captioning_from_audio_mapper.py
- video_captioning_from_frames_mapper.py
- video_captioning_from_human_tracks_mapper.py
- video_captioning_from_summarizer_mapper.py
- video_captioning_from_video_mapper.py
- video_captioning_from_vlm_mapper.py
- video_clip_reassembly_mapper.py
- video_depth_estimation_mapper.py
- video_extract_frames_mapper.py
- video_face_blur_mapper.py
- video_ffmpeg_wrapped_mapper.py
- video_hand_action_compute_mapper.py
- video_hand_motion_smooth_mapper.py
- video_hand_reconstruction_hawor_mapper.py
- video_hand_reconstruction_mapper.py
- video_human_tracks_extraction_mapper.py
- video_human_tracks_face_demographic_mapper.py
- video_object_segmenting_mapper.py
- video_remove_watermark_mapper.py
- video_resize_aspect_ratio_mapper.py
- video_resize_resolution_mapper.py
- video_split_by_duration_mapper.py
- video_split_by_key_frame_mapper.py
- video_split_by_scene_mapper.py
- video_tagging_from_audio_mapper.py
- video_tagging_from_frames_mapper.py
- video_trajectory_overlay_mapper.py
- video_undistort_mapper.py
- video_whole_body_pose_estimation_mapper.py
- whitespace_normalization_mapper.py
- __init__.py
- llm_inference_with_ray_vllm_pipeline.py
- ray_repartition_pipeline.py
- ray_vllm_pipeline.py
- vlm_inference_with_ray_vllm_pipeline.py
- __init__.py
- frequency_specified_field_selector.py
- image_ohem_selector.py
- random_selector.py
- range_specified_field_selector.py
- tags_specified_field_selector.py
- topk_specified_field_selector.py
- __init__.py
- base_op.py
- fused_batch_executor.py
- fused_sequential_batch_op.py
- load.py
- mixins.py
- op_env.py
- op_fusion.py
- process.yaml
- quality_score_hpo.yaml
- __init__.py
- demo-redpajama-c4-refined.jsonl
- execute_hpo_3sigma.py
- execute_hpo_wandb.py
- objects.py
- README.md
- README_ZH.md
- __init__.py
- eval.py
- predict.py
- qc_utils.py
- README.md
- README_ZH.md
- train.py
- __init__.py
- DJ_mcp_granular_ops.py
- DJ_mcp_recipe_flow.py
- mcp_server.py
- mcp_tool.py
- op_search.py
- __init__.py
- common.py
- monitor.py
- snapshot.py
- stopper.py
- __init__.py
- agent_output_locale.py
- ASD_mapper_utils.py
- asset_utils.py
- availability_utils.py
- cache_utils.py
- ckpt_utils.py
- common_utils.py
- compress.py
- config_utils.py
- constant.py
- datasets_json_compat.py
- encryption_utils.py
- file_utils.py
- fingerprint_utils.py
- fs_utils.py
- hdfs_utils.py
- jsonl_lenient_loader.py
- lazy_loader.py
- llm_semantic_ops.py
- logger_utils.py
- mm_utils.py
- model_utils.py
- nltk_utils.py
- process_utils.py
- ray_cluster_utils.py
- ray_utils.py
- registry.py
- resource_utils.py
- s3_utils.py
- sample.py
- unittest_utils.py
- video_utils.py
- webdataset_utils.py
- __init__.py
- 01_normalize_only.yaml
- 02_normalize_pii.yaml
- 03_clean.yaml
- 04_usage_tool.yaml
- 05_filters.yaml
- 06_one_dialog_mapper.yaml
- 07_entity_keyword.yaml
- 08_one_llm_filter.yaml
- 09_bad_case_smoke.yaml
- README.md
- analyze_bad_case_cohorts.py
- bad_case_signal_support.py
- compute_percentile_thresholds.py
- dj_export_row.py
- fetch_demo_data.sh
- generate_bad_case_report.py
- README.md
- run_bad_case_pipeline.sh
- slice_export_by_tier.py
- verify_bad_case_export.py
- agent_interaction_quality_analysis.yaml
- BAD_CASE_INSIGHTS.md
- BAD_CASE_REPORT.md
- PERFORMANCE_LLM.md
- QUICKSTART_BAD_CASE.md
- README.md
- analyzer.yaml
- ray_analyzer.yaml
- dj_config_template.yaml
- model_configs.json
- react_data_filter_process.ipynb
- react_data_mapper_process.ipynb
- utils.py
- wrapped_filters.py
- wrapped_mappers.py
- data-juicer.png
- eval-01.png
- eval-02.png
- app.py
- README_ZH.md
- demo-dataset-prompts.jsonl
- init_infer_dataset.jsonl
- demo-dataset-annotation-human-preference.jsonl
- demo-dataset-chatml.jsonl
- demo-dataset-content.jsonl
- demo-dataset-deduplication.jsonl
- demo-dataset-images-bytes.parquet
- demo-dataset-images.jsonl
- demo-dataset-videos.jsonl
- demo-dataset-videos2.jsonl
- demo-dataset.jsonl
- demo-dataset_1725870268.jsonl
- demo-dataset_1725870628.jsonl
- redpajama-c4-refined.jsonl
- redpajama-cc-2023-06-refined.jsonl
- redpajama-pile-stackexchange-refined.jsonl
- the-pile-nih-refined.jsonl
- the-pile-uspto-refined.jsonl
- app.py
- data-juicer.png
- app.py
- demo.yaml
- demo-dataset.jsonl
- app.py
- demo.yaml
- demo-dataset.jsonl
- app.py
- demo_en.yaml
- demo_zh.yaml
- demo-dataset.jsonl
- app.py
- .gitkeep
- app.css
- app.py
- demo.yaml
- demo-dataset.jsonl
- app.py
- vla_pipeline.yaml
- __init__.py
- video_action_captioning_mapper.py
- video_action_captioning_mapper2.py
- 1018.mp4
- 1034.mp4
- demo-dataset.jsonl
- __init__.py
- process.py
- run_vis2.py
- __init__.py
- common_utils.py
- Dockerfile
- Dockerfile.uv
- README.md
- vis_hand_action_demo.py
- vis_segments.py
- vla_pipeline.py
- demo.yaml
- gpu_demo.yaml
- gpu_demo_4gpu.yaml
- gpu-demo-dataset.jsonl
- dlc_job.py
- README.md
- README_ZH.md
- shard_job.py
- two_node_test.py
- demo-dataset.jsonl
- app.py
- partition-checkpoint-eventlog-control.yaml
- partition-checkpoint-eventlog.yaml
- example_event_log.jsonl
- example_processing_summary.json
- partition_determinism_benchmark.py
- robustness_benchmark.py
- run_demo.py
- alpaca-cot.jsonl
- app.py
- stack_exchange.jsonl
- app.py
- dedup.yaml
- demo-new-config.yaml
- demo.yaml
- demo_hdfs.yaml
- demo-dataset.json
- demo-dataset.jsonl
- arxiv.jsonl
- app.py
- process.yaml
- process_hdfs.yaml
- demo-new-config.yaml
- demo.yaml
- s3_video_processing_config.yaml
- video1.mp4
- video2.mp4
- video3.mp4
- demo-dataset-s3.jsonl
- demo-dataset.jsonl
- Note.md
- process.yaml
- README_ZH.md
- role_playing_system_prompt.yaml
- system_prompt_generator.py
- demo-dataset.jsonl
- app.py
- dataset_splitting_by_language.py
- demo-dataset.jsonl
- __init__.py
- eval.py
- predict.py
- qc_utils.py
- train.py
- app.py
- analyzer.yaml
- README.md
- README_ZH.md
- README.md
- README_ZH.md
- data-juicer.jpg
- eval-01.png
- eval-02.png
- check_render.py
- op_doc.md.j2
- __init__.py
- example_ir.py
- extractor.py
- llm_service.py
- md_parser.py
- model.py
- parse_class.py
- prompts.py
- router.py
- view_model.py
- examples.json
- generate_op_details.py
- rewrite_op_docstrings.py
- runner.py
- entity_attribute_aggregator.md
- meta_tags_aggregator.md
- most_relevant_entities_aggregator.md
- nested_aggregator.md
- document_deduplicator.md
- document_minhash_deduplicator.md
- document_minhash_deduplicator_with_uid.md
- document_simhash_deduplicator.md
- image_deduplicator.md
- ray_bts_minhash_cpp_deduplicator.md
- ray_bts_minhash_deduplicator.md
- ray_bts_minhash_deduplicator_with_uid.md
- ray_document_deduplicator.md
- ray_image_deduplicator.md
- ray_video_deduplicator.md
- video_deduplicator.md
- alphanumeric_filter.md
- audio_duration_filter.md
- audio_nmf_snr_filter.md
- audio_size_filter.md
- average_line_length_filter.md
- character_repetition_filter.md
- flagged_words_filter.md
- general_field_filter.md
- image_aesthetics_filter.md
- image_aspect_ratio_filter.md
- image_face_count_filter.md
- image_face_ratio_filter.md
- image_nsfw_filter.md
- image_pair_similarity_filter.md
- image_shape_filter.md
- image_size_filter.md
- image_subplot_filter.md
- image_text_matching_filter.md
- image_text_similarity_filter.md
- image_watermark_filter.md
- in_context_influence_filter.md
- instruction_following_difficulty_filter.md
- language_id_score_filter.md
- llm_analysis_filter.md
- llm_condition_filter.md
- llm_difficulty_score_filter.md
- llm_perplexity_filter.md
- llm_quality_score_filter.md
- llm_task_relevance_filter.md
- maximum_line_length_filter.md
- perplexity_filter.md
- phrase_grounding_recall_filter.md
- special_characters_filter.md
- specified_field_filter.md
- specified_numeric_field_filter.md
- stopwords_filter.md
- suffix_filter.md
- text_action_filter.md
- text_embd_similarity_filter.md
- text_entity_dependency_filter.md
- text_length_filter.md
- text_pair_similarity_filter.md
- token_num_filter.md
- video_aesthetics_filter.md
- video_aspect_ratio_filter.md
- video_duration_filter.md
- video_face_ratio_filter.md
- video_frames_text_similarity_filter.md
- video_motion_score_filter.md
- video_motion_score_ptlflow_filter.md
- video_motion_score_raft_filter.md
- video_nsfw_filter.md
- video_ocr_area_ratio_filter.md
- video_resolution_filter.md
- video_tagging_from_frames_filter.md
- video_watermark_filter.md
- word_repetition_filter.md
- words_num_filter.md
- csv_formatter.md
- empty_formatter.md
- json_formatter.md
- parquet_formatter.md
- ray_empty_formatter.md
- text_formatter.md
- tsv_formatter.md
- key_value_grouper.md
- naive_grouper.md
- naive_reverse_grouper.md
- audio_add_gaussian_noise_mapper.md
- audio_ffmpeg_wrapped_mapper.md
- calibrate_qa_mapper.md
- calibrate_query_mapper.md
- calibrate_response_mapper.md
- chinese_convert_mapper.md
- clean_copyright_mapper.md
- clean_email_mapper.md
- clean_html_mapper.md
- clean_ip_mapper.md
- clean_links_mapper.md
- detect_character_attributes_mapper.md
- detect_character_locations_mapper.md
- detect_main_character_mapper.md
- dialog_intent_detection_mapper.md
- dialog_sentiment_detection_mapper.md
- dialog_sentiment_intensity_mapper.md
- dialog_topic_detection_mapper.md
- download_file_mapper.md
- expand_macro_mapper.md
- extract_entity_attribute_mapper.md
- extract_entity_relation_mapper.md
- extract_event_mapper.md
- extract_keyword_mapper.md
- extract_nickname_mapper.md
- extract_support_text_mapper.md
- extract_tables_from_html_mapper.md
- fix_unicode_mapper.md
- general_fused_op.md
- generate_qa_from_examples_mapper.md
- generate_qa_from_text_mapper.md
- human_preference_annotation_mapper.md
- image_blur_mapper.md
- image_captioning_mapper.md
- image_detection_yolo_mapper.md
- image_diffusion_mapper.md
- image_face_blur_mapper.md
- image_mmpose_mapper.md
- image_remove_background_mapper.md
- image_sam_3d_body_mapper.md
- image_segment_mapper.md
- image_tagging_mapper.md
- image_tagging_vlm_mapper.md
- imgdiff_difference_area_generator_mapper.md
- imgdiff_difference_caption_generator_mapper.md
- latex_figure_context_extractor_mapper.md
- latex_merge_tex_mapper.md
- llm_extract_mapper.md
- mllm_mapper.md
- nlpaug_en_mapper.md
- nlpcda_zh_mapper.md
- optimize_prompt_mapper.md
- optimize_qa_mapper.md
- optimize_query_mapper.md
- optimize_response_mapper.md
- pair_preference_mapper.md
- punctuation_normalization_mapper.md
- python_file_mapper.md
- python_lambda_mapper.md
- query_intent_detection_mapper.md
- query_sentiment_detection_mapper.md
- query_topic_detection_mapper.md
- relation_identity_mapper.md
- remove_bibliography_mapper.md
- remove_comments_mapper.md
- remove_header_mapper.md
- remove_long_words_mapper.md
- remove_non_chinese_character_mapper.md
- remove_repeat_sentences_mapper.md
- remove_specific_chars_mapper.md
- remove_table_text_mapper.md
- remove_words_with_incorrect_substrings_mapper.md
- replace_content_mapper.md
- s3_download_file_mapper.md
- s3_upload_file_mapper.md
- sdxl_prompt2prompt_mapper.md
- sentence_augmentation_mapper.md
- sentence_split_mapper.md
- text_chunk_mapper.md
- text_tagging_by_prompt_mapper.md
- vggt_mapper.md
- video_active_speaker_detect_mapper.md
- video_audio_ASR_mapper.md
- video_audio_detect_age_gender_mapper.md
- video_audio_speech_emotion_mapper.md
- video_camera_calibration_deepcalib_mapper.md
- video_camera_calibration_moge_mapper.md
- video_captioning_face_attribute_emotion_mapper.md
- video_captioning_from_audio_mapper.md
- video_captioning_from_frames_mapper.md
- video_captioning_from_human_tracks_mapper.md
- video_captioning_from_summarizer_mapper.md
- video_captioning_from_video_mapper.md
- video_captioning_from_vlm_mapper.md
- video_depth_estimation_mapper.md
- video_extract_frames_mapper.md
- video_face_blur_mapper.md
- video_ffmpeg_wrapped_mapper.md
- video_hand_reconstruction_mapper.md
- video_human_tracks_extraction_mapper.md
- video_human_tracks_face_demographic_mapper.md
- video_object_segmenting_mapper.md
- video_remove_watermark_mapper.md
- video_resize_aspect_ratio_mapper.md
- video_resize_resolution_mapper.md
- video_split_by_duration_mapper.md
- video_split_by_key_frame_mapper.md
- video_split_by_scene_mapper.md
- video_tagging_from_audio_mapper.md
- video_tagging_from_frames_mapper.md
- video_undistort_mapper.md
- video_whole_body_pose_estimation_mapper.md
- whitespace_normalization_mapper.md
- llm_ray_vllm_engine_pipeline.md
- ray_repartition_pipeline.md
- vlm_ray_vllm_engine_pipeline.md
- frequency_specified_field_selector.md
- image_ohem_selector.md
- random_selector.md
- range_specified_field_selector.md
- tags_specified_field_selector.md
- topk_specified_field_selector.md
- api.rst
- docs_index.rst
- docs_index_ZH.rst
- extra_assets.yaml
- index.rst
- index_ZH.rst
- DJ-Cookbook.md
- DJ-Cookbook_ZH.md
- Installation.md
- Installation_ZH.md
- QuickStart.md
- QuickStart_ZH.md
- awesome_llm_data.md
- BadDataExhibition.md
- BadDataExhibition_ZH.md
- Cache.md
- Cache_ZH.md
- DatasetCfg.md
- DatasetCfg_ZH.md
- DeveloperGuide.md
- DeveloperGuide_ZH.md
- Distributed.md
- Distributed_ZH.md
- DJ_service.md
- DJ_service_ZH.md
- DJ_SORA.md
- DJ_SORA_ZH.md
- Export.md
- Export_ZH.md
- JobManagement.md
- JobManagement_ZH.md
- news.md
- news_zh.md
- OperatorPlugins.md
- OperatorPlugins_ZH.md
- Operators.md
- PartitionAndCheckpoint.md
- PartitionAndCheckpoint_ZH.md
- Tracing.md
- Tracing_ZH.md
- partition_data_dlc.py
- run_on_dlc.sh
- README.md
- run_slurm.sh
- __init__.py
- test_collector.py
- test_column_wise_analysis.py
- test_correlation_analysis.py
- test_diversity_analysis.py
- test_measure.py
- test_overall_analysis.py
- audio.yaml
- image.yaml
- text.yaml
- video.yaml
- report.py
- run.sh
- __init__.py
- demo_4_test.yaml
- demo_4_test_bad_val.yaml
- demo_4_test_multiple_text_keys.yaml
- demo_4_test_same_ops.yaml
- test_config.py
- test_config_functions.py
- sample.parquet
- sample.jsonl
- sample.txt
- test_config.yaml
- test_config_list.yaml
- test_config_ray.yaml
- __init__.py
- test_config_validator.py
- test_data_validator.py
- test_dataset_builder.py
- test_dataset_error_propagation.py
- test_dj_dataset.py
- test_load_strategy.py
- test_ray_dataset.py
- test_schema.py
- __init__.py
- test_auto_partition_cluster.py
- test_base.py
- test_dag_execution_mixin.py
- test_dag_execution_strategies.py
- test_default_executor.py
- test_event_logging_mixin.py
- test_factory.py
- test_partition_size_optimizer.py
- test_partitioned_integration.py
- test_pipeline_dag.py
- test_ray_executor.py
- test_ray_executor_partitioned.py
- test_ray_executor_partitioned_parallel.py
- test_ray_partition_resume.py
- __init__.py
- test_ray_tracer.py
- test_tracer.py
- __init__.py
- test_adapter.py
- test_analyzer.py
- test_analyzer_mapper_fusion.py
- test_exporter.py
- test_monitor.py
- test_ray_analyzer.py
- test_ray_exporter.py
- test_generate_bad_case_report_smoke.py
- test_elastic_sharding.py
- __init__.py
- test_download.py
- demo-dataset.csv
- demo-dataset.jsonl
- demo-dataset.parquet
- demo-dataset.tsv
- sample1.txt
- sample2.txt
- sample3.txt
- sample4.txt
- sample5.txt
- sample6.txt
- __init__.py
- test_csv_formatter.py
- test_empty_formatter.py
- test_formatter.py
- test_json_formatter.py
- test_load.py
- test_parquet_formatter.py
- test_text_formatter.py
- test_tsv_formatter.py
- __init__.py
- test_entity_attribute_aggregator.py
- test_meta_tags_aggregator.py
- test_most_relevant_entities_aggregator.py
- test_nested_aggregator.py
- __init__.py
- test_dwpose_func.py
- test_hawor_constants.py
- test_hawor_func.py
- test_hawor_func_vit.py
- test_helper_func.py
- test_mano_func.py
- test_prompt2prompt_pipeline.py
- test_special_characters.py
- audio1.wav
- audio2.wav
- audio3.ogg
- blip.jpg
- cat.jpg
- image_nosubplot.jpg
- image_subplot.jpg
- img1.png
- img10.jpg
- img2.jpg
- img3.jpg
- img4.png
- img5.jpg
- img6.jpg
- img7.jpg
- img8.jpg
- img9.jpg
- img_pair_1.jpg
- img_pair_2.jpg
- lena-face.jpg
- lena.jpg
- video1.mp4
- video10.mp4
- video11.mp4
- video12.mp4
- video13.mp4
- video14.mp4
- video15.mp4
- video2.mp4
- video3-no-audio.mp4
- video3.mp4
- video4.mp4
- video5.mp4
- video6.mp4
- video7.mp4
- video8.mp4
- video9.mp4
- __init__.py
- test_document_deduplicator.py
- test_document_line_deduplicator.py
- test_document_minhash_deduplicator.py
- test_document_simhash_deduplicator.py
- test_image_deduplicator.py
- test_minhash_blockwise_permutation.py
- test_ray_basic_deduplicator.py
- test_ray_bts_minhash_cpp_deduplicator.py
- test_ray_bts_minhash_deduplicator.py
- test_ray_document_deduplicator.py
- test_ray_image_deduplicator.py
- test_ray_video_deduplicator.py
- test_video_deduplicator.py
- __init__.py
- test_alphanumeric_filter.py
- test_audio_duration_filter.py
- test_audio_nmf_snr_filter.py
- test_audio_size_filter.py
- test_average_line_length_filter.py
- test_character_repetition_filter.py
- test_flagged_words_filter.py
- test_general_field_filter.py
- test_image_aesthetics_filter.py
- test_image_aspect_ratio_filter.py
- test_image_face_count_filter.py
- test_image_face_ratio_filter.py
- test_image_nsfw_filter.py
- test_image_pair_similarity_filter.py
- test_image_shape_filter.py
- test_image_size_filter.py
- test_image_subplot_filter.py
- test_image_text_matching_filter.py
- test_image_text_similarity_filter.py
- test_image_watermark_filter.py
- test_in_context_influence_filter.py
- test_instruction_following_difficulty_filter.py
- test_language_id_score_filter.py
- test_llm_analysis_filter.py
- test_llm_condition_filter.py
- test_llm_difficulty_score_filter.py
- test_llm_perplexity_filter.py
- test_llm_quality_score_filter.py
- test_llm_task_relevance_filter.py
- test_maximum_line_length_filter.py
- test_ngram_streaming_frequency.py
- test_perplexity_filter.py
- test_phrase_grounding_recall_filter.py
- test_special_characters_filter.py
- test_specified_field_filter.py
- test_specified_numeric_field_filter.py
- test_stopwords_filter.py
- test_suffix_filter.py
- test_text_action_filter.py
- test_text_embd_similarity_filter.py
- test_text_entity_dependency_filter.py
- test_text_length_filter.py
- test_text_pair_similarity_filter.py
- test_token_num_filter.py
- test_token_num_filter_bounded_batch.py
- test_video_aesthetics_filter.py
- test_video_aspect_ratio_filter.py
- test_video_duration_filter.py
- test_video_face_ratio_filter.py
- test_video_frames_text_similarity_filter.py
- test_video_motion_score_filter.py
- test_video_motion_score_ptlflow_filter.py
- test_video_motion_score_raft_filter.py
- test_video_nsfw_filter.py
- test_video_ocr_area_ratio_filter.py
- test_video_resolution_filter.py
- test_video_tagging_from_frames_filter.py
- test_video_watermark_filter.py
- test_word_repetition_filter.py
- test_words_num_filter.py
- __init__.py
- test_key_value_grouper.py
- test_naive_grouper.py
- test_naive_reverse_grouper.py
- __init__.py
- test_annotation_mapper.py
- test_human_preference_annotation_mapper.py
- __init__.py
- test_agent_bad_case_signal_mapper.py
- test_agent_dialog_normalize_mapper.py
- test_agent_insight_llm_mapper.py
- test_agent_skill_insight_mapper.py
- test_agent_tool_type_mapper.py
- test_audio_add_gaussian_noise_mapper.py
- test_audio_ffmpeg_wrapped_mapper.py
- test_calibrate_qa_mapper.py
- test_calibrate_query_mapper.py
- test_calibrate_response_mapper.py
- test_chinese_convert_mapper.py
- test_clean_copyright_mapper.py
- test_clean_email_mapper.py
- test_clean_html_mapper.py
- test_clean_ip_mapper.py
- test_clean_links_mapper.py
- test_detect_character_attributes_mapper.py
- test_detect_character_locations_mapper.py
- test_detect_main_character_mapper.py
- test_dialog_intent_detection_mapper.py
- test_dialog_llm_input_utils.py
- test_dialog_quality_llm.py
- test_dialog_sentiment_detection_mapper.py
- test_dialog_sentiment_intensity_mapper.py
- test_dialog_topic_detection_mapper.py
- test_download_file_mapper.py
- test_expand_macro_mapper.py
- test_export_to_lerobot_mapper.py
- test_extract_entity_attribute_mapper.py
- test_extract_entity_relation_mapper.py
- test_extract_event_mapper.py
- test_extract_keyword_mapper.py
- test_extract_nickname_mapper.py
- test_extract_support_text_mapper.py
- test_extract_tables_from_html_mapper.py
- test_fix_unicode_mapper.py
- test_generate_qa_from_examples_mapper.py
- test_generate_qa_from_text_mapper.py
- test_image_blur_mapper.py
- test_image_captioning_mapper.py
- test_image_detection_yolo_mapper.py
- test_image_diffusion_mapper.py
- test_image_face_blur_mapper.py
- test_image_mmpose_mapper.py
- test_image_remove_background_mapper.py
- test_image_sam_3d_body_mapper.py
- test_image_segment_mapper.py
- test_image_tagging_mapper.py
- test_image_tagging_vlm_mapper.py
- test_imgdiff_difference_area_generator_mapper.py
- test_imgdiff_difference_caption_generator_mapper.py
- test_latex_figure_context_extractor_mapper.py
- test_latex_merge_tex_mapper.py
- test_llm_extract_mapper.py
- test_mllm_mapper.py
- test_nlpaug_en_mapper.py
- test_nlpcda_zh_mapper.py
- test_optimize_prompt_mapper.py
- test_optimize_qa_mapper.py
- test_optimize_query_mapper.py
- test_optimize_response_mapper.py
- test_pair_preference_mapper.py
- test_pii_llm_suspect_mapper.py
- test_pii_redaction_mapper.py
- test_punctuation_normalization_mapper.py
- test_python_file_mapper.py
- test_python_lambda_mapper.py
- test_query_intent_detection_mapper.py
- test_query_sentiment_detection_mapper.py
- test_query_topic_detection_mapper.py
- test_relation_identity_mapper.py
- test_remove_bibliography_mapper.py
- test_remove_comments_mapper.py
- test_remove_header_mapper.py
- test_remove_long_words_mapper.py
- test_remove_non_chinese_character_mapper.py
- test_remove_repeat_sentences_mapper.py
- test_remove_specific_chars_mapper.py
- test_remove_table_text_mapper.py
- test_remove_words_with_incorrect_substrings_mapper.py
- test_replace_content_mapper.py
- test_s3_download_file_mapper.py
- test_s3_upload_file_mapper.py
- test_sdxl_prompt2prompt_mapper.py
- test_sentence_augmentation_mapper.py
- test_sentence_split_mapper.py
- test_text_chunk_mapper.py
- test_text_tagging_by_prompt_mapper.py
- test_tool_success_tagger_mapper.py
- test_usage_counter_mapper.py
- test_vggt_mapper.py
- test_video_active_speaker_detect_mapper.py
- test_video_atomic_action_segment_mapper.py
- test_video_audio_ASR_mapper.py
- test_video_audio_detect_age_gender_mapper.py
- test_video_audio_speech_emotion_mapper.py
- test_video_camera_calibration_deepcalib_mapper.py
- test_video_camera_calibration_droidcalib_mapper.py
- test_video_camera_calibration_moge_mapper.py
- test_video_camera_pose_megasam_mapper.py
- test_video_captioning_face_attribute_emotion_mapper.py
- test_video_captioning_from_audio_mapper.py
- test_video_captioning_from_frames_mapper.py
- test_video_captioning_from_human_tracks_mapper.py
- test_video_captioning_from_summarizer_mapper.py
- test_video_captioning_from_video_mapper.py
- test_video_captioning_from_vlm_mapper.py
- test_video_clip_reassembly_mapper.py
- test_video_depth_estimation_mapper.py
- test_video_extract_frames_mapper.py
- test_video_face_blur_mapper.py
- test_video_ffmpeg_wrapped_mapper.py
- test_video_hand_action_compute_mapper.py
- test_video_hand_motion_smooth_mapper.py
- test_video_hand_reconstruction_hawor_mapper.py
- test_video_human_tracks_extraction_mapper.py
- test_video_human_tracks_face_demographic_mapper.py
- test_video_object_segmenting_mapper.py
- test_video_remove_watermark_mapper.py
- test_video_resize_aspect_ratio_mapper.py
- test_video_resize_resolution_mapper.py
- test_video_split_by_duration_mapper.py
- test_video_split_by_key_frame_mapper.py
- test_video_split_by_scene_mapper.py
- test_video_tagging_from_audio_mapper.py
- test_video_tagging_from_frames_mapper.py
- test_video_trajectory_overlay_mapper.py
- test_video_undistort_mapper.py
- test_video_whole_body_pose_estimation_mapper.py
- test_whitespace_normalization_mapper.py
- __init__.py
- test_llm_inference_with_ray_vllm_pipeline.py
- test_ray_repartition_pipeline.py
- test_ray_vllm_pipeline.py
- test_vlm_inference_with_ray_vllm_pipeline.py
- __init__.py
- test_frequency_specified_field_selector.py
- test_image_ohem_selector.py
- test_random_selector.py
- test_range_specified_field_selector.py
- test_tags_specified_field_selector.py
- test_topk_specified_field_selector.py
- __init__.py
- test_base_op.py
- test_load.py
- test_load_op_plugins.py
- test_mapper_fusion.py
- test_mixins.py
- test_op_env.py
- test_op_fusion.py
- __init__.py
- test_DJ_mcp_granular_ops.py
- test_DJ_mcp_recipe_flow.py
- test_mcp_server.py
- test_mcp_tool.py
- test_op_search.py
- test_process_data.py
- test_common.py
- test_monitor.py
- test_snapshot.py
- test_stopper.py
- __init__.py
- test_agent_output_locale.py
- test_ASD_mapper_utils.py
- test_asset_utils.py
- test_availability_utils.py
- test_cache_utils.py
- test_ckpt_utils.py
- test_common_utils.py
- test_compress.py
- test_config_utils.py
- test_constant.py
- test_datasets_json_compat.py
- test_encryption_utils.py
- test_file_utils.py
- test_fingerprint_utils.py
- test_fs_utils.py
- test_hdfs_utils.py
- test_jsonl_lenient_loader.py
- test_lazy_loader.py
- test_llm_semantic_ops.py
- test_logger_utils.py
- test_mm_utils.py
- test_model_utils.py
- test_nltk_utils.py
- test_process_utils.py
- test_ray_utils.py
- test_registry.py
- test_resource_utils.py
- test_s3_utils.py
- test_sample.py
- test_unittest_utils.py
- test_video_utils.py
- test_webdataset_utils.py
- __init__.py
- run.py
- wav2vec_age_gender.py
- .gitmodules
- Light-ASD_changes.diff
- README.md
- SenseVoice_changes.diff
- YOLOv8_human_changes.diff
- helm.diff
- megatron.diff
- README.md
- README_ZH.md
- setup_helm.sh
- setup_megatron.sh
- easyanimate.diff
- README.md
- README_ZH.md
- setup_easyanimate.sh
- batch_convert.sh
- convert_gpt_to_transformers.py
- modeling_megatron_llama.py
- __init__.py
- dedup_utils.py
- README.md
- README_ZH.md
- spark_dedup.py
- evaluator_example.yaml
- helm_spec_template.conf
- gpt-3.5-turbo.jsonl
- config.yaml
- prompt.jsonl
- question.jsonl
- reviewer.jsonl
- __init__.py
- answer_generator.py
- gpt_evaluator.py
- README.md
- README_ZH.md
- leaderboard_example.yaml
- llama_example.yaml
- mymodel_example.yaml
- __init__.py
- README.md
- README_ZH.md
- wandb_writer.py
- evaluator.py
- README.md
- README_ZH.md
- dj_to_internvid.py
- dj_to_llava.py
- dj_to_mmc4.py
- dj_to_msrvtt.py
- dj_to_video_chatgpt.py
- dj_to_wavcaps.py
- dj_to_youku.py
- internvid_to_dj.py
- llava_to_dj.py
- mmc4_to_dj.py
- msrvtt_to_dj.py
- video_chatgpt_to_dj.py
- wavcaps_to_dj.py
- youku_to_dj.py
- absolute_path_to_relative_path.py
- README.md
- README_ZH.md
- utils.py
- dj_to_alpaca.py
- dj_to_llama_factory_sharegpt.py
- dj_to_messages.py
- dj_to_ms_swift_sharegpt.py
- alpaca_to_dj.py
- llama_factory_sharegpt_to_dj.py
- messages_to_dj.py
- ms_swift_sharegpt_to_dj.py
- README.md
- README_ZH.md
- README.md
- README_ZH.md
- enable_legacy_token.png
- label_studio_service.py
- README.md
- frechet_inception_distance.py
- frechet_video_distance.py
- inception_score.py
- kernel_inception_distance.py
- kernel_video_distance.py
- metric_main.py
- metric_utils.py
- precision_recall.py
- video_inception_score.py
- video_precision_recall.py
- calc_metrics_for_videos.py
- dataset.py
- distributed.py
- README.md
- README_ZH.md
- util.py
- evaluate.py
- README.md
- README_ZH.md
- VBench_full_info.json
- VBench_mini_info.json
- __init__.py
- __init__.py
- count_token.py
- data_mixture.py
- deserialize_meta.py
- README.md
- README_ZH.md
- dataset_split_by_language.py
- raw_alpaca_cot_merge_add_meta.py
- raw_arxiv_to_jsonl.py
- raw_stackexchange_to_jsonl.py
- README.md
- README_ZH.md
- reformat_csv_nan_value.py
- reformat_jsonl_nan_value.py
- serialize_meta.py
- __init__.py
- analyze_data.py
- check_ray_cluster.py
- check_s3_integration.py
- count_rows.py
- data_resplit.py
- dj_install.py
- generate_smtp_cert.py
- generate_uv_lock.py
- process_data.py
- .coveragerc
- .gitignore
- .pre-commit-config.yaml
- .secrets.baseline
- app.py
- Dockerfile
- Dockerfile.embodied
- hatch_build.py
- label_studio_localhost_connection.json
- LICENSE
- pyproject.toml
- README.md
- README_ZH.md
- service.py
- uv.lock
# Installation Guide
1. Get the code
git clone https://github.com/datajuicer/data-juicer
Downloads the entire project code from GitHub to your computer.
cd data-juicer
Moves into the project folder you just downloaded.
2. Docker
Easy RecommendedPrerequisites
- Git Needed to download the project code from GitHub.
- Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
docker build -t data-juicer .
Builds a runnable image based on the Dockerfile.
docker run -p 8080:80 data-juicer
Runs the built image as an actual container.
Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.
3. Python
EasyPrerequisites
uv pip install py-data-juicer
Installs the Python libraries listed in requirements.txt (or similar).
If it runs without errors and prints output in the terminal, it worked.
Pulled directly from this repo's README.
// repository documentation
Was this content helpful?
(0 ratings)