Rec-R1
[TMLR 2025] A general framework for bridging LLMs and recommendation systems via reinforcement learning. https://arxiv.org/pdf/2503.24289
파일 탐색기
최종 버전 다운로드 (.zip)- test.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- test.json
- train.json
- test.json
- train.json
- test.json
- train.json
- test.json
- train.json
- others.json
- test.json
- train.json
- test.parquet
- train.parquet
- val.parquet
- test.json
- train.json
- val.json
- test.json
- train.json
- val.json
- test.jsonl
- train.jsonl
- val.jsonl
- test.jsonl
- train.jsonl
- val.jsonl
- test.jsonl
- train.jsonl
- val.jsonl
- test.parquet
- train.parquet
- val.parquet
- train.parquet
- train.parquet
- test.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- test.parquet
- train.parquet
- val.parquet
- Baby_Products.json
- Office_Products.json
- Others.json
- Sports_and_Outdoors.json
- train.json
- Video_Games.json
- Dockerfile.ngc.vllm
- Dockerfile.vemlp.vllm.te
- logo.png
- dpo_extension.rst
- fsdp_extension.rst
- megatron_extension.rst
- placement.rst
- config.rst
- gsm8k_example.rst
- ppo_code_architecture.rst
- ppo.rst
- faq.rst
- prepare_data.rst
- reward_function.rst
- install.rst
- quickstart.rst
- fsdp_workers.rst
- megatron_workers.rst
- ray_trainer.rst
- conf.py
- index.rst
- Makefile
- README.md
- requirements-docs.txt
- arth.py
- full_hh_rlhf.py
- gsm8k.py
- hellaswag.py
- math_dataset.py
- multiply.py
- run_deepseek_v2_lite_math.sh
- run_deepseek7b_llm.sh
- run_deepseek7b_llm_seq_balance.sh
- run_qwen2-7b.sh
- run_qwen2-7b_seq_balance.sh
- run_deepseek7b_llm.sh
- run_deepseek7b_llm_sp2.sh
- run_deepseek_full_hh_rlhf.sh
- run_deepseek_math_gsm8k_megatron.sh
- run_deepseek_megatron.sh
- run_gemma.sh
- run_qwen2-7b.sh
- run_qwen2-7b_rm.sh
- run_qwen2-7b_rm_seq_balance.sh
- run_qwen2-7b_seq_balance.sh
- run_qwen2.5-32b.sh
- verl_getting_started.ipynb
- tutorial.ipynb
- run_deepseek_6b7.sh
- run_gemma_2b.sh
- run_gemma_7b.sh
- ppo_trainer_split.yaml
- main_ppo_split.py
- README.md
- run_deepseek7b_llm.sh
- split_monkey_patch.py
- megatron_v4.patch
- llm4rec.png
- rec-r1-compa.png
- eval_results_rec_r1.json
- gpt-4o-amazon-c4_Baby.json
- gpt-4o-amazon-c4_Office.json
- gpt-4o-amazon-c4_Sports.json
- gpt-4o-amazon-c4_Video_Games.json
- Qwen-inst-amazon-c4_Baby.json
- Qwen-inst-amazon-c4_Office.json
- Qwen-inst-amazon-c4_Sports.json
- Qwen-inst-amazon-c4_Video_Games.json
- query_metric_results-gpt-4o-inductive.json
- query_metric_results-gpt-4o-transductive.json
- query_metric_results-Rec-r1-inductive.json
- query_metric_results-Rec-r1-transductive.json
- gpt-4o-amazon-review_inductive.json
- gpt-4o-amazon-review_transductive.json
- Qwen-inst-amazon-review_inductive.json
- Qwen-inst-amazon-review_transductive.json
- Rec-r1-amazon-review_inductive.json
- Rec-r1-amazon-review_transductive.json
- eval_results_rec-r1.json
- query_metric_results-claude-3.5.json
- query_metric_results-claude-haiku.json
- query_metric_results-gpt-4o.json
- ori_query.json
- query_metric_results-claude-3.5.json
- query_metric_results-claude-haiku.json
- query_metric_results-gpt-4o.json
- query_metric_results-llama3-3b-inst.json
- query_metric_results-Qwen-0.5b-inst.json
- query_metric_results-Qwen-1.5b-inst.json
- query_metric_results-Qwen-3b-dpo-new.json
- query_metric_results-Qwen-3b-dpo.json
- query_metric_results-Qwen-3b-rej-sft.json
- query_metric_results-Qwen-7b-inst.json
- query_metric_results-rec-r1.json
- claude-3.5-esci_esci.json
- claude-haiku-esci_esci.json
- gpt-4o-esci_esci.json
- claude-3.5-esci_esci.json
- claude-haiku-esci_esci.json
- gpt-4o-esci_esci.json
- claude-3.5-esci_Baby_Products.json
- claude-3.5-esci_Office_Products.json
- claude-3.5-esci_Sports_and_Outdoors.json
- claude-3.5-esci_Video_Games.json
- claude-haiku-esci_Baby_Products.json
- claude-haiku-esci_Office_Products.json
- claude-haiku-esci_Sports_and_Outdoors.json
- claude-haiku-esci_Video_Games.json
- gpt-4o-esci_Baby_Products.json
- gpt-4o-esci_Office_Products.json
- gpt-4o-esci_Sports_and_Outdoors.json
- gpt-4o-esci_Video_Games.json
- llama3-3b-inst-esci_Baby_Products.json
- llama3-3b-inst-esci_Office_Products.json
- llama3-3b-inst-esci_Sports_and_Outdoors.json
- llama3-3b-inst-esci_Video_Games.json
- Qwen-0.5b-inst-esci_Baby_Products.json
- Qwen-0.5b-inst-esci_Office_Products.json
- Qwen-0.5b-inst-esci_Sports_and_Outdoors.json
- Qwen-0.5b-inst-esci_Video_Games.json
- Qwen-1.5b-inst-esci_Baby_Products.json
- Qwen-1.5b-inst-esci_Office_Products.json
- Qwen-1.5b-inst-esci_Sports_and_Outdoors.json
- Qwen-1.5b-inst-esci_Video_Games.json
- Qwen-3b-dpo-esci_Baby_Products.json
- Qwen-3b-dpo-esci_Office_Products.json
- Qwen-3b-dpo-esci_Sports_and_Outdoors.json
- Qwen-3b-dpo-esci_Video_Games.json
- Qwen-3b-dpo-new-esci_Baby_Products.json
- Qwen-3b-dpo-new-esci_Office_Products.json
- Qwen-3b-dpo-new-esci_Sports_and_Outdoors.json
- Qwen-3b-dpo-new-esci_Video_Games.json
- Qwen-3b-rej-sft-esci_Baby_Products.json
- Qwen-3b-rej-sft-esci_Office_Products.json
- Qwen-3b-rej-sft-esci_Sports_and_Outdoors.json
- Qwen-3b-rej-sft-esci_Video_Games.json
- Qwen-7b-inst-esci_Baby_Products.json
- Qwen-7b-inst-esci_Office_Products.json
- Qwen-7b-inst-esci_Sports_and_Outdoors.json
- Qwen-7b-inst-esci_Video_Games.json
- Qwen-inst-esci_Baby_Products.json
- Qwen-inst-esci_Office_Products.json
- Qwen-inst-esci_Sports_and_Outdoors.json
- Qwen-inst-esci_Video_Games.json
- Qwen-sft-esci_Baby_Products.json
- Qwen-sft-esci_Office_Products.json
- Qwen-sft-esci_Sports_and_Outdoors.json
- Qwen-sft-esci_Video_Games.json
- Qwen-sft-no-json-no-reason-esci_Baby_Products.json
- Qwen-sft-no-json-no-reason-esci_Office_Products.json
- Qwen-sft-no-json-no-reason-esci_Sports_and_Outdoors.json
- Qwen-sft-no-json-no-reason-esci_Video_Games.json
- Qwen-sft-no-reason-esci_Baby_Products.json
- Qwen-sft-no-reason-esci_Office_Products.json
- Qwen-sft-no-reason-esci_Sports_and_Outdoors.json
- Qwen-sft-no-reason-esci_Video_Games.json
- Qwen-sft-with-reason-no-json-esci_Baby_Products.json
- Qwen-sft-with-reason-no-json-esci_Office_Products.json
- Qwen-sft-with-reason-no-json-esci_Sports_and_Outdoors.json
- Qwen-sft-with-reason-no-json-esci_Video_Games.json
- rec-r1-esci_Video_Games.json
- gpt-4o-amazon-c4_Baby.json
- gpt-4o-amazon-c4_Office.json
- gpt-4o-amazon-c4_Sports.json
- gpt-4o-amazon-c4_Video_Games.json
- Qwen-inst-amazon-c4_Baby.json
- Qwen-inst-amazon-c4_Office.json
- Qwen-inst-amazon-c4_Sports.json
- Qwen-inst-amazon-c4_Video_Games.json
- gpt-4o-amazon-c4_Baby.json
- gpt-4o-amazon-c4_Office.json
- gpt-4o-amazon-c4_Sports.json
- gpt-4o-amazon-c4_Video_Games.json
- Qwen-inst-amazon-c4_Baby.json
- Qwen-inst-amazon-c4_Office.json
- Qwen-inst-amazon-c4_Sports.json
- Qwen-inst-amazon-c4_Video_Games.json
- rec-r1-amazon-c4_Baby.json
- gpt-4o-esci_Baby_Products.json
- gpt-4o-esci_Office_Products.json
- gpt-4o-esci_Sports_and_Outdoors.json
- gpt-4o-esci_Video_Games.json
- Qwen-inst-esci_Baby_Products.json
- Qwen-inst-esci_Office_Products.json
- Qwen-inst-esci_Sports_and_Outdoors.json
- Qwen-inst-esci_Video_Games.json
- gsm8k.sh
- humaneval.sh
- ifeval.sh
- mmlu.sh
- appliances.sh
- fashion.sh
- blair_large.sh
- roberta_base.sh
- roberta_large.sh
- simcse_base.sh
- simcse_large.sh
- gpt.sh
- qwen.sh
- rec-r1.sh
- gpt.sh
- qwen.sh
- rec-r1.sh
- rec-r1.sh
- gpt.sh
- qwen.sh
- rec-r1.sh
- blair_large.sh
- roberta_base.sh
- roberta_large.sh
- simcse_base.sh
- simsce_large.sh
- gpt.sh
- qwen.sh
- claude-haiku.sh
- claude.sh
- gpt.sh
- llama3-3b.sh
- qwen-0.5b.sh
- qwen-1.5b.sh
- qwen-7b.sh
- qwen-sft-no_reason_no_json.sh
- qwen-sft.sh
- qwen-sft_no_reason_with_json.sh
- qwen-sft_with_reason_no_json.sh
- qwen.sh
- qwen_dpo.sh
- qwen_rej_sft.sh
- rec-r1.sh
- eval_ori_query.sh
- eval_search.sh
- eval_search.sh
- eval_search.sh
- eval_search.sh
- eval_search.sh
- eval_sft_train_data.sh
- train-esci_3b.sh
- train_rec-amazon_c4_3b.sh
- train-esci_3b.sh
- train-esci_7b.sh
- train_rec-amazon_c4.sh
- train_rec-amazon_c4_3b.sh
- train_rec-amazon_review_3b.sh
- train_tiny_zero-7b.sh
- train_tiny_zero.sh
- train_tiny_zero_2gpus_stage1.sh
- train_tiny_zero_4gpus_stage2.sh
- train_tiny_zero_ppo.sh
- README.md
- amazon-c4-example.png
- blair.png
- kcore_filtering.py
- last_out_split.py
- README.md
- timestamp_split.py
- __init__.py
- models.py
- tool.py
- trainers.py
- base.sh
- large.sh
- README.md
- sample_pretraining_data.py
- test_load_checkpoints.py
- train.py
- process_esci.py
- bm25.py
- eval_search.py
- generate_emb.py
- README.md
- overall.yaml
- SASRecText.yaml
- UniSRec.yaml
- dataset.py
- process_amazon_2023.py
- sasrectext.py
- unisrec.py
- README.md
- run.py
- utils.py
- .gitignore
- LICENSE
- README.md
- convert_data_blair_format.py
- eval_search.py
- generate_emb.py
- subset_data.py
- data_preprocess.py
- subset_data.py
- subset_data_.py
- 1_filter_data.py
- 2_data_gen.py
- 2_self_instruct.py
- 3_data_org.py
- utils.py
- others.py
- test_subset.py
- new.py
- amazon_beauty.py
- subset_data_.py
- convert_to_json.py
- built_corpus.py
- process_amazon_2023.py
- subset_data.py
- subset_data.py
- process_esci.py
- split_category.py
- 1_doc_ids.py
- 2_build_index.py
- search.py
- 1_doc_ids.py
- 2_build_index.py
- search.py
- eval_inst.py
- gpt.py
- model_generate.py
- search_inst.py
- utils.py
- eval_aws.py
- eval_gpt.py
- eval_inst.py
- gpt.py
- model_generate.py
- claude.py
- gpt.py
- model_generate.py
- post_process.py
- amazon_c4.py
- esci.py
- amazon_c4.py
- amazon_review.py
- esci.py
- amazon_c4.py
- esci.py
- amazon_c4.py
- amazon_review.py
- utils.py
- 1_convert_format.py
- 2_build_database.sh
- search.py
- 1_convert_format.py
- 2_build_database.sh
- search.py
- 1_convert_format.py
- 2_build_database.sh
- search.py
- README.md
- utils.py
- data_construct.py
- dpo.py
- data_construct.py
- data_construct.py
- data_construct_no_json_no_reason.py
- data_construct_no_reason_with_json.py
- data_construct_with_reason_no_json.py
- sft.py
- claude_aws.py
- gpt.py
- gpt_azure.py
- config.json
- create_model_tokenizer.py
- generation_config.json
- model.safetensors
- tokenizer_config.json
- ray_trainer.yaml
- main_trainer.py
- README.md
- __init__.py
- task.py
- tokenizer.py
- __init__.py
- __init__.py
- check_results.py
- run_qwen_gsm8k_function_rm.sh
- run_qwen_gsm8k_function_rm_no_rmpad.sh
- run_qwen_gsm8k_model_rm.sh
- run_qwen_gsm8k_model_rm_no_rmpad.sh
- run_qwen_gsm8k_model_rm_seq_balance.sh
- run_qwen_gsm8k_model_rm_ulysses.sh
- run_ray_trainer.sh
- run_ray_trainer_rmpad.sh
- test_memory_buffers.py
- test_ops.py
- test_torch_functional.py
- test_transformer.py
- test_transformers_ulysses.py
- main.py
- client.py
- README.md
- run.sh
- server.py
- test_check_worker_alive.py
- test_colocated_workers.py
- test_data_transfer.py
- test_driverfunc_to_worker.py
- test_high_level_scheduling_api.py
- test_ray_local_envs.py
- test_rvdz.py
- test_worker_group_basics.py
- test_worker_group_torch.py
- run_fsdp_vllm.py
- test_vllm_hf_loader.py
- check_license.py
- test_import.py
- test_tensor_dict_utilities.py
- test_rl_dataset.py
- test_rm_dataset.py
- test_sft_dataset.py
- __init__.py
- __init__.py
- llama_loader.py
- llama_saver.py
- __init__.py
- parallel_attention.py
- parallel_decoder.py
- parallel_linear.py
- parallel_mlp.py
- parallel_rmsnorm.py
- __init__.py
- modeling_llama_megatron.py
- __init__.py
- __init__.py
- llama.py
- monkey_patch.py
- qwen2.py
- __init__.py
- README.md
- registry.py
- weight_loader_registry.py
- __init__.py
- worker.py
- worker_group.py
- __init__.py
- ray.py
- __init__.py
- decorator.py
- worker.py
- worker_group.py
- __init__.py
- base.py
- megatron.py
- version
- __init__.py
- __init__.py
- arg_utils.py
- config.py
- llm.py
- llm_engine_sp.py
- model_loader.py
- model_runner.py
- parallel_state.py
- tokenizer.py
- weight_loaders.py
- worker.py
- __init__.py
- arg_utils.py
- config.py
- dtensor_weight_loaders.py
- hf_weight_loader.py
- llm.py
- llm_engine_sp.py
- megatron_weight_loaders.py
- model_loader.py
- model_runner.py
- parallel_state.py
- spmd_gpu_executor.py
- tokenizer.py
- worker.py
- __init__.py
- arg_utils.py
- config.py
- dtensor_weight_loaders.py
- hf_weight_loader.py
- llm.py
- llm_engine_sp.py
- megatron_weight_loaders.py
- model_loader.py
- model_runner.py
- parallel_state.py
- spmd_gpu_executor.py
- tokenizer.py
- worker.py
- __init__.py
- arg_utils.py
- config.py
- dtensor_weight_loaders.py
- hf_weight_loader.py
- llm.py
- llm_engine_sp.py
- megatron_weight_loaders.py
- model_loader.py
- model_runner.py
- parallel_state.py
- spmd_gpu_executor.py
- tokenizer.py
- worker.py
- __init__.py
- __init__.py
- evaluation.yaml
- generation.yaml
- ppo_megatron_trainer.yaml
- ppo_trainer.yaml
- sft_trainer.yaml
- __init__.py
- core_algos.py
- ray_trainer.py
- __init__.py
- fsdp_sft_trainer.py
- main_eval.py
- main_generation.py
- main_ppo.py
- runtime_env.yaml
- __init__.py
- README.md
- rl_dataset.py
- rm_dataset.py
- sft_dataset.py
- __init__.py
- performance.py
- trajectory_tracker.py
- __init__.py
- aggregate_logger.py
- __init__.py
- memory.py
- optimizer.py
- optimizer_config.py
- pipeline_parallel.py
- sequence_parallel.py
- tensor_parallel.py
- __init__.py
- ray_backend.py
- __init__.py
- amazon_c4.py
- amazon_review.py
- esci.py
- amazon_c4.py
- esci.py
- __init__.py
- config.py
- distributed.py
- flops_counter.py
- fs.py
- fsdp_utils.py
- hdfs_io.py
- import_utils.py
- logging_utils.py
- megatron_utils.py
- memory_buffer.py
- model.py
- py_functional.py
- ray_utils.py
- seqlen_balancing.py
- tokenizer.py
- torch_dtypes.py
- torch_functional.py
- tracking.py
- ulysses.py
- version
- __init__.py
- base.py
- dp_actor.py
- megatron_actor.py
- __init__.py
- base.py
- dp_critic.py
- megatron_critic.py
- __init__.py
- reward_model.py
- __init__.py
- base.py
- __init__.py
- naive_rollout.py
- __init__.py
- vllm_rollout.py
- __init__.py
- base.py
- hf_rollout.py
- tokenizer.py
- __init__.py
- base.py
- fsdp_ulysses.py
- fsdp_vllm.py
- megatron_vllm.py
- __init__.py
- fsdp_workers.py
- megatron_workers.py
- __init__.py
- protocol.py
- .gitignore
- LICENSE
- README.md
- requirements.txt
- setup.py
# 설치 가이드
git clone https://github.com/linjc16/Rec-R1
깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.
cd Rec-R1
방금 내려받은 프로젝트 폴더 안으로 이동합니다.
2. 공식 설치 스크립트
쉬움 추천- Python 3 pip 명령어를 쓰려면 Python이 필요합니다.
pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121
PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.
pip3 install vllm==0.6.3 # or you can install 0.5.4, 0.4.2 and 0.3.1
PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.
pip3 install ray
PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.
pip3 install flash-attn --no-build-isolation
PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.
이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.
3. Docker
쉬움- Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
- Docker Desktop 컨테이너를 빌드하고 실행하려면 필요합니다. 설치 후 실행해서 백그라운드에 켜두세요.
docker build -f docker/Dockerfile.ngc.vllm -t rec-r1 .
Dockerfile을 기반으로 실행 가능한 이미지를 빌드합니다.
docker run -p 8080:80 rec-r1
빌드된 이미지를 실제 컨테이너로 실행합니다.
4. Python
쉬움pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121
PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.
pip3 install vllm==0.6.3 # or you can install 0.5.4, 0.4.2 and 0.3.1
PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.
pip3 install ray
PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.
pip install -e .
requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.
pip3 install flash-attn --no-build-isolation
PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.
이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.
