RouterArena
RouterArena: An open framework for evaluating LLM routers with standardized datasets, metrics, an automated framework, and a live leaderboard.
File Explorer
Download Latest Version (.zip)- config.yaml
- pr-evaluation.yml
- pre-commit.yml
- sync-website.yml
- process_pr_submission.py
- claude-3-haiku-20240307.jsonl
- gemini-2.0-flash-001.jsonl
- gpt-4o-mini.jsonl
- AIME.json
- ArcMMLU.json
- AsDiv.json
- ChessInstruct.json
- ChessInstruct_mcq.json
- Ethics_commonsense.json
- Ethics_deontology.json
- Ethics_justice.json
- Ethics_virtue.json
- FinQA.json
- GeoBench.json
- GeoGraphyData.json
- GPQA.json
- GSM8K.json
- LiveCodeBench.json
- MATH.json
- MathQA.json
- MedMCQA.json
- MMLU.json
- MMLUPro.json
- MusicTheoryBench.json
- NarrativeQA.json
- OpenTDB.json
- PubMedQA.json
- QANTA.json
- SocialiQA.json
- SuperGLUE-CausalReasoning.json
- SuperGLUE-ClozeTest.json
- SuperGLUE-Entailment.json
- SuperGLUE-QA.json
- SuperGLUE-RC.json
- SuperGLUE-Wic.json
- SuperGLUE-Wsc.json
- WMT19-cs-en.json
- WMT19-de-en.json
- WMT19-fi-en.json
- WMT19-gu-en.json
- WMT19-kk-en.json
- WMT19-lt-en.json
- WMT19-ru-en.json
- WMT19-zh-en.json
- azure.json
- gpt5.json
- nadir.json
- notdiamond.json
- __init__.py
- robustness.py
- leaderboard.png
- pipeline.png
- routerarena-diagram.png
- routerarena_logo_v2.png
- batch_evaluate.py
- enhanced_extractor.py
- eval_reasoning.py
- evaluate_models.py
- livecodebench_util.py
- metric_utils.py
- metrics.py
- parallel_evaluation.py
- run.py
- utils.py
- batch_inference.py
- main.py
- main_utils.py
- model_inference.py
- parallel_inference.py
- pipeline.py
- README.md
- run.py
- shared_utils.py
- model_cost.json
- compute_scores.py
- agentforge-router.json
- AsiaInfo-Router.json
- auto_router.json
- azure-model-router.json
- BARouter.json
- chayan.json
- chuzom-solo-v32.json
- cross-router.json
- divyam-router.json
- glm-4-air-router.json
- hybrid-router.json
- llm-router.json
- lynkr.json
- nadir-caliper.json
- nadir-cascade-v2.json
- nadir-tumbler.json
- openrouter-router.json
- orcarouter-adaptive.json
- Paix2-router.json
- r2-router.json
- sqwish-fvcta8.json
- sqwish-router.json
- vllm-sr.json
- weave-router.json
- your-router.json
- agentforge-router-robustness.json
- agentforge-router.json
- AsiaInfo-Router-robustness.json
- AsiaInfo-Router.json
- auto_router-robustness.json
- auto_router.json
- azure-model-router-robustness.json
- azure-model-router.json
- BARouter-robustness.json
- BARouter.json
- chayan.json
- chuzom-solo-v32-robustness.json
- chuzom-solo-v32.json
- cross-router-robustness.json
- cross-router.json
- divyam-router-robustness.json
- divyam-router.json
- glm-4-air-router.json
- hybrid-router-robustness.json
- hybrid-router.json
- llm-router-robustness.json
- llm-router.json
- lynkr-robustness.json
- lynkr.json
- nadir-caliper-robustness.json
- nadir-caliper.json
- nadir-cascade-v2-robustness.json
- nadir-cascade-v2.json
- nadir-tumbler-robustness.json
- nadir-tumbler.json
- orcarouter-adaptive-robustness.json
- orcarouter-adaptive.json
- Paix2-router-robustness.json
- Paix2-router.json
- r2-router-robustness.json
- r2-router.json
- sqwish-fvcta8-robustness.json
- sqwish-fvcta8.json
- sqwish-router-robustness.json
- sqwish-router.json
- vllm-sr-robustness.json
- vllm-sr.json
- weave-router-robustness.json
- weave-router.json
- __init__.py
- auto_router.py
- base_router.py
- chuzom_solo_v32.py
- example_router.py
- llm_router.py
- lynkr_router.py
- nadir_adapter.py
- NADIR_NOTES.txt
- vllm_sr.py
- check_config_prediction_files.py
- compare_router_accuracy.py
- generate_prediction_file.py
- prep_datasets.py
- build_site_data.py
- prepare_gpqa_data.py
- rescore_affected.sh
- addlicense.sh
- audit_token_accounting.py
- mypy.sh
- .env.example
- .gitignore
- .license-header.txt
- .pre-commit-config.yaml
- .python-version
- leaderboard_manifest.yaml
- LICENSE
- pyproject.toml
- README.md
- universal_model_names.py
- uv.lock
// repository documentation
Was this content helpful?
(0 ratings)