llm-structured-output-benchmarks

(โ˜… 190)

Benchmark various LLM Structured Output frameworks: Instructor, Mirascope, Langchain, LlamaIndex, Fructose, Marvin, Outlines, etc on tasks like multi-label classification, named entity recognition, synthetic data generation, etc.

File Explorer

Download Latest Version (.zip)
  • .gitignore
  • config.yaml
  • LICENSE
  • main.py
  • README.md
  • requirements.txt

# Installation Guide

1. Get the code
git clone https://github.com/stephenleo/llm-structured-output-benchmarks

Downloads the entire project code from GitHub to your computer.

cd llm-structured-output-benchmarks

Moves into the project folder you just downloaded.

2. Python

Easy Recommended
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.
- Multilabel classification: `python -m main generate-results`

Runs the Python script (or module).

- NER: `python -m main generate-results --task ner`

Runs the Python script (or module).

- Synthetic data generation: `python -m main generate-results --task synthetic_data_generation`

Runs the Python script (or module).

- Benchmarking test is run using synthetic data generated by running: `python -m data_sources.generate_dataset generate-multilabel-data`.

Runs the Python script (or module).

- The synthetic data is generated by sampling and combining rows from the base data to achieve multiple classes per row according to some distribution for num classes per row. See `python -m data_sources.generate_dataset generate-multilabel-data --help` for more details.

Runs the Python script (or module).

โœ… If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

Today's picks

Understand-Anything โ˜… 80,004

Graphs that teach > graphs that impress. Turn any code into an interactive knowledge graph

TypeScript
react โ˜… 247,532

The library for web and native user interfaces.

JavaScript
awesome-vue โ˜… 73,543

๐ŸŽ‰ A curated list of awesome things related to Vue.js
mermaid โ˜… 89,874

Generation of diagrams like flowcharts or sequence diagrams from text in a similar manner

TypeScript
Microsoft-Activation-Scripts โ˜… 187,792

Open-source Windows and Office activator featuring HWID, Ohook, TSforge, and Online KMS ac

Batchfile
JavaGuide โ˜… 157,804

Java ้ข่ฏ• & ๅŽ็ซฏ้€š็”จ้ข่ฏ•ๆŒ‡ๅ—๏ผŒ่ฆ†็›–่ฎก็ฎ—ๆœบๅŸบ็ก€ใ€ๆ•ฐๆฎๅบ“ใ€ๅˆ†ๅธƒๅผใ€้ซ˜ๅนถๅ‘ใ€็ณป็ปŸ่ฎพ่ฎกไธŽ AI ๅบ”็”จๅผ€ๅ‘

JavaScript
// repository documentation