promptrepo-score

KO | EN

gitlite — search

#python #hacktoberfest #ai #typescript #machine-learning #deep-learning #linux #nodejs #windows #awesome #cli #awesome-list #security #ios

promptrepo-score

(★ 15)

Calculate confidence score for structured output generated by LLMs like OpenAI

promptrepo-score Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# @promptrepo/score

Calculate confidence scores for OpenAI JSON outputs. Transform black box responses into reliable data with granular field-level scoring.

## Why @promptrepo/score?

- 📊 **Precise Scoring**: Convert OpenAI's logprobs into confidence metrics for each JSON field
- 🚀 **Simple Integration**: Works directly with OpenAI's completion and chat endpoints
- ⚡ **Lightweight**: Zero dependencies for scoring logic

## Installation

```bash
npm install @promptrepo/score
npm install openai  # Required for OpenAI API examples
```

## Model Requirements

This package requires OpenAI models that support both logprobs and structured JSON output:
- For chat completion: `gpt-3.5-turbo` or `gpt-4` or `gpt-4o`
- For completion: `gpt-3.5-turbo-instruct`

## Basic Usage

### With OpenAI Chat Completion
```javascript
import { calculateConfidenceScores } from '@promptrepo/score';
import OpenAI from 'openai';

const openai = new OpenAI({
    apiKey: 'sk-...' // Replace with your OpenAI API key
});

// Make an API call
const response = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [
        {
            role: "system",
            content: "You are a helpful assistant that returns JSON responses."
        },
        {
            role: "user",
            content: "Extract product information from: 'Product: Premium Widget, Price: $29.99'"
        }
    ],
    response_format: { type: "json_object" },
    logprobs: true,
    max_tokens: 500,
    temperature: 0
});

// Parse JSON response
const jsonOutput = JSON.parse(response.choices[0].message.content);

// Raw OpenAI output
console.log('OpenAI output:', jsonOutput);
// {
//   "Product name": "Premium Widget",
//   "Price": "29.99"
// }

// Calculate confidence scores
const result = calculateConfidenceScores(jsonOutput, response.choices[0].logprobs.content);

// @promptrepo/score output with confidence scores
console.log('Confidence scores:', result);
// {
//   "Product name": { value: "Premium Widget", score: 0.95 },
//   "Price": { value: "29.99", score: 0.92 }
// }
```

### Nested Structures
```javascript
import { calculateConfidenceScores } from '@promptrepo/score';
import OpenAI from 'openai';

const openai = new OpenAI({
    apiKey: 'sk-...' // Replace with your OpenAI API key
});

// Make an API call
const response = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [
        {
            role: "system",
            content: "You are a helpful assistant that returns JSON responses with nested product information."
        },
        {
            role: "user",
            content: "Extract detailed product information from: 'Product: Premium Widget, Price: $29.99, Weight: 1.5kg, Dimensions: 10x20x30cm'"
        }
    ],
    response_format: { type: "json_object" },
    logprobs: true,
    max_tokens: 500,
    temperature: 0
});

// Parse JSON response
const jsonOutput = JSON.parse(response.choices[0].message.content);

// Raw OpenAI output
console.log('OpenAI output:', jsonOutput);
// {
//   product: {
//     details: {
//       name: "Premium Widget",
//       price: "29.99"
//     },
//     specifications: {
//       weight: "1.5kg",
//       dimensions: "10x20x30cm"
//     }
//   }
// }

// Calculate confidence scores
const result = calculateConfidenceScores(jsonOutput, response.choices[0].logprobs.content);

// @promptrepo/score output with confidence scores
console.log('Confidence scores:', result);
// {
//   product: {
//     value: {
//       details: {
//         value: {
//           name: { value: "Premium Widget", score: 0.95 },
//           price: { value: "29.99", score: 0.92 }
//         },
//         score: 0.93
//       },
//       specifications: {
//         value: {
//           weight: { value: "1.5kg", score: 0.94 },
//           dimensions: { value: "10x20x30cm", score: 0.91 }
//         },
//         score: 0.92
//       }
//     },
//     score: 0.92
//   }
// }
```

promptrepo-score

promptrepo-score Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

promptrepo-score

promptrepo-score Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README