DISKOS-QA

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 10 Open GitHub ↗

DISKOS-QA: Generative pipeline for LLM benchmark development, based on open source reports from Norwegian DISKOS data.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DISKOS-QA
An open source benchmark for subsurface disciplines developed by the FORCE consortium with participation from Equinor, AkerBP, Conoco Phillips, DNO, OMV, Inpex Idemitsu, and Søkkeldirektoratet.

https://www.sodir.no/en/force/, https://www.equinor.com/, https://akerbp.com/en/, https://www.conocophillips.no/, https://www.dno.no/, https://www.omv.no/, https://inpex-idemitsu.no/, https://www.sodir.no/

The project builds high-quality question–answer pairs from a released DISKOS dataset, using a Neo4j knowledge graph in combination with Azure OpenAI + DeepEval, and ships a Streamlit tool for human feedback.

Based on an existing language dataset released by Force in a Zenodo repository.

Link to Force dataset on Zenodo https://zenodo.org/records/10775273

## Pipeline overview
<img width="100%" height="100%" alt="image" src="https://github.com/user-attachments/assets/75a3fce5-32bf-4c04-bf68-ea36c87813e5" />

## Topic distribution
![Untitled_Report (9)](https://github.com/user-attachments/assets/d1008532-64ce-463d-a604-ddb789ddac22)

## What you can do
- Pull contextual chunks from the DISKOS Neo4j graph filtered by wells, fields, or formations and topic embeddings.
- Synthesize QA pairs with ordered DeepEval evolutions and Azure OpenAI models (closed or multiple-choice styling).
- Score each pair with DeepEval metrics plus custom checks (clarity, instruction adherence, hallucination, closed-book).
- Run the pipeline in DataHandler-driven batches with CSV/Excel exports for accepted and discarded pairs.
- Optionally run post-eval metrics (e.g., Faithfulness) on accepted pairs only.
- Collect human feedback through the Streamlit app backed by Supabase (see `streamlit_app/README.md`).

## Getting started
### Prerequisites
- Python 3.11+ recommended (project requires >=3.8).
- Access to Azure OpenAI deployments for embedding, generation, and critic models.
- Access to the DISKOS Neo4j instance populated with chunk embeddings.
- (Optional) Supabase project for the feedback tool.

### Install
```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip setuptools wheel
pip install -e .
```

### Environment variables
Create a `.env` (or export in your shell) with at least:

| Purpose | Keys |
| --- | --- |
| Azure OpenAI models | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_API_VERSION`, `AZURE_OPENAI_MODEL_ENDPOINT` |
| Azure OpenAI embeddings | `AZURE_OPENAI_EMBEDDER_ENDPOINT`, `AZURE_OPENAI_EMBEDDER_NAME` |
| Neo4j connection | `NEO4J_URI`, `NEO4J_USERNAME`, `NEO4J_PASSWORD` |
| (Feedback app) Supabase | `SUPABASE_URL`, `SUPABASE_SERVICE_KEY` |

Model deployments default to `gpt-5-mini` for generation, `o4-mini` for critic/judge, and `text-embedding-ada-002` for embeddings; adjust in `ForceQA/models/azure_models.py` if needed.

## Repo map
- `ForceQA/pipeline`: Orchestration (`QA_pipeline.py`), DataHandler-driven batching, context building, result collection.
- `ForceQA/synthesizer`: DeepEval-based QA synthesis with styling configs for closed and multiple-choice questions.
- `ForceQA/eval`: DeepEval evaluator wiring and custom metrics/templates.
- `ForceQA/graph`: Neo4j connectivity and chunk retrieval filtered by topics (`ForceQA/utils/topics.py`).
- `ForceQA/embedding`: Embedding helpers for chunk files and batched Azure embedding calls.
- `streamlit_app`: Human-in-the-loop feedback UI (see its README for setup).
- `examples`: Notebooks and scripts demonstrating pipeline use.
- `benchmark`: Final public benchmark (`benchmark/dataset/FORCE_QA_public.csv` with question, answer, context, metadata columns) plus scripts/notebooks for compilation.
- `NEO4J`: ingestion and prep assets for the DISKOS Neo4j graph—Cypher loaders, sampling/notebooks, raw chunk/well/field CSVs, and NER utilities (GLiNER/SpaCy modules plus prep scripts and processed lookup tables).
- `data`: Example data from the DISKOS dataset.
- `tests`: Tests for the ForceQA package, which can with run with `pytest` from the project root:
  ```bash
  pytest
  ```

## Examples and notebooks
- `examples/notebooks/neo4j_example.ipynb`: Query Neo4j for chunks by target and topic, verify env and data presence.
- `examples/notebooks/neo4j_synthesizer_integration.ipynb`: Pull contexts from Neo4j and pass them into the synthesizer end to end.
- `examples/notebooks/synthesizer_example.ipynb`: Run ordered DeepEval evolutions with `FORCE_Synthesizer`.
- `examples/notebooks/pipeline_example.ipynb`: Full QA pipeline for closed and multiple-choice generation, batch runs, and result inspection.

## Pipeline API (current)
- `QA_Pipeline.run_pipeline(...)` is the public entry point and expects a `DataHandler` (e.g., `FORCE_DataHandler` or `ListDataHandler`).
- Batching is handled via `DataHandler.batches(...)` and results are aggregated across batches.
- Post-eval runs only on accepted QA pairs and is stored as `post_eval__*` flags in the final results DataFrame.

## Tips
- Topic names must match keys in `ForceQA/utils/topics.py`; embeddings are cached in `ForceQA/utils/pkg_data/topic_embeddings.json`.
- To change model temperature, deployments, or thresholds, edit `ForceQA/models/azure_models.py` and `ForceQA/eval/deepeval_evaluator.py`.
- Exports: `QA_pipeline.run_pipeline` returns three dataframes (all, accepted, discarded) and writes final CSV/Excel outputs for the full run.

## License
NLOD 2.0

https://data.norge.no/nlod/en/2.0

DISKOS-QA

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README