document_ai_agents - GitLite

KO | EN

gitlite — search

#javascript #react #go #machine-learning #rust #deep-learning #docker #macos #windows #kubernetes #vue #awesome-list #security #mcp

document_ai_agents

(★ 98)

A set of re-usable AI agent for document processing

파일 탐색기

최종 버전 다운로드 (.zip)

- - ci.yml
  - detect-secrets.yml
- docs.pdf
- __init__.py
- document_multi_tool_agent.py
- document_parsing_agent.py
- document_qa_agent.py
- document_rag_agent.py
- document_utils.py
- image_utils.py
- logger.py
- schema_utils.py
- tools.py
- agents_demo.ipynb
- eval_qa.py
- - test_document_multi_tool_agent.py
  - test_document_parsing_agent.py
  - test_document_qa_agent.py
  - test_document_rag_agent.py
  - test_document_utils.py
  - test_image_utils.py
  - test_schema_utils.py
  - test_tools.py
.env.example
.gitignore
.secrets.baseline
LICENSE
Makefile
pyproject.toml
README.md
requirements-dev.txt
requirements.txt
ruff.toml
setup.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/CVxTz/document_ai_agents

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd document_ai_agents

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

3. Make

보통

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Make Linux/macOS는 보통 기본 설치되어 있습니다. Windows는 별도 설치(예: MSYS2, WSL)가 필요합니다.

make

생성된 빌드 설정을 바탕으로 실제 컴파일을 진행해 실행 파일을 만듭니다.

✅ 에러 없이 끝나면 성공입니다. 생성된 실행 파일을 직접 실행해보세요.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Document Parsing and Retrieval Agent Pipeline

This repository contains a two-agent pipeline for processing, chunking, embedding, and retrieving information from unstructured documents, such as PDFs. The pipeline leverages large language models (LLMs), specifically **Gemini**, for document parsing and question answering, and **ChromaDB** for vector-based retrieval.

## Overview

The pipeline is designed to handle documents with various formats, such as tables, figures, images, and text. The two main steps are:

1. **Document Parsing and Chunking**: Extracts and summarizes key sections (tables, figures, text blocks) from each page of a PDF, leveraging Gemini's capabilities to process and understand mixed content.
   
2. **Document Retrieval and Question Answering**: Vectorizes the summarized chunks and stores them in a vector database (Chroma). When a question is posed, it retrieves the most relevant sections and generates an answer, providing contextual information (including page images) to improve accuracy.

### Features
- **Document parsing**: Extracts images from PDFs and splits the content into relevant sections.
- **Chunking and summarization**: Uses a LLM to categorize content into chunks (e.g., tables, figures, text).
- **Vector-based search**: Utilizes a vector store (ChromaDB) to index and retrieve relevant document chunks.
- **Question answering**: Allows querying of the document, with the response generated using the context of the most relevant chunks and images.

```bash
pip install -r requirements.txt
```

## Setup

### 1. Install and Set up Google Generative AI (Gemini)
The pipeline uses Gemini for both parsing the document and generating responses. You will need access to Gemini's API. Ensure that you have the necessary credentials to use the model.

### 2. Document Parsing Agent

This agent handles the parsing and chunking of the document. It extracts each page of the document as a JPEG image, splits the page into relevant chunks (tables, figures, text), and generates summaries for each chunk.

The relevant code is located in `document_parsing_agent.py`.

### 3. Document Retrieval and Question Answering Agent

This agent handles indexing the parsed document in a vector store and performing search and retrieval based on user queries. It uses ChromaDB to store the vectorized summaries of each chunk and retrieve the most relevant ones for generating answers.

The relevant code is located in `document_rag_agent.py`.

## Running the Code

1. **Parse and Chunk the Document**: The first agent will extract images, identify chunks, and summarize the content.

2. **Index the Documents**: The second agent will take the parsed document, embed the summaries, and index them in ChromaDB for retrieval.

3. **Ask Questions**: Once the document is indexed, you can query it by asking questions. The system will search for the most relevant chunks and provide an answer using the context from those chunks, along with the images.

### Example Usage

Check notebooks/agents_demo.ipynb

## Future Improvements

1. **Persistent Storage**: Currently, the vector store is in-memory using ChromaDB. In production, consider using persistent storage options like Pinecone or Weaviate.
2. **Real-Time Updates**: Enhance the system to support dynamic document updates (e.g., re-indexing on document modification).
3. **Advanced Retrieval**: Implement hybrid search techniques for better retrieval accuracy by combining dense and sparse embeddings.
4. **Support for More Document Formats**: Expand the system to handle other formats like Word or PowerPoint in addition to PDFs.

## License

This code is licensed under the MIT License. See the LICENSE file for more details.

## Data source

Sample PDF: https://raw.githubusercontent.com/SharifiZarchi/Introduction_to_Machine_Learning/main/Slides/Chapter_05_Natural_Language_Processing/04-LLM%26Adaptation/LLM%20%26%20Adaptation.pdf
Document VQA: https://rrc.cvc.uab.es/?ch=17&com=downloads or https://huggingface.co/datasets/lmms-lab/DocVQA