KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agents #developer-tools #codex #claude-code #mcp #rust #deepseek-harness #typescript #ai-agent #cli

deepsearch-examples

★ 90 Open GitHub ↗

Examples using the Deep Search functionalities

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Deep Search Examples

![Python Version](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10-blue)
[![Jupyter](https://img.shields.io/static/v1?label=jupyter&message=py&color=orange&logo=jupyter)](https://mybinder.org/v2/gh/DS4SD/deepsearch-examples/main)
[![License MIT](https://img.shields.io/github/license/ds4sd/deepsearch-toolkit)](https://opensource.org/licenses/MIT)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Docs](https://img.shields.io/badge/website-live-brightgreen)](https://ds4sd.github.io/deepsearch-toolkit/)
[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/DS4SD/deepsearch-examples/main)

In this repository we showcase some common usage of Deep Search
for **Document conversion** as well as **Data and Knowledge exploration**.

## Quick links

- [Deep Search Toolkit](https://github.com/ds4sd/deepsearch-toolkit)
- [Documentation](https://ds4sd.github.io/deepsearch-toolkit/)

## Examples

### Setup and usage

Each example starts by defining its input parameters. This is supported by Pydantic
Settings, allowing automated loading from a `.env` file or env vars. Furthermore, access
is based on [Profiles][profiles]. Unless otherwise configured, the profile used is the active one.

### Document conversion

| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [Convert documents quick start][doc_conv_quick_start] | Full example on programmatic document conversion [<img src=".readme_resources/doc_conversion_quick_start.png" width="300px" />][doc_conv_quick_start] |
| 2. | [Convert documents with custom settings][doc_conv_custom_pipeline] | Full example on programmatic document conversion with custom conversion settings [<img src=".readme_resources/doc_conversion_quick_start.png" width="300px" />][doc_conv_custom_pipeline] |
| 3. | [Visualize bounding boxes][doc_conv_visualize_bbox] | Visualize the bbox of the text elements [<img src="./.readme_resources/visualize_bbox.png" width="300px" />][doc_conv_visualize_bbox] |
| 4. | [Extract figures from documents][doc_conv_extract_figures] | Given a PDF file, extract the figures [<img src=".readme_resources/extract_figures.png" width="300px" />][doc_conv_extract_figures] |
| 5. | [Extract tables][doc_conv_extract_tables] | Given a PDF file, extract the tables [<img src=".readme_resources/extract_tables.png" width="300px" />][doc_conv_extract_tables] |

### NLP on Documents

|    | Name              | Description |
| -- | ----------------- | ----------- |
| 1. | [NLP on documents](examples/nlp_on_documents/nlp_on_documents.ipynb)* | A few quick examples on how to apply NLP models on documents (eg extracting key-terms) |
| 2. | [Reference Parsing](examples/nlp_for_references/nlp_for_references.ipynb)* | Examples on how to parse references from Documents |
| 3. | [Material Extraction](examples/nlp_for_materials/nlp_for_materials.ipynb)* | Examples on how to extract materials from Documents |

### Data queries

This section will showcase examples which query data processed via Deep Search.

| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [Data query quick start][data_query_quick_start] | Example listing data collections, making search in one and more document collections, using source for projection [<img src=".readme_resources/data_query_quick_start.png" width="300px" />][data_query_quick_start] |
| 2. | [Chemistry search queries][data_query_chemistry] | Search the chemistry databases for known molecules [<img src=".readme_resources/data_query_chemistry.png" width="300px" />][data_query_chemistry] |
| 3. | [Chemistry and patent searches via PatCID][data_query_chemistry_patcid] | Explore the chemistry databases using substructure and similarity searches and navigate to the world-wide patents which reference molecules [<img src=".readme_resources/data_query_chemistry_patcid.png" width="300px" />][data_query_chemistry] |
| 4. | [Snippets and aggregations in data queries][data_query_snippets] | Extract snippets in search queries and leverage aggregations for exploratory analysis [<img src=".readme_resources/data_query_snippets.png" width="300px" />][data_query_snippets] |

### Semantic queries & RAG

This section will showcase examples of semantic capabilitilies in the area of Q&A using RAG.

| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [QA quick start][qa_quick_start] | Get started with semantic ingestion, RAG, and retrieval. [<img src=".readme_resources/documentqa.png" width="300px" />][qa_quick_start] |
| 2. | [QA deep dive][qa_deep_dive] | Explore advanced RAG and semantic retrieval capabilities. [<img src=".readme_resources/qa_doc_collection.png" width="300px" />][qa_deep_dive] |

### Bring your own

This section will showcase examples for bringing your own documents, csv data, nlp models and more.

| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [Bring your own PDF][bring_your_own_pdf] | Upload your own PDF documents, search on them and export the result as JSON files. [<img src=".readme_resources/bring_your_own_pdf.png" width="300px" />][bring_your_own_pdf] |
| 2. | [Bring your own converted documents][bring_your_own_converted_documents] | Upload your documents already formatted as JSON. [<img src=".readme_resources/bring_your_own_converted_documents.png" width="300px" />][bring_your_own_converted_documents] |
| 3. | Bring your own DataFrame | Bring your own DataFrame from CSV, XLSX, etc and explore the content in a knowledge graph |

### Attachments and metadata

This section will showcase examples for managing index item attachments and metadata.

| | Name | Description |
| -- | ----------------- | ----------- |
| 1. | [Manage attachments][manage_attachments] | Manage index item attachments [<img src=".readme_resources/manage_attachments.png" width="300px" />][manage_attachments] |

### Knowledge graphs

This section will showcase examples related to the use of knowledge graphs (KGs) in Deep Search.

| | Name | Description |
| -- | ------------------ | ----------- |
| 1. | [Using Deep Search KGs with PyTorch Geometric][kg_download_quick_start] | Download knowledge graphs from Deep Search and import them in PyTorch Geometric. [<img src=".readme_resources/kg_download_quick_start.png" width="300px" />][kg_download_quick_start] |

### Integrations

This section will showcase examples related to the integration of Deep Search with other tools and utilities.

| | Name | Description |
| -- | ------------------ | ----------- |
| 1. | [Annotations on argilla.io][integration_argilla] | Use [argilla.io](https://argilla.io/) for annotating the content of documents. [<img src=".readme_resources/integration_argilla.png" width="300px" />][integration_argilla] |

## Example dependencies

The examples contained in this catalog depend on the `deepsearch-toolkit` as well as
other modules needed for the showcase demonstrated (e.g. `pandas`, `matplotlib`, `rdkit`, etc).
Please refer to the poetry `pyproject.toml` or  `requirements.txt` for a complete list.

Python dependencies are installed with

```console
pip install -r requirements.txt
```

Additionally, some examples rely on system packages. When this is the case, the README of the individual
example will contain more details on which package is required.
The auxiliary file [apt.txt](./apt.txt) list all such packages for a Debian-bases OS. They can be installed with

```console
xargs sudo apt-get install < apt.txt
```

### Windows compatibility

Note that some examples require dependencies that are not available on Windows platform. We flagged those examples with an asterisk `*` in the index above.

## License

The `Deep Search Toolkit` codebase is under MIT license.
For individual model usage, please refer to the model licenses found in the original packages.

[doc_conv_quick_start]: ./examples/document_conversion_quick_start/
[doc_conv_custom_pipeline]: ./examples/document_conversion_custom_settings/
[doc_conv_visualize_bbox]: ./examples/document_conversion_visualize_bbox/
[doc_conv_extract_figures]: ./examples/document_conversion_extract_figures/
[doc_conv_extract_tables]: ./examples/document_conversion_extract_tables/
[data_query_quick_start]: ./examples/data_query_quick_start/
[data_query_snippets]: ./examples/data_query_snippets/
[data_extract_figures]: ./examples/data_extract_figures/
[data_query_chemistry]: ./examples/data_query_chemistry/
[data_query_chemistry_patcid]: ./examples/data_query_chemistry_patcid/
[qa_quick_start]: ./examples/qa/qa_quick_start.ipynb
[qa_deep_dive]: ./examples/qa/qa_deep_dive.ipynb
[qa_doc_collection]: ./examples/qa_doc_collection
[bring_your_own_pdf]: ./examples/bring_your_own_pdf/
[bring_your_own_converted_documents]: ./bring_your_own_converted_documents/
[manage_attachments]: ./examples/attachment_management/
[kg_download_quick_start]: ./examples/kg_download_quick_start/
[integration_argilla]: ./examples/integration_argilla/
[json_export]: ./examples/json_export/
[profiles]: https://ds4sd.github.io/deepsearch-toolkit/#set-up-your-profile

deepsearch-examples

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README