KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #typescript #ai-agent #developer-tools #claude-code #codex #deepseek-harness #ai-agents #react #mcp #rust

ArtSearch

★ 14 Open GitHub ↗

A local search system implementation using Elasticsearch for Wikipedia data indexing and retrieval.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ArtSearch 🔍 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A local search system implementation using Elasticsearch for Wikipedia data indexing and retrieval.

## Table of Contents

- [ArtSearch 🔍 ](#artsearch--)
  - [Table of Contents](#table-of-contents)
  - [Features ✨](#features-)
  - [Prerequisites 🛠️](#prerequisites-️)
  - [Installation ⚙️](#installation-️)
    - [1. Download Elasticsearch Engine](#1-download-elasticsearch-engine)
    - [2. Download Wikipedia Data](#2-download-wikipedia-data)
  - [Usage 🚀](#usage-)
    - [Folder Structure](#folder-structure)
    - [Building Index](#building-index)
    - [Performing Searches](#performing-searches)
  - [Configuration ⚙️](#configuration-️)
  - [Development 🧑💻](#development-)
  - [Contributing 🤝](#contributing-)
  - [License 📄](#license-)

## Features ✨

- Multi-language support for Wikipedia data
- Elasticsearch-powered search backend
- CLI interface for index management and queries
- Configurable search parameters

## Prerequisites 🛠️

- Python 3.12
- Java 11+ (for Elasticsearch)
- 30GB+ free disk space (for data storage)

## Installation ⚙️

### 1. Download Elasticsearch Engine

```bash
# Create data directory
mkdir -p data && cd data

# Download and extract Elasticsearch
wget -O elasticsearch-8.17.3.tar.gz \
  https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.17.3-linux-x86_64.tar.gz
tar zxvf elasticsearch-8.17.3.tar.gz
rm elasticsearch-8.17.3.tar.gz
cd ..
```

### 2. Download Wikipedia Data

Download specific language version of Wikipedia dataset:

```bash
# Default English dataset (November 2023)
modelscope download --dataset wikimedia/wikipedia \
  --include 20231101.en/* \
  --local_dir ./data/wikipedia
```

Example data structure:

```json
{
    "id": "1",
    "url": "https://simple.wikipedia.org/wiki/April",
    "title": "April",
    "text": "April is the fourth month..."
}
```

## Usage 🚀

### Folder Structure

```
├── data                      # Data folder
│   ├── elasticsearch-8.17.3  # Elasticsearch engine
│   └── wikipedia             # Wikipedia data folder
│       ├── 20231101.en       # English data
│       ├── 20231101.zh       # Chinese data
│       ...                   # more language data
│
├── es_wiki_build.py          # Scripts for build wiki index
├── es_wiki_test.py           # Unit test for elasticsearch
├── README.md
├── requirements.txt
└── wiki_searcher.py          # Search client for wiki data
```

### Building Index

```bash
# Build index for default language (en)
python es_wiki_build.py

# Build index for specific language (e.g., French)
python es_wiki_build.py --language fr
```

### Performing Searches

```bash
# Default search setting
python es_wiki_test.py

# Direct query execution
python es_wiki_test.py \
  --language en \
  --query "Paris 2024 Olympic Games" 
```

## Configuration ⚙️

Setting environment variables for Elasticsearch configuration:

```bash
export ELASTIC_PASSWORD="changeme"
```

<!-- Modify config.yaml for custom settings:

```yaml
elasticsearch:
  host: localhost
  port: 9200
  index_prefix: "wiki_"
languages: [en, fr, es]
batch_size: 500
``` -->

## Development 🧑💻

```bash
# Install dependencies
pip install -r requirements.txt
```

## Contributing 🤝

We welcome contributions! Please follow these steps:

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

## License 📄

This project is licensed under the MIT License - see the [LICENSE](./LICENSE) file for details.

ArtSearch

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README