KO
|
EN
gitlite — search
Search
#python
#java
#python3
#arduino
#golang
#machine-learning
#rust
#html
#flask
#javascript
#seismology
#nodejs
deduplicate
★ 18
Open GitHub ↗
No description available.
Download README (.md)
Explore Similar Repositories
research-note-wrap
:
Turn research, triage, and analysis sessions into reusable Markdown conclusion notes for maintainers.
AIEKit
:
AI Agents skills for AI Engineer
ritual
:
An open source and minimal habit tracker
plottter
:
A free, open-source desktop app for generating pen-plotter-ready vector art from math equations, generative algorithms, and raster images.
xuzilong-skills
:
This is a skill repository for AI agent. And it is developed especially for Economic and Management academic research.
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
deduplicate
?
Download (.md)
<div align="center"> <h1><code>deduplicate_lib</code></h1> <p><i>deduplication algorithms in python</i></p> </div> *** [](https://github.com/julianholland/deduplicate) [](https://codecov.io/gh/julianholland/deduplicate) [](https://badge.fury.io/py/deduplicate_lib) [](https://www.python.org/downloads/) [](https://opensource.org/licenses/MIT) [](https://github.com/julianholland/deduplicate/actions/workflows/ci.yml) [](https://github.com/astral-sh/ruff) [](https://deduplicate-lib.readthedocs.io/en/latest/) ## Key Features - Easy to use deduplication algorithms for any vector array - Suite of tolerance tuning algorithms to help you find the right tolerance value for your system - Suite of benchmarking tools to ensure rigor, accuracy, and speed (not yet implemented) - Factory Plugin architecture, for easy extensibility and modification *** ## Implemented Algorithms - Distance Matrix (Simple, accurate, expensive): Computes the distance matrix for all vectors and determines duplicates by finding those that fall below a given distance - Multi Hashing (Fast): Smears and rounds the vectors using a normal distribution and computes the hashes for each which are then used to determine duplicates by proportion of hash clashes. <!-- - Locality Sensitive Hashing (Fast, Accurate) --> ## Quick Start ```bash pip install deduplicate_lib ``` ```python from deduplicate_lib.plugins.duplicate_detection_algorithms.multi_hashing import MultiHashing import numpy as np dataset = np.array([[1.0, 2.0], [1.01, 2.01], [5.0, 6.0]]) dda = MultiHashing(tolerance=0.1, dataset_array=dataset) # return unique vectors print(dda.deduplicate()) # check a single vector against the dataset dda.input_vector = np.array([1.0, 2.0]) print(dda.duplicate_check()) # True ``` See the **[full documentation](https://deduplicate-lib.readthedocs.io)** for API reference, tolerance tuning, and more examples. ### Dependencies - Python 3.9+ - `numpy` - `numba` - `scipy` ## 🤝 Contributing We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details. ### Development Setup ```bash # Clone the repository git clone https://github.com/julianholland/deduplicate.git cd deduplicate # Install in development mode pip install -e ".[dev]" # Run tests pytest # Run linting ruff check . ruff format . ``` ### Running Tests ```bash # Run all tests pytest # Run specific test categories pytest tests/core/ pytest tests/plugins/ pytest tests/plugins/duplicate_detection_algorithms/distance_matrix # Run with coverage pytest --cov ``` ## 📝 Citation If you use deduplicate_lib in your research, please cite: ```bibtex @software{deduplicate2026, title={deduplicate_lib: Auto Tolerance Finding Deduplication Algorithms in Python}, author={Julian Holland}, year={2026}, url={https://github.com/julianholland/deduplicate}, version={0.0.5} } ``` ## 📄 License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. ## 🙏 Acknowledgments - The Fritz Haber Institute - Juan Manuel Lombardi <3 - Maximillion Ach - Chiara Panosetti ## Project Links - [GitHub Repository](https://github.com/julianholland/deduplicate) - [Documentation](https://deduplicate-lib.readthedocs.io) ## Project To-Do - [x] Add example.ipynb - [x] Create general Pre-allocation protocal - [ ] Add benchmarks for time and robustness - [ ] Add Locality-Sensitive Hashing as an option - [x] Speedup slow tasks with Numba - [x] Set up Read the Docs - [x] Create general deduplicate function - [x] Speed up NTPP
