HuggingFace-Datasets-Text-Quality-Analysis

KO | EN

gitlite — search

#typescript #golang #android #machine-learning #deep-learning #macos #java #windows #cli #awesome-list #security #ios #openai #open-source

HuggingFace-Datasets-Text-Quality-Analysis

(★ 54)

Retrieves parquet files from Hugging Face, identifies and quantifies junky data, duplication, contamination, and biased content in dataset using pandas

HuggingFace-Datasets-Text-Quality-Analysis 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Hugging Face Datasets Text Quality Analysis

![](https://raw.githubusercontent.com/onesuper/HuggingFace-Datasets-Text-Quality-Analysis/main/screenshot.png)

The purpose of this repository is to let people evaluate the quality of datasets on Hugging Face. It retrieves parquet files from Hugging Face, identifies the junk data, duplication, contamination, biased content, and other quality issues within a given dataset.

## Related Resources

* [Google Colab Notebook](https://colab.research.google.com/drive/1c8rWB2gtUrBHQcmmvA_NAXxc7Cexn1vM?usp=sharing)

## Running the app
### Instructions

1. Prerequisites
Note that the code only works `Python >= 3.9` and `streamlit >= 1.23.1`

```
$ conda create -n datasets-quality python=3.9
$ conda activate datasets-quality
```

2. Install dependencies
```
$ cd HuggingFace-Datasets-Text-Quality-Analysis
$ pip install -r requirements.txt
```

3. Run Streamlit application
```
python -m streamlit run app.py
```

### Need to know

When the dataset you download from Hugging Face is too large, running the application may exceed the memory of your machine and causes some errors. Sample the data or refer to some libraries that can run Pandas on a cluster, such as Xorbits, Dask.

## Todos

- [ ] Introduce more dimensions to evaluate the dataset quality
  - Another indicator of poor quality data is excessive repetition of certain words or phrases within a document (Gopher)
- [ ] Optimize the deduplication example using parallel computing technique
- [ ] More robust junk data detecting 
- [ ] Use a classifier to evaluate the quality of data
- [ ] Test the code with larger dataset in a cluster environment
- [ ] More data frame manipulation backend: e.g. Dask

HuggingFace-Datasets-Text-Quality-Analysis

HuggingFace-Datasets-Text-Quality-Analysis 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

HuggingFace-Datasets-Text-Quality-Analysis

HuggingFace-Datasets-Text-Quality-Analysis 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README