long_tail_knowledge

KO | EN

gitlite — search

#python #react #android #machine-learning #rust #macos #nodejs #java #windows #awesome #awesome-list #security #mcp #open-source

long_tail_knowledge

(★ 77)

Repo for the paper "Large Language Models Struggle to Learn Long-Tail Knowledge"

long_tail_knowledge 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Large Language Models Struggle to Learn Long-Tail Knowledge

This repo contains the code and data used for the analysis in [Large Language Models Struggle to Learn Long-Tail Knowledge](https://arxiv.org/abs/2211.08411)

## Pre-training Dataset Entities
As part of this project, we ran the [DBPedia Spotlight](https://www.dbpedia-spotlight.org) entity linker on large-scale LM pre-training datasets such as The Pile, C4, ROOTS, OpenWebText, and Wikipedia. The entity linking data is hosted on [HuggingFace Hub](https://huggingface.co/datasets/nkandpa2/pretraining_entities). To get this data, you can either download it from their web UI, download it with Python using the instructions from [here](https://huggingface.co/docs/huggingface_hub/how-to-downstream), or run `git clone https://huggingface.co/datasets/nkandpa2/pretraining_entities` to clone the HuggingFace repo (note that this requires git lfs to be installed to actually download the data).

### Format
There are five files:
- `the_pile_entity_map.npz`
- `c4_entity_map.npz`
- `roots_entity_map.npz`
- `openwebtext_entity_map.npz`
- `wikipedia_entity_map.npz`

Each file can be loaded with `numpy.load` and is a dictionary mapping from DBPedia URI strings to numpy arrays of pre-training dataset indices where that entity occurs.

## QA Dataset Entities
To analyze the effect of pre-training entities on QA performance, we entity link Natural Questions and Trivia QA. The QA entity linking data is hosted on [HuggingFace Hub](https://huggingface.co/datasets/nkandpa2/qa_entities). To get this data, you can either download it from their web UI, download it with Python using the instructions from [here](https://huggingface.co/docs/huggingface_hub/how-to-downstream), or run `git clone https://huggingface.co/datasets/nkandpa2/qa_entities` to clone the HuggingFace repo (note that this requires git lfs to be installed to actually download the data).

### Format
There are four files:
- `nq_train_entities.jsonl`
- `nq_validation_entities.jsonl`
- `trivia_qa_unfiltered.nocontext_train_entities.jsonl`
- `trivia_qa_unfiltered.nocontext_validation_entities.jsonl`

The Natural Questions files are in the order of the [nq_open](https://huggingface.co/datasets/nq_open) dataset and the Trivia QA files are in the order of the `unfiltered.nocontext` split of the [trivia_qa](https://huggingface.co/datasets/trivia_qa) dataset.

These are each jsonlines files. Each line in the file is a dictionary with the following structure:
```
{   
    'q_entities': <list of entities found in the question>,
    'a_entities': <list of entities found in the answer aliases>
}
```
The entities in the `q_entities` and `a_entities` lists are also dictionaries with the structure:
```
{
    'URI': <dbpedia entity URI>,
    'support': <dbpedia support>,
    'types': <dbpedia types>,
    'surfaceForm': <string in Q or A that was linked to this entity>,
    'offset': <location in Q or A where surface form is found>,
    'similarityScore': <dbpedia similarity score>,
    'percentageOfSecondRank': <dbpedia percentage of second ranked entity>
}
```
## Code
- The code and instructions for running the entity linker can be found in `entity_linking/`
- The code and instructions for running the relevant document counting heuristic can be found in `relevant_docs/`

long_tail_knowledge

long_tail_knowledge 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

long_tail_knowledge

long_tail_knowledge 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README