GUR

KO | EN

gitlite — search

#python #hacktoberfest #typescript #llm #golang #android #docker #java #awesome #cli #kubernetes #vue #security #open-source

(★ 14)

“Generate to Understand for Representation”

GUR Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Generate to Understand for Representation

This repository contains the code and models discussed in our paper "Generate to Understand for Representation"(https://arxiv.org/abs/2306.10056).

> Introducing GUR: a pretraining framework that combines language modeling and contrastive learning objectives in a single training step. We select similar text pairs based on their Longest Common Substring (LCS) from raw unlabeled documents and train the model using masked language modeling and unsupervised contrastive learning. The resulting model, GUR, achieves impressive results without any labeled training data, outperforming all other pretrained baselines as a retriever at the recall benchmark in a zero-shot setting. Additionally, GUR maintains its language modeling ability, as demonstrated in our ablation experiment.

## Architecture
![Model Architecture](https://raw.githubusercontent.com/laohur/GUR/main/Architecture.PNG)

## generate samples
`python sents2pair.py` convet corpus to pairs

## train
`bash train.sh`

## init model(optional)
`python convert.py`

## Citation
```bib
@INPROCEEDINGS{10438270,
  author={Xue, Changshang and Zhong, Xiande and Liu, Xiaoqing},
  booktitle={2023 11th International Conference on Information Systems and Computing Technology (ISCTech)}, 
  title={Generate to Understand for Representation in One Pre-training Stage}, 
  year={2023},
  volume={},
  number={},
  pages={258-267},
  keywords={Training;Computational modeling;Self-supervised learning;Benchmark testing;Market research;Natural language processing;Task analysis;self-supervised pre-train;contrastive learning;language model;zero-shot learning;text representation;NLP;NLU;NLG;retrieval},
  doi={10.1109/ISCTech60480.2023.00054}}

```

GUR

GUR Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

GUR

GUR Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README