spacy-universal-sentence-encoder - GitLite

KO | EN

gitlite — search

#ai #typescript #go #android #machine-learning #nodejs #java #windows #awesome #kubernetes #vue #pytorch #mcp #open-source

spacy-universal-sentence-encoder

(★ 182)

Google USE (Universal Sentence Encoder) for spaCy

파일 탐색기

최종 버전 다운로드 (.zip)

- - run-tests.sh
- - tests.yml
- settings.json
- Dockerfile
- README.md
- - en_use_cmlm_lg.json
  - en_use_cmlm_md.json
  - en_use_lg.json
  - en_use_md.json
  - xx_use_cmlm.json
  - xx_use_cmlm_br.json
  - xx_use_lg.json
  - xx_use_md.json
- __init__.py
- language.py
- util.py
- __init__.py
- test_config.py
- test_load_nlp.py
- test_similarity.py
- utils.py
.gitignore
build_models.sh
create.py
LICENSE
MANIFEST.in
README.md
requirements.txt
setup.cfg
setup.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/MartinoMensio/spacy-universal-sentence-encoder

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd spacy-universal-sentence-encoder

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Docker

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Docker Desktop 컨테이너를 빌드하고 실행하려면 필요합니다. 설치 후 실행해서 백그라운드에 켜두세요.

docker build -f docker/Dockerfile -t spacy-universal-sentence-encoder .

Dockerfile을 기반으로 실행 가능한 이미지를 빌드합니다.

docker run -p 8080:80 spacy-universal-sentence-encoder

빌드된 이미지를 실제 컨테이너로 실행합니다.

✅ 터미널에 docker compose ps 를 입력해 컨테이너들이 Up 상태인지 확인하세요. README에 포트 번호가 적혀있다면 브라우저에서 http://localhost:포트번호 로 접속해보세요.

3. Python

쉬움

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![Tests](https://github.com/MartinoMensio/spacy-universal-sentence-encoder/actions/workflows/tests.yml/badge.svg)](https://github.com/MartinoMensio/spacy-universal-sentence-encoder/actions/workflows/tests.yml)
[![Downloads](https://static.pepy.tech/badge/spacy-universal-sentence-encoder)](https://pepy.tech/project/spacy-universal-sentence-encoder)
[![Current Release Version](https://img.shields.io/github/release/MartinoMensio/spacy-universal-sentence-encoder.svg?&logo=github)](https://github.com/MartinoMensio/spacy-universal-sentence-encoder/releases)
[![pypi Version](https://img.shields.io/pypi/v/spacy-universal-sentence-encoder.svg?&logo=pypi&logoColor=white)](https://pypi.org/project/spacy-universal-sentence-encoder/)
[![Coverage Status](https://coveralls.io/repos/github/MartinoMensio/spacy-universal-sentence-encoder/badge.svg?branch=master)](https://coveralls.io/github/MartinoMensio/spacy-universal-sentence-encoder?branch=master)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

# Spacy - Universal Sentence Encoder

Make use of Google's Universal Sentence Encoder directly within SpaCy.
This library lets you embed [Docs](https://spacy.io/api/doc), [Spans](https://spacy.io/api/span) and [Tokens](https://spacy.io/api/token) from the [Universal Sentence Encoder family available on TensorFlow Hub](https://tfhub.dev/google/collections/universal-sentence-encoder/1).

For using sentence-BERT in spaCy, see https://github.com/MartinoMensio/spacy-sentence-bert

## Motivation
There are many different reasons to not always use BERT. For example to have embeddings that are tuned specifically for another task (e.g. sentence similarity). See this very useful blog article:
https://blog.floydhub.com/when-the-best-nlp-model-is-not-the-best-choice/

The Universal Sentence Encoder is trained on different tasks which are more suited to identifying sentence similarity. [Google AI blog](https://ai.googleblog.com/2018/05/advances-in-semantic-textual-similarity.html) [paper](https://arxiv.org/abs/1803.11175)

This library uses the [`user_hooks` of spaCy](https://spacy.io/usage/processing-pipelines#custom-components-user-hooks) to use an external model for the vectors, in this case a simple wrapper to the models available on TensorFlow Hub.

## Install

You can install this library from:
- github: `pip install git+https://github.com/MartinoMensio/spacy-universal-sentence-encoder.git`
- pyPI: `pip install spacy-universal-sentence-encoder`

Compatibility:
- python:
  - 3.6: compatible but not actively tested
  - 3.7/3.8/3.9/3.10: compatible and actively tested
  - 3.11 compatible but relies on rc version of [tensorflow-text](https://pypi.org/project/tensorflow-text/) for multilingual models
- tensorflow>=2.4.0,<3.0.0
- spacy>=3.0.0,<4.0.0 (SpaCy v3 API changed a lot from v2)

To use the multilingual version of the models, you need to install the extra named `multi` with the command: `pip install spacy-universal-sentence-encoder[multi]`. This installs the dependency `tensorflow-text` that is required to run the multilingual models. Note that `tensorflow-text` is currently in RC version for python3.11.

In alternative, you can install the following standalone pre-packaged models with pip. Each model can be installed independently:

| model name | source | pip package |
|------------|--------|---|
| en_use_md  | https://tfhub.dev/google/universal-sentence-encoder | `pip install https://github.com/MartinoMensio/spacy-universal-sentence-encoder/releases/download/v0.4.6/en_use_md-0.4.6.tar.gz#en_use_md-0.4.6` |
| en_use_lg  | https://tfhub.dev/google/universal-sentence-encoder-large | `pip install https://github.com/MartinoMensio/spacy-universal-sentence-encoder/releases/download/v0.4.6/en_use_lg-0.4.6.tar.gz#en_use_lg-0.4.6` |
| xx_use_md  | https://tfhub.dev/google/universal-sentence-encoder-multilingual | `pip install https://github.com/MartinoMensio/spacy-universal-sentence-encoder/releases/download/v0.4.6/xx_use_md-0.4.6.tar.gz#xx_use_md-0.4.6` |
| xx_use_lg  | https://tfhub.dev/google/universal-sentence-encoder-multilingual-large | `pip install https://github.com/MartinoMensio/spacy-universal-sentence-encoder/releases/download/v0.4.6/xx_use_lg-0.4.6.tar.gz#xx_use_lg-0.4.6` |

In addition, also [CMLM models](https://openreview.net/pdf?id=WDVD4lUCTzU) are now available:

| model name | source | pip package |
|------------|--------|---|
| en_use_cmlm_md  | https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base | `pip install https://github.com/MartinoMensio/spacy-universal-sentence-encoder/releases/download/v0.4.6/en_use_cmlm_md-0.4.6.tar.gz#en_use_cmlm_md-0.4.6` |
| en_use_cmlm_lg  | https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-large | `pip install https://github.com/MartinoMensio/spacy-universal-sentence-encoder/releases/download/v0.4.6/en_use_cmlm_lg-0.4.6.tar.gz#en_use_cmlm_lg-0.4.6` |
| xx_use_cmlm  | https://tfhub.dev/google/universal-sentence-encoder-cmlm/multilingual-base | `pip install https://github.com/MartinoMensio/spacy-universal-sentence-encoder/releases/download/v0.4.6/xx_use_cmlm-0.4.6.tar.gz#xx_use_cmlm-0.4.6` |
| xx_use_cmlm_br  | https://tfhub.dev/google/universal-sentence-encoder-cmlm/multilingual-base-br | `pip install https://github.com/MartinoMensio/spacy-universal-sentence-encoder/releases/download/v0.4.6/xx_use_cmlm_br-0.4.6.tar.gz#xx_use_cmlm_br-0.4.6` |

## Usage

### Loading the model

If you installed the model standalone packages (see table above) you can use the usual spacy API to load this model:

```python
import spacy
nlp = spacy.load('en_use_md')
```

Otherwise you need to load the model in the following way:

```python
import spacy_universal_sentence_encoder
nlp = spacy_universal_sentence_encoder.load_model('xx_use_lg')
```

The third option is to load the model on your existing spaCy pipeline:

```python
import spacy
# this is your nlp object that can be any spaCy model
nlp = spacy.load('en_core_web_sm')

# add the pipeline stage (will be mapped to the most adequate model from the table above, en_use_md)
nlp.add_pipe('universal_sentence_encoder')
```

In all of the three options, the first time that you load a certain Universal Sentence Encoder model, it will be downloaded from TF Hub (see section below to use an already downloaded model, or to change the location of the model files).

The last option (using `nlp.add_pipe`) can be customised with the following configurations:

- `use_model_url`: allows to use a specific TFHub URL
- `preprocessor_url`: for TFHub models that need specific preprocessing with another TFHub model (e.g., see documentation of [CMLM models](https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base))
- `model_name`: to load a specific model instead of mapping the current (language, size) to one of the options in the table above
- `enable_cache`: default `True`, enables an internal cache to avoid embedding the same text (doc/span/token) twice. It makes the computation faster (when enough duplicates are embedded) but has a memory footprint because all the embeddings extracted are kept in the cache
- `debug`: default `False` shows debugging information.

To use the configurations, when adding the pipe stage pass a dict as additional argument, for example:

```python
nlp.add_pipe('universal_sentence_encoder', config={'enable_cache': False})
```

### Use the embeddings

After adding to the pipeline, you can use the embedding models by using the various properties and methods of Docs, Spans and Tokens:

```python
# load as before
import spacy
nlp = spacy.load('en_core_web_lg')
nlp.add_pipe('universal_sentence_encoder')

# get two documents
doc_1 = nlp('Hi there, how are you?')
doc_2 = nlp('Hello there, how are you doing today?')
# Inspect the shape of the Doc, Span and Token vectors
print(doc_1.vector.shape) # the full document representation
print(doc_1[3], doc_1[3].vector.shape) # the word "how"
print(doc_1[3:6], doc_1[3:6].vector.shape) # the span "how are you"

# or use the similarity method that is based on the vectors, on Doc, Span or Token
print(doc_1.similarity(doc_2[0:7]))
```

## Common issues

Here you can find the most common issues with possible solutions.

### Using a pre-downloaded model

If you want to use a model that you have already downloaded from TensorFlow Hub, belonging to the [Universal Sentence Encoder family](https://tfhub.dev/google/collections/universal-sentence-encoder/1), you can use it by doing the following:

- locate the full path of the folder where you have downloaded and extracted the model. Let's suppose the location is `$HOME/tfhub_models`
- rename the folder of the extracted model (the one directly containing the folders `variables` and the file `saved_model.pb`) to the sha1 hash of the TFHub model [source](https://medium.com/@xianbao.qian/how-to-run-tf-hub-locally-without-internet-connection-4506b850a915). The mapping URL / sha1 values is the following:
  - [`en_use_md`](https://tfhub.dev/google/universal-sentence-encoder/4): `063d866c06683311b44b4992fd46003be952409c`
  - [`en_use_lg`](https://tfhub.dev/google/universal-sentence-encoder-large/5): `c9fe785512ca4a1b179831acb18a0c6bfba603dd`
  - [`xx_use_md`](https://tfhub.dev/google/universal-sentence-encoder-multilingual/3): `26c892ffbc8d7b032f5a95f316e2841ed4f1608c`
  - [`xx_use_lg`](https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3): `97e68b633b7cf018904eb965602b92c9f3ad14c9`
- set the environment variable `TFHUB_CACHE_DIR` to the location containing the renamed folder, for example `$HOME/tfhub_models` (set it before trying to download the model: `export TFHUB_CACHE_DIR=$HOME/tfhub_models`)
- Now load your model and it should see that it was already downloaded

### Serialisation

To serialise and deserialise nlp objects, SpaCy does not restore `user_hooks` after deserialisation, so a call to `from_bytes` will result in not using the TensorFlow vectors, so the similarities won't be good. For this reason the suggested solution is:

- serialise with `bytes = doc.to_bytes()` normally
- deserialise with `spacy_universal_sentence_encoder.doc_from_bytes(nlp, bytes)` which will also restore the user hooks

### Multiprocessing

This library, relying on TensorFlow, is not fork-safe. This means that if you are using this library inside multiple processes (e.g. with a `multiprocessing.pool.Pool`), your processes will deadlock.
The solutions are:
- use a thread-based environment (e.g. `multiprocessing.pool.ThreadPool`)
- only use this library inside the created processes (first create the processes and then import and use the library)

### `Using `nlp.pipe` with multiple processes

Spacy does not restore user hooks (`UserWarning: [W109]`) therefore if you use `nlp.pipe` with multiple processes, you won't be able to use the `.vector` on `doc`, `span` and `token`. I am developing a workaround.

## Utils

To build and upload
```bash
# change version
VERSION=0.4.6
# change version references everywhere
# update locally installed package
pip install -r requirements.txt
# build the standalone models (8)
./build_models.sh
# build the archive at dist/spacy_universal_sentence_encoder-${VERSION}.tar.gz
python setup.py sdist
# upload to pypi
twine upload dist/spacy_universal_sentence_encoder-${VERSION}.tar.gz
# upload language packages to github
```