finnish-word-embeddings

KO | EN

gitlite — search

#javascript #python #ai #llm #golang #android #rust #deep-learning #nodejs #java #cli #vue #ios #openai

finnish-word-embeddings

(★ 8)

Collection of word embeddings for Finnish language

finnish-word-embeddings Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Finnish Word Embeddings

This repository contains links to [word embeddings](https://en.wikipedia.org/wiki/Word_embedding) for [Finnish language](https://en.wikipedia.org/wiki/Finnish_language), including code for training your own embeddings. Word embeddings represent words as low dimensional numerical vectors, which are helpful in various [NLP](https://en.wikipedia.org/wiki/Natural_language_processing) applications, such as building chatbots, calculating semantic similarities or detecting fake news.

![Finnish words](https://raw.githubusercontent.com/jmyrberg/finnish-word-embeddings/master/picture.JPG "Finnish words")

## List of available embeddings

## Example usage of word embeddings

```python
# Word embeddings in word2vec-format can easily be loaded and queried with gensim
# See https://radimrehurek.com/gensim/models/keyedvectors.html for reference
from gensim.models.keyedvectors import KeyedVectors

# Load vectors into memory (bin in filename means binary=True)
embeddings_path = './data/embeddings/fasttext.fi.all.1045M.100d.bin.gz'
kv = KeyedVectors.load_word2vec_format(embeddings_path, binary=True)

# Find most similar word to 'koira'
print(kv.most_similar('koira'))
```

## Training your own word embeddings

This repository also contains the code used for crawling data from popular Finnish web sites, extracting sentences from those, and training word embeddings. The spiders used for web scraping can be found from the [*crawling*-folder](crawling/), whereas preprocessing and training of embeddings can be found from the [*embeddings*-folder](embeddings/).

Three steps are required:

1. Clone this repository using ```git clone https://github.com/jmyrberg/finnish-word-embeddings``` and install required packages with ```pip install -r requirements.txt```.

2. Crawl data by starting a spider by running [*run_spider.bat*](scripts/run_spider.bat) and typing in the name of the spider, such as *iltalehti*. All available spider names can be found from the spider class definitions in [*all_spiders.py*](crawling/spiders/all_spiders.py). See [Scrapy](https://scrapy.org/) for more information on how to create your own spiders. Optionally, you may also use your own source documents for training.

3. Preprocess crawled material and train word embeddings by running [*update.py*](embeddings/update.py). Or optionally, prepare your own documents into sentence lines and train them by running [*train.py*](embeddings/train.py).

If you follow the steps above without modifying any code, you should be able to reproduce the custom word embeddings provided in this repository. The provided code should also automatically create the folder structure under *./data/\** as follows:

* *crawl*: State of the spider to avoid duplicate scrapes
* *feed*: Crawled material with JSON line files named like *\<spiderName\>.jl*
* *processed*: Preprocessed crawled material in sentence line files like *all.sl*
* *embeddings*: Trained word embeddings named like *\<modelName\>.fi.\<sentenceLineFilename\>.\<numberOfTokensTrainedOn\>.\<embeddingsDimension\>.\<format\>.gz*

## Contributing

If you want to add, modify or remove something in the list of word embeddings or code, please feel free to make a pull request, file an issue, or contact me.

---
Jesse Myrberg (jesse.myrberg@gmail.com)

finnish-word-embeddings

finnish-word-embeddings Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

finnish-word-embeddings

finnish-word-embeddings Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README