londogard-nlp-toolkit

KO | EN

gitlite — search

#hacktoberfest #javascript #ios #php #python #macos #swift #css #golang #android #ansible #sublime-text

londogard-nlp-toolkit

★ 74 Open GitHub ↗

Londogard Natural Language Processing Toolkit written in Kotlin

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![Maven Central](https://img.shields.io/maven-central/v/com.londogard/nlp.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:%22com.londogard%22%20AND%20a:%22nlp%22)
<a href="https://www.buymeacoffee.com/hlondogard" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-green.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>

# londogard-nlp-toolkit
Londogard Natural Language Processing Toolkit written in Kotlin for the JVM.  
This toolkit will be used throughout Londogard libraries/products such as our Summarizer, Text-Generation & more.

The `LanguageSupport` enum is used to determine what support different tools like Embeddings or Stopwords have out-of-the-box.

| Tool | Info | Docs | Samples (Kotlin Notebook) |
|--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
| Word Embeddings | Word & Subword Embeddings available in 157 ([fastText.cc](https://fasttext.cc)) & 275 languages ([bpemb](https://bpemb.h-its.org/)) out-of-the-box. | [embeddings](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.embeddings/index.html) | [wordembeddings.ipynb](https://github.com/londogard/londogard-nlp-toolkit/blob/main/docs/samples/components/wordembeddings.ipynb) |
| Sentence Embeddings | Average & [Unsupervised Random Walk Sentence Embeddings](https://aclanthology.org/W18-3012/) | [sentence-embeddings](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.embeddings.sentence/index.html) | [sentence-embeddings.ipynb](https://github.com/londogard/londogard-nlp-toolkit/blob/main/docs/samples/components/sentence-embeddings.ipynb) |
| Stopwords | Supports 23 languages out-of-the-box through NLTK's list of stopword | [stopwords](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.stopwords/index.html) | [stopwords.ipynb](https://github.com/londogard/londogard-nlp-toolkit/blob/main/docs/samples/components/stopwords.ipynb) |
| Word Frequencies | Supports 34 languages out-of-the-box through [LuminosoInsight](https://github.com/LuminosoInsight/wordfreq/) word frequency tables | [wordfrequency](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.wordfreq/index.html) | [wordfreq.ipynb](https://github.com/londogard/londogard-nlp-toolkit/blob/main/docs/samples/components/wordfreq.ipynb) | 
| Stemming | Supports 14 languages out-of-the-box using [Snowball Stemmer](https://snowballstem.org/) under the hood | [stemming](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.stemmer/index.html) | [stemmer.ipynb](https://github.com/londogard/londogard-nlp-toolkit/blob/main/docs/samples/components/stemmer.ipynb) | 
| Tokenizers | Char, Word, Subword & Sentence Tokenizer support! SentencePiece? HuggingFace? It's there! | - [tokenizers](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.tokenizer/index.html) - [sentence-tokenizers](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.tokenizer.sentence/index.html) | [tokenizer.ipynb](https://github.com/londogard/londogard-nlp-toolkit/blob/main/docs/samples/components/tokenizers.ipynb) |
| Vectorizers & Encoders | BagOfWords, TF-IDF, BM25 & OneHot | - [vectorizers (TF-IDF, BM-25,..)](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.meachinelearning.vectorizer/index.html) - [count-vectorizers (Count, Hash, ..)](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.meachinelearning.vectorizer.count/index.html) [encoders (OneHot)](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.meachinelearning.encoders/index.html) - [transforms (TF-IDF, BM-25,..)](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.meachinelearning.transformers/index.html) | TODO | 
| Keyword Extractions | CooccurenceKeywords based on algorithm proposed in [DOI:10.1142/S0218213004001466](https://www.researchgate.net/publication/2572200_Keyword_Extraction_from_a_Single_Document_using_Word_Co-occurrence_Statistical_Information) || [keywords.ipynb](https://github.com/londogard/londogard-nlp-toolkit/blob/main/docs/samples/components/keyword_extraction.ipynb) |
| Machine Learning | LogisticRegression Classifier (using Gradient Descent), NaïveBayes (binary) & Hidden Markov Model (HMM) as Sequence Classifier | - [classifiers (LogisticRegression, NaïveBayes)](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.meachinelearning.predictors.classifiers/index.html) [regression (LinearRegression)](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.meachinelearning.predictors.regression/index.html) - [sequence classifier (HiddenMarkovModel)](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.meachinelearning.predictors.sequence/index.html) | See [e2e-examples](https://github.com/londogard/londogard-nlp-toolkit/tree/main/docs/samples/e2e) |
| Deep Learning (Transformers / HuggingFace) | `ClassifierPipeline` and `TokenClassifierPipeline` which supports HuggingFace ONNX model-names & PyTorch from local files | [transformers](https://londogard.github.io/londogard-nlp-toolkit/nlp/com.londogard.nlp.meachinelearning.predictors.transformers/index.html) | See [e2e-examples](https://github.com/londogard/londogard-nlp-toolkit/tree/main/docs/samples/e2e) |
| spaCy-like API | 🚧WIP🚧 |||

## Installation

**MavenCentral**  
```kotlin
implementation("com.londogard:nlp:$version")
```

## Guides

Simple end-2-end guides available as notebooks via [docs/samples](https://github.com/londogard/londogard-nlp-toolkit/tree/main/docs/samples/e2e).

This includes:
1. IMDB Sentiment Analysis using Logistic Regression or Naïve Bayes
2. IMDB Sentiment Analysis using HuggingFace Transformers, using `ClassifierPipeline.create(<model-name>)`
3. POS-Tagging using Hidden Markov Model
4. POS-Tagging using HuggingFace Transformers, using `TokenClassifierPipeline.create(<model-name>)`

& potentially more.

londogard-nlp-toolkit

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README