how-to-understand-bert

KO | EN

gitlite — search

#javascript #python #hacktoberfest #ai #typescript #go #android #rust #deep-learning #docker #awesome #vue #openai #open-source

how-to-understand-bert

(★ 9)

In this repository, I have collected different sources, visualizations, and code examples of BERT

how-to-understand-bert Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# how-to-understand-bert 🤗
In this repository, I have collected different sources, visualizations and code examples of BERT. I had been doing this during learning material of BERT. Perhaps, I should call this repository like "how-I-understand-bert" 🤔

****

## Theory 🙌
So, I have started my long way in BERT with [blog posts](https://mccormickml.com/tutorials/) and [videos](https://www.youtube.com/watch?v=FKlPCK1uFrc&list=PLam9sigHPGwOBuH4_4fr-XvDbe5uneaf6&index=1) produced by Chris McCormick. After watching the first video, I clearly understood that it is necessary to read more about Attention and Transformer.

1. Attention
   + [en] [Visualizing A Neural Machine Translation Model](https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/) (Mechanics of Seq2seq Models With Attention) ❤️

+ [en] [Attention is all you need](https://arxiv.org/abs/1706.03762) - original paper

+ [en] [Attention is all you need](https://www.youtube.com/watch?v=iDulhoQ2pro) - short video about this paper

+ [en] [Attention is all you need](https://www.youtube.com/watch?v=nPuWGx_wF3I&list=WL&index=74) - short video about this paper with excellent explanation ❤️

+ [ru] [Attention, attention!](https://www.youtube.com/watch?v=q9svwVYduSo&list=WL&index=98&t=4s) - seminar organized by JetBrains Research

2. Transformer
   + [en] [Awesome illustrations](http://jalammar.github.io/illustrated-transformer/) was created by Jay Alammar ❤️

+ [en] [The Narrated Transformer Language Model](https://www.youtube.com/watch?v=-QH8fRhqFHM) - explanation by Jay Alammar ❤️

+ [en] [Transformer (Attention is all you need)](https://www.youtube.com/watch?v=z1xs9jdZnuY) - short video with great visualizations

+ [ru] [Self-Attention. Transformer overview](https://www.youtube.com/❤️watch?v=UETKUIlYE6g) ❤️

+ [en] Just a short recap [video](https://www.youtube.com/watch?v=S27pHKBEp30&list=WL&index=95&t=41s)

+ also, in **_transformer_** folder you can find different implementations of this architecture ❤️‍🔥

****

Next, let's talk about tokenization in context of BERT.
   + [en] [WordPiece Embeddings](https://www.youtube.com/watch?v=zJW57aCBCTk&list=PLam9sigHPGwOBuH4_4fr-XvDbe5uneaf6&index=2) - II part of BERT Research by Chris McCormick ❤️

+ [en] Tokenizers: [How machines read](https://blog.floydhub.com/tokenization-nlp/) ❤️

+ also, in __*vocabulary*__ folder you can find different examples ❤️‍🔥

****

And finally, let's read more about BERT.
   + [en] The Illustrated BERT [blog post](http://jalammar.github.io/illustrated-bert/) ❤️

+ [en] [A Visual Guide to Using BERT for the First Time](http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/)

+ [en] Google AI Blog [post about open sourcing BERT](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html)

+ [en] One more [blog post explaining BERT](https://yashuseth.blog/2019/06/12/bert-explained-faqs-understand-bert-working/)

+ [ru] [Context based models. BERT overview](https://www.youtube.com/watch?v=1DygevyV2eA) ❤️

****

## Examples 💪
   + [en] [BERT Research - Fine Tuning](https://www.youtube.com/watch?v=x66kkDnbzi4&list=PLam9sigHPGwOBuH4_4fr-XvDbe5uneaf6&index=3) - Chris McCormick tutorial ❤️

+ [en] [BERT classifier fine-tuning with PyTorch, HuggingFace, and Catalyst](https://github.com/Yorko/bert-finetuning-catalyst) - Yury Kashnitsky tutorial ❤️

+ in __*sequence-classification*__ folder you can find different examples ❤️‍🔥

+ in __*document-classification*__ folder you can find different examples ❤️‍🔥

+ in __*bert-finetuning-catalyst*__ folder you can find great example based on Yury Kashnitsky tutorial ❤️‍🔥

****

## Additional sources 📚
   + [en] Check [this repository](https://github.com/tomohideshibata/BERT-related-papers) if you wanna more papers

+ [en] [Transformer encoder - visualized](https://github.com/mertensu/transformer-tutorial)

+ [en] [Firing a cannon at sparrows: BERT vs. logreg](https://www.youtube.com/watch?v=JIU6WZuWl6k&list=WL&index=50)

+ [en] DistillBERT overview (distillation will be covered later in our course) [blog post](https://medium.com/huggingface/distilbert-8cf3380435b5)

+ [en] OpenAI blog post [Better Language Models and Their Implications (GPT-2)](https://openai.com/blog/better-language-models/)

+ [en] Great PyTorch library: [pytorch-transformers](https://github.com/huggingface/transformers)

+ [en] [Post about GPT-2 in OpenAI blog (by 04.10.2019)](https://openai.com/blog/fine-tuning-gpt-2/)

+ [en] OpenAI API [request](https://openai.com/blog/openai-api/)

how-to-understand-bert

how-to-understand-bert Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

how-to-understand-bert

how-to-understand-bert Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README