KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #cli #claude-code #codex #developer-tools #ai #dsh-plugin #ai-agent #mcp #deepseek-harness #windows

MATRN

★ 74 Open GitHub ↗

Official PyTorch implementation for Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Features (MATRN) in ECCV 2022.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Features (ECCV 2022)

| [paper](https://arxiv.org/pdf/2111.15263) | [slide](https://www.dropbox.com/s/idd21hlc2v7e72h/MATRN_PPT_v2.pdf?dl=0) | [poster](https://www.dropbox.com/s/8iug3wnon1aarwk/6583-poster.pdf?dl=0) |

Official PyTorch implementation for Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Features (MATRN) in [ECCV 2022](https://eccv2022.ecva.net/).

**[Byeonghu Na](https://github.com/byeonghu-na), [Yoonsik Kim](https://github.com/terryoo), and [Sungrae Park](https://github.com/sungraepark)**

This paper introduces a novel method, called Multi-modAl Text Recognition Network (MATRN), that enables interactions between visual and semantic features for better recognition performances.

## Datasets

We use lmdb dataset for training and evaluation dataset.
The datasets can be downloaded in [clova (for validation and evaluation)](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) and [ABINet (for training and evaluation)](https://github.com/FangShancheng/ABINet#datasets).

* Training datasets
 * [MJSynth (MJ)](https://www.robots.ox.ac.uk/~vgg/data/text/)
 * [SynthText (ST)](https://www.robots.ox.ac.uk/~vgg/data/scenetext/)
 * [WikiText](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip)
* Validation datasets
 * The union of the training set of [ICDAR2013](https://rrc.cvc.uab.es/?ch=2), [ICDAR2015](https://rrc.cvc.uab.es/?ch=4), [IIIT5K](http://cvit.iiit.ac.in/projects/SceneTextUnderstanding/IIIT5K.html), and [Street View Text](http://www.iapr-tc11.org/mediawiki/index.php/The_Street_View_Text_Dataset)
* Evaluation datasets
 * Regular datasets 
 * [IIIT5K](http://cvit.iiit.ac.in/projects/SceneTextUnderstanding/IIIT5K.html) (IIIT)
 * [Street View Text](http://www.iapr-tc11.org/mediawiki/index.php/The_Street_View_Text_Dataset) (SVT)
 * [ICDAR2013](https://rrc.cvc.uab.es/?ch=2): IC13S with 857 images, IC13L with 1015 images
 * Irregular dataset
 * [ICDAR2015](https://rrc.cvc.uab.es/?ch=4): IC15S with 1811 images, IC15L with 2077 images
 * [Street View Text Perspective](https://openaccess.thecvf.com/content_iccv_2013/papers/Phan_Recognizing_Text_with_2013_ICCV_paper.pdf) (SVTP)
 * [CUTE80](http://cs-chan.com/downloads_CUTE80_dataset.html) (CUTE)
* Tree structure of `data` directory
 ```
 data
 ├── charset_36.txt
 ├── evaluation
 │   ├── CUTE80
 │   ├── IC13_857
 │   ├── IC13_1015
 │   ├── IC15_1811
 │   ├── IC15_2077
 │   ├── IIIT5k_3000
 │   ├── SVT
 │   └── SVTP
 ├── training
 │   ├── MJ
 │   │   ├── MJ_test
 │   │   ├── MJ_train
 │   │   └── MJ_valid
 │   └── ST
 ├── validation
 ├── WikiText-103.csv
 └── WikiText-103_eval_d1.csv
 ```

## Requirements

```
pip install torch==1.7.1 torchvision==0.8.2 fastai==1.0.60 lmdb pillow opencv-python tensorboardX editdistance
```

### Pretrained Models

* Download pretrained model of MATRN from this [link](https://www.dropbox.com/s/pjcarm73cqwbxh4/best-train-matrn.pth?dl=0). Performances of the pretrained model are:

|Model|IIIT|SVT|IC13S|IC13L|IC15S|IC15L|SVTP|CUTE|
|-|-|-|-|-|-|-|-|-|
|MATRN|96.7|94.9|97.9|95.8|86.6|82.9|90.5|94.1|

* If you want to train with pretrained visioan and language model, download pretrained model of vision and language model from [ABINet](https://github.com/FangShancheng/ABINet#pretrained-models).

## Training and Evaluation

* Training
```
python main.py --config=configs/train_matrn.yaml
```

* Evaluation
```
python main.py --config=configs/train_matrn.yaml --phase test --image_only
```
Additional flags:
- `--checkpoint /path/to/checkpoint` set the path of evaluation model 
- `--test_root /path/to/dataset` set the path of evaluation dataset
- `--model_eval [alignment|vision|language]` which sub-model to evaluate
- `--image_only` disable dumping visualization of attention masks

## Acknowledgements

This implementation has been based on [ABINet](https://github.com/FangShancheng/ABINet).

## Citation
Please cite this work in your publications if it helps your research.
```bash 
@inproceedings{na2022multi,
 title={Multi-modal text recognition networks: Interactive enhancements between visual and semantic features},
 author={Na, Byeonghu and Kim, Yoonsik and Park, Sungrae},
 booktitle={European Conference on Computer Vision},
 pages={446--463},
 year={2022},
 organization={Springer}
}
 ```

MATRN

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README