sato - GitLite

KO | EN

gitlite — search

#python #ai #typescript #llm #golang #android #rust #nodejs #macos #awesome #awesome-list #pytorch #ios #openai

sato

(★ 117)

Code and data for Sato https://arxiv.org/abs/1911.06311.

File Explorer

Download Latest Version (.zip)

- - char_col.tsv
  - par_col.tsv
  - rest_col.tsv
  - topic_col.tsv
  - word_col.tsv
- data_name_mapping.json
- types.json
- - model.pt
  - sherlock_None.pt
- - 1.png
  - 2.png
- - error.html
  - table.html
  - test.html
  - upload.html
- - files.csv
  - theaters.csv
  - University_employees.csv
- predict.py
- server.py
- - __init__.py
  - sherlock_features.py
  - topic_features_LDA.py
- - dateparser.py
  - general_helpers.py
  - read_raw_data.py
  - type_detection.py
  - utils.py
- extract_features.py
- extract_header.py
- extract_matrix.py
- extract_table.py
- raw_col_label_pairs.py
- split_train_test.py
- - matrix_co_W0_type78.npy
- - sherlock+LDA.txt
  - sherlock.txt
- - __init__.py
- datasets.py
- factor.py
- models_sherlock.py
- train_CRF_LC.py
- train_sherlock.py
- - feature_importance_CRF_None.csv
  - feature_importance_CRF_num-directstr_thr-0_tn-400.csv
  - feature_importance_single_None.csv
  - feature_importance_single_num-directstr_thr-0_tn-400.csv
  - log_co-occur.csv
  - result_CRF_LDA_multi-col.csv
  - result_CRF_multi-col.csv
  - result_LDA_multi-col.csv
  - result_sherlock_multi-col.csv
  - type-dist.csv
- FinalPlots-paper.ipynb
- analysis_functions.py
- exp.sh
- feature_importance.py
- per_type.py
- - bag_of_characters.py
  - bag_of_words.py
  - paragraph_vectors.py
  - word_embeddings.py
- download.sh
- README.md
- LDA_cluster.py
- train_LDA.py
.gitignore
diagram-overview.svg
download_data.sh
LICENSE
README.md
requirements.txt
utils.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Sato: Contextual Semantic Type Detection in Tables

This repository includes source code, scripts, and data for training the **Sato** model.  The repo also includes a pretrained  model to help replicate the results in [our VLDB 2020 paper](http://www.vldb.org/pvldb/vol13/p1835-zhang.pdf).  Sato is a hybrid machine learning model to automatically detect the semantic types of columns in tables, exploiting the signals from the context as well as the column values. Sato combines a deep learning model trained on a large-scale table corpus with topic modeling and structured prediction.

<img src="diagram-overview.svg" width="800"/>
Above: Sato architecture. Sato's hyrid architecture consists of two basic modules; a topic-aware single-column prediction module and a structured output prediction module. The topic-aware module extends 
<a href=https://arxiv.org/pdf/1905.10688.pdf>Sherlock</a>'s single-column prediction model (a deep neural network) with additional topic subnetworks, incorporating <em>table intent</em> into the model. The structured output prediction module then combines the topic-aware results for all m columns, providing the final semantic type prediction for the columns in the table.

## What is Sato useful for?
Myriad data preparation and information retrieval tasks, including data cleaning, integration, discovery and search, rely
on the ability to accurately detect data column types. Schema matching for data integration leverages data types to find correspondences between data columns across tables. Similarly, data discovery benefits from detecting types of data
columns in order to return semantically relevant results for user queries. Recognizing the semantics of table values helps aggregate information from multiple tabular data sources. Search engines also rely on the detection of semantically relevant column names to extend support to tables. Natural language based query interfaces for structured data can also benefit from semantic type detection.

### Demo
We set up a simple online [demo](http://18.191.96.23:5000/) where you can upload small tables and get semantic predictions for column types.

![screenshot1](https://raw.githubusercontent.com/megagonlabs/sato/master/demo/screenshots/1.png)
![screenshot2](https://raw.githubusercontent.com/megagonlabs/sato/master/demo/screenshots/2.png)

## 
## Environment setup
We recommend using a python virtual environment:
```
mkdir virtualenvs
virtualenv --python=python3 virtualenvs/col2type
```
Fill in and set paths:
```
export BASEPATH=[path to the repo]
# RAW_DIR can be empty if using extracted feature files.
export RAW_DIR=[path to the raw data]
export SHERLOCKPATH=$BASEPATH/sherlock
export EXTRACTPATH=$BASEPATH/extract
export PYTHONPATH=$PYTHONPATH:$SHERLOCKPATH
export PYTHONPATH=$PYTHONPATH:$BASEPATH
export TYPENAME='type78'

source ~/virtualenvs/col2type/bin/activate
```
Install required packages
```
cd $BASEPATH
pip install -r requirements.txt
```
To specify GPUID, use `CUDA_VISIBLE_DEVICES`. `CUDA_VISIBLE_DEVICES=""` to use CPU.

## Replicating results
Results in the paper can be replicated with and pre-trained models features we extracted.

1. Download data.
`./download_data.sh`
2. Run experiments
`cd $BASEPATH/scripts; ./exp.sh`
3. Generate plots from notebooks/FinalPlotsPaper

##  Additional 
This repo also allows training new Sato models with other hyper-parameters or extract features from additional data.

Download the [VIZNET]([https://github.com/mitmedialab/viznet](https://github.com/mitmedialab/viznet)) data and set RAW_DIR path to location of VIZNET raw data.

### Column feature extraction
```
cd $BASEPATH/extract
python extract_features.py [corpus_chunk] --f sherlock --num_processes [N]
```
corpus_chunk： corpus with potential partition post-fix, e.g. webtables0-p1, plotly-p1
N: number of processes used to extract features

### Table topic feature extraction
Download nltk data
```
import nltk
nltk.download('stopwords')
nltk.download('punkt')
```
[Optional] To train a new LDA model
```
cd topic_model
python train_LDA.py 
```
Extract topic features
```
cd $BASEPATH/extract
python extract_features.py [corpus_chunk] --f topic --LDA [LDA_name] --num_processes [N]
```
corpus_chunk： corpus with potential partition post-fix, e.g. webtables0-p1, plotly-p1
LDA_name: name of LDA model to extract topic features. Models are located in `topic_model/LDA_cache`
N: number of processes used to extract features

The extracted feature files go to `extract/out/features/[TYPENAME]` .

### Split train/test sets

Split the dataset into training and testing (8/2).

```
cd $BASEPATH/extract
python split_train_test.py --multi_col_only [m_col] --corpus_list [c_list]
```
m_col:`--multi_col_only` is set, filter the result and remove tables with only one column
c_list: corpus list

Output is a dictionary with entries ['train','test'].  Dictionary values are lists of `table_id`.

### Train Sato
```
cd $BASEPATH/model
python train_CRF_LC.py -c [config_file]
```
Check out `train_CRF_LC.py` for supported configurations.

## Original tables

Please see [table_data](table_data) for the original tables used for the experiments.

## Citing Sato

Please cite [our VLDB 2020 paper](http://www.vldb.org/pvldb/vol13/p1835-zhang.pdf)

```
@article{zhang2020sato,
    title={Sato: Contextual Semantic Type Detection in Tables},
    author={Dan Zhang and 
            Yoshihiko Suhara and 
            Jinfeng Li and 
            Madelon Hulsebos and 
            {\c{C}}a{\u{g}}atay Demiralp and 
            Wang-Chiew Tan},
    year = {2020},
    volume = {13},
    number = {12},
    journal = {Proc. VLDB Endow.},
    pages = {1835–1848},
    numpages = {14},
    url = {https://doi.org/10.14778/3407790.3407793}
}
```

## Contact

To get help with problems using Sato or replicating our results, please submit a GitHub issue.