UniTouch

KO | EN

gitlite — search

#typescript #ai-agents #deepseek-harness #dsh-plugin #open-source #ai #claude-code #cli #windows #codex #developer-tools #dsh

★ 95 Open GitHub ↗

[CVPR 2024] Binding Touch to Everything: Learning Unified Multimodal Tactile Representations

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (CVPR 2024)
[![arXiv](https://img.shields.io/badge/arXiv-2401.18084-df2a2a.svg)](https://arxiv.org/abs/2401.18084)
[![HF Models](https://img.shields.io/badge/%F0%9F%A4%97-Models-yellow)](https://huggingface.co/chfeng/Touch-LLM)
[![Static Badge](https://img.shields.io/badge/Project-Page-a)](https://cfeng16.github.io/UniTouch/)

This repo contains code of **Touch-LLM** for **UniTouch**.
Our code is built on top of the [ImageBind](https://github.com/facebookresearch/ImageBind) and [LLaMA-Adapter](https://github.com/OpenGVLab/LLaMA-Adapter) codebases.

![UniTouch model](https://raw.githubusercontent.com/cfeng16/UniTouch/main/images/unitouch-method.png)

# Inference with Pretrained Models

## Option 1: Touch Encoder Only (Lightweight)

If you only need the touch encoder to extract tactile embeddings (without the language model):

1. Download the pretrained touch encoder (`last_new.ckpt`) from the [HuggingFace model hub](https://huggingface.co/chfeng/Touch-LLM) and put it in the `./UniTouch` folder, same level as `touch_qa.py`.

2. Run the standalone touch encoder:

```bash
python load_touch_encoder.py
```

This will load only the touch encoder and extract 1024-dimensional embeddings from touch images. You can use these embeddings for:
- Touch-based similarity search
- Touch classification
- Touch feature extraction
- Cross-modal retrieval (touch-to-vision, touch-to-text, etc.)

Example usage:
```python
import torch
import ImageBind.data as data
from ImageBind.models.x2touch_model_part import x2touch, ModalityType

# Load touch encoder
model = x2touch(pretrained=True)
model.eval()

# Extract embeddings from touch images
touch_images = data.load_and_transform_vision_data(["path/to/touch.jpg"], device="cuda")
with torch.no_grad():
    embeddings = model({ModalityType.TOUCH: touch_images})[ModalityType.TOUCH]  # Shape: [1, 1024]
```

## Option 2: Full Touch-LLM (with Language Model)

If you need the full Touch-LLM for tactile question answering:

1. Download the pretrained touch encoder (`last_new.ckpt`) from the [HuggingFace model hub](https://huggingface.co/chfeng/Touch-LLM) and put it in the `./UniTouch` folder, same level as `touch_qa.py`.
2. Download the folder `ckpts` from the [HuggingFace model hub](https://huggingface.co/chfeng/Touch-LLM) and put it in the `./UniTouch` folder, same level as `touch_qa.py`.
3. Download the folder `llama_ori` from the [HuggingFace model hub](https://huggingface.co/chfeng/Touch-LLM) and put it in the `./UniTouch` folder, same level as `touch_qa.py`.

4. Run Touch-LLM for tactile question answering:

```bash
CUDA_VISIBLE_DEVICES=0 python touch_qa.py
```

## Citation

```bibtex
@inproceedings{yang2024binding,
  title={Binding touch to everything: Learning unified multimodal tactile representations},
  author={Yang, Fengyu and Feng, Chao and Chen, Ziyang and Park, Hyoungseob and Wang, Daniel and Dou, Yiming and Zeng, Ziyao and Chen, Xien and Gangopadhyay, Rit and Owens, Andrew and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={26340--26353},
  year={2024}
}
```

UniTouch

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README