KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

IAG

★ 33 Open GitHub ↗

[🏆CVPR'26] Official Repo for IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# [🏆CVPR'26] IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding  
[<strong>Junxian Li*</strong>](https://lijunxian111.github.io), Beining Xu*, [Simin Chen](https://siminchen.site), [Jiatong Li](https://phenixace.github.io), [Jingdi Lei](https://kyrielei.github.io), [Haodong Zhao^](https://dongdongzhaoup.github.io), [Di Zhang^](https://github.com/trotsky1997)

</div>

#### 🔥🔥🔥 News

- **2026-03-26:** We add the model, train and evaluation codes. You can merge them into the official repository of VLMs and use IAG!  
- **2026-03-02:** This repo is released.  
- **2026-02-21:** IAG is accepted by CVPR 2026.

---

> **Abstract:** Recent advances in vision-language models (VLMs) have significantly enhanced the visual grounding task, which involves locating objects in an image based on natural language queries. Despite these advancements, the security of VLM-based grounding systems has not been thoroughly investigated. This paper reveals a novel and realistic vulnerability: the first multi-target backdoor attack on VLM-based visual grounding.  
> Unlike prior attacks that rely on static triggers or fixed targets, we propose IAG, a method that dynamically generates input-aware, text-guided triggers conditioned on any specified target object description to execute the attack. This is achieved through a text-conditioned UNet that embeds imperceptible target semantic cues into visual inputs while preserving normal grounding performance on benign samples. We further develop a joint training objective that balances language capability with perceptual reconstruction to ensure imperceptibility, effectiveness, and stealth. Extensive experiments on multiple VLMs (e.g., LLaVA, InternVL, Ferret) and benchmarks (RefCOCO, RefCOCO+, RefCOCOg, Flickr30k Entities, and ShowUI) demonstrate that IAG achieves the **best** ASRs compared with other baselines on almost all settings without compromising clean accuracy, maintaining robustness against existing defenses, and exhibiting transferability across datasets and models. These findings underscore critical security risks in grounding-capable VLMs and highlight the need for further research on trustworthy multimodal understanding.

![](https://raw.githubusercontent.com/lijunxian111/IAG/main/figs/examples.png)

---

### Pipeline

![](https://raw.githubusercontent.com/lijunxian111/IAG/main/figs/IAG_framework.png)

---

## <a name="results"></a>🔎 Results

We present the performance of IAG, compared with baselines.

<details open>
<summary>Quantitative Results (click to expand)</summary>

- Results in Tab. 1 of the main paper

</details>

<details open>
<summary>Qualitative Results (click to expand)</summary>

- Results in Fig. 3 of the main paper

---

### Acknowledgements

Our method is built based on several MLLMs, like [LlaVA](https://github.com/haotian-liu/LLaVA), [InternVL](https://github.com/OpenGVLab/InternVL) and so on. Thanks to them!

---

### Citation

```ruby
@inproceedings{li2026iag,
  title={Iag: Input-aware backdoor attack on vlm-based visual grounding},
  author={Li, Junxian and Xu, Beining and Chen, Simin and Li, Jiatong and Lei, Jingdi and Zhao, Haodong and Zhang, Di},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={27872--27883},
  year={2026}
}
```

IAG

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README