AutoGUI

KO | EN

gitlite — search

#javascript #ai #typescript #llm #go #machine-learning #rust #linux #macos #java #awesome #kubernetes #vue #mcp

(★ 17)

The official implementation of AutoGUI.

AutoGUI 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# AutoGUI: Scaling GUI Grounding with Autonomous Functionality Annotations from LLMs

![Multi-Modal](https://img.shields.io/badge/Task-Multi--Modal-red)
<a href='https://huggingface.co/HongxinLi/AutoGUI-Qwen-v0.1'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue'></a>
<a href='https://huggingface.co/datasets/HongxinLi/AutoGUI-v1'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Data-green'></a>
[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://GitHub.com/Naereen/StrapDown.js/graphs/commit-activity) 
[![Awesome](https://awesome.re/badge.svg)]()

This repo opensources the training and evaluation code for AutoGUI, an automatic and scalable GUI annotation pipeline

TODO: 
1. AutoGUI-v2 data collected on thousands of mobile apps is coming soon.
2. Functionality-grounding-assisted GUI agents are coming soon.

***

## AutoGUI pipeline - Revolutionizing Large-Scale GUI Data Annotation
Existing UI annotation methods typically collect data from static UIs, focusing on describing either the visual appearance (e.g., a button beside
30 the navigation bar), element categories (e.g., “menu button”), or brief functions weakly related to the UI context (e.g., “show more information”).

Here, we are thrilled to unveil **AutoGUI**, a groundbreaking and scalable UI annotation pipeline. AutoGUI can autonomously annotate the contextual functionalities of diverse UI elements at scale, entirely eliminating the need for human experts. This innovation not only accelerates the data collection process but also enhances the depth and accuracy of UI functionality descriptions, opening a new path in the field of UI annotation.

**AutoGUI** initiates by collecting interaction trajectories on Common Crawl websites. Each trajectory step captures all interactable elements and the accessibility tree (AXTree) that briefly outlines the UI structure. The content changes in the AXTrees before and after interaction will be used by an open-source LLM (e.g., Llama-3-70B) to predict functionality annotations of the interacted elements.

This annotation process provides rich functional semantics in the generated annotations, thereby allowing for curating a GUI dataset that can potentially enhance the GUI understanding capabilities of GUI agents.

## Installation
You can install the AutoGUI package by cloning the repository and running the following command:

```
git clone https://github.com/BraveGroup/AutoGUI
cd AutoGUI
pip install -e .
```

### Optional Packages
Please also follow the installation instructions of [LLaVA](https://github.com/haotian-liu/LLaVA), [vLLM==0.7.3](https://vllm.readthedocs.io) and [SGLang](https://github.com/sgl-project/sglang) to install them for evaluation of LLaVA.

Note that Qwen2-VL needs `transformers >= 4.47.1` to avoid the MROPE bugs while `Qwen2.5-VL` needs `transformers >= 4.49.0`.

## AutoGUI Dataset

### Training Set
We provide 702k functionality grounding/captioning tasks that are generated by populating task templates with the collected element-functionality pairs. To mitigate the gap between various device types, the screenshots are rendered at various resolutions to mimic web browsers and mobile devices.

Please view the training data [here]([https://huggingface.co/datasets/WebAgent/AutoGUI-v1](https://huggingface.co/datasets/AutoGUI/AutoGUI-v1-702k)).

**A functionality grounding example:**

*User*: In this web page image, please locate the element as I describe it (with point). This element triggers a user registration process, allowing new users to create a PayPal account and gain access to the platform's services.

*Assistant*: (91,6)

**A functionality captioning example:**

*User*: What happens when you tap position (61,73) on the screen?

*Assistant*: This element serves as an input field for users to provide their birth date, contributing to the registration process by ensuring that users meet the age requirements for creating a Yahoo account.

### **Funcpred** - Functionality Grounding Benchmark
We also curate a 2k split used for evaluating the functionality grounding capabilities of existing vision-language models (VLMs). This split contains 1k samples at web resolution (1280 x 720) and 1k at mobile resolution (428x746).

Download this test split on [*Google Drive*](https://drive.google.com/file/d/1VfhVtcN1B4WSrb4DXJczQQ5zx1_pOvBu/view?usp=sharing).

Each test sample contains: 
* `image`: the GUI screenshot.
* `func`: the functionality annotation of a target element on the screenshot.
* `point`: the center point (X,Y) of the target element. Note that the coordinates are normalized with the range 0-100.
* `unnormalized_box`: the bounding box of the target element in the image coordinate frame.
* `elem_text`: the displayed or alt text of the element.
* `elem_tag`: the HTML tag of the element.
* `device`: the device type of the screenshot.

### Finetuning Code
1. Prepare Data

After downloading [the tar-format data](https://huggingface.co/datasets/WebAgent/AutoGUI-v1-zip), please generate a json file that records all samples with the absolute image paths required by the Qwen-VL model.

For example, the `conversations` field must start with a user message that looks like `"<img>path/to/autogui_625k/1_web.png</img>\n (instruction)"`

2. Finetuning [Qwen-VL-Chat](https://huggingface.co/Qwen/Qwen-VL-Chat)

Set the `data_path` in `finetune/finetune_autogui_lora.sh` and then run it.

### Evaluation code
Our evaluation code is adapted from lmms-eval. To evaluate a model on a specific UI grounding benchmark, run this command:

```
python3 -m accelerate.commands.launch
    --num_processes=8 \
    -m lmms_eval \
    --model autogui \
    --model_args pretrained=WebAgent/AutoGUI-Qwen-v0.1-LoRA \
    --tasks func_pred_rec,screenspot_rec \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix autogui_funcpred \
    --output_path ./logs/ \
    ["--limit", "0.01"] \ # For debugging
```

The evaluation tasks used in our paper include: `func_pred_rec`, `screenspot_rec`, `screenspot_v2_rec`, `refexp`, `motif`, `vwb_ag`, `vwb_eg`.

The supported models include: `autogui`, `qwen_vl_chat`, `llava_sglang`, `llava_hf`, `deepseek_vl_chat.py`, `cogagent`, `qwen2_vl`, `ferret_ui`, `UGround`, `OS-ATLAS` (Please see `lmms_eval/models`). If `autogui` is used, the `pretrained` argument can be either a LoRA model path that contains only the adapter or a merged model path.

## Acknowledgement
Our project codes are based on the [Qwen-VL](https://github.com/QwenLM/Qwen-VL), [SeeClick](https://github.com/njucckevin/SeeClick), and [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval). We thank the authors for their open-source works.

AutoGUI

AutoGUI 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

AutoGUI

AutoGUI 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README