Open-AutoGLM-App

KO | EN

gitlite — search

#python #typescript #javascript #android #cpp #html #php #nodejs #react #video #python3 #docker

Open-AutoGLM-App

★ 29 Open GitHub ↗

只需一个APP就能实现AutoGLM的功能

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Open-AutoGLM-App

An experimental Android AI automation assistant powered by visual language models.

<a href="#features">Features</a> •
 <a href="#quick-start">Quick Start</a> •
 <a href="#usage">Usage</a> •
 <a href="#architecture">Architecture</a> •
 <a href="#development">Development</a>

---

## Overview

Open-AutoGLM-App is an open-source Android automation project that explores how visual language models (VLMs) can understand a phone screen and execute user tasks through accessibility or Shizuku-based actions.

The project focuses on a local-first Android workflow:

1. The user describes a task in natural language.
2. The app captures the current screen.
3. A VLM analyzes the screen and returns structured actions.
4. The Android execution layer performs actions such as tap, swipe, input, launch, back, home, done, or ask-user.
5. The loop continues until the task is completed or user intervention is required.

Example tasks:

- "Open WeChat and send a message to Zhang San."
- "Search for Bluetooth earphones and sort by sales."
- "Open Settings and increase the screen brightness."

> This project is experimental. UI automation may fail on unsupported apps, unstable layouts, model errors, permission issues, or device-specific behavior. Please review every action before using it for sensitive tasks.

## Project Status

The project is still in an early experimental stage. It is suitable for learning, prototyping, and exploring mobile AI automation, but it should not be treated as a production-grade automation system.

Current focus areas:

- VLM-based screen understanding
- Set-of-Marks UI grounding
- Android Accessibility Service and Shizuku execution
- Multi-provider model configuration
- Script-like debugging commands
- Task list import/export experiments
- Frontend/native bridge for a hybrid Android UI

## Features

### Core Automation

- **Natural-language task execution**: describe a task and let the app plan and execute actions step by step.
- **Visual language model support**: connect to OpenAI, Claude, Gemini, or OpenAI-compatible providers.
- **Set-of-Marks (SoM)**: overlay numbered marks on interactive screen elements so the model can refer to element IDs instead of estimating raw coordinates.
- **Multiple execution modes**: use Android Accessibility Service or Shizuku/ADB-style execution depending on device setup.
- **Action loop**: repeatedly observe the screen, ask the model for the next action, execute it, and continue.
- **Manual control**: pause, resume, stop, or debug automation at any time.
- **Detailed logs**: inspect model responses, executed actions, and task progress.

### User Interface

- **Hybrid Android UI**: Kotlin native layer with a Vue 3 frontend.
- **Dark mode**: follows the system theme.
- **Floating window support**: observe logs or task status while automation is running.
- **Task list management**: create, import, export, and run script-like task sequences.
- **Repeat blocks**: experimental support for repeated command blocks in task scripts.

### Safety-Oriented Behavior

- **Local-first design**: the app runs locally on the device except for requests sent to the configured model provider.
- **Permission awareness**: automation requires explicit Accessibility, Shizuku, or related Android permissions.
- **Stop controls**: users can stop task execution when behavior is unexpected.
- **Sensitive-task warning**: do not use the app for payments, irreversible operations, private data handling, or actions that violate app/platform rules.

## Set-of-Marks (SoM)

Set-of-Marks is one of the core UI-grounding techniques used in this project. The app places numbered visual marks on detected interactive elements before sending the screenshot to the model.

Instead of asking the model to estimate exact pixel coordinates, the model can return an action such as:

```text
Tap mark 5
```

The app then maps mark `5` back to the detected UI element and performs the tap at the corresponding location.

### Why SoM Helps

| Approach | Description | Limitation |
| --- | --- | --- |
| Raw coordinates | The model estimates a pixel location such as `(540, 1280)` | Sensitive to screen size, scaling, layout changes, and model precision |
| Set-of-Marks | The model chooses a visible numbered mark | Easier to inspect, debug, and align with UI elements |

SoM improves element targeting and makes model decisions easier to trace, but it is not guaranteed to be perfect. Mark detection, UI hierarchy extraction, screenshots, and model interpretation can still fail.

### Implementation Flow

1. Read the current UI tree through Accessibility Service or Shizuku-supported execution.
2. Detect clickable or otherwise actionable UI elements.
3. Draw numbered marks on the screenshot.
4. Send the marked screenshot and task context to the model.
5. Parse the model's structured action.
6. Map the selected mark ID back to an element position.
7. Execute the action on the Android device.

## Experimental Features

The following features are experimental and may change frequently.

### Task List Management

Task lists allow users to create and run script-like command sequences.

Supported workflows include:

- Create a task list in the visual editor.
- Import commands from chat history.
- Import `.txt` files with command sequences.
- Export task lists for backup or sharing.

### Repeat Blocks

Task scripts can contain repeated command blocks:

```text
#repeat_start
#tap 500,800
#wait 1000
#repeat_end 10
```

The editor displays repeat regions with visual grouping and indentation to make command blocks easier to read.

## Quick Start

### Requirements

- Android 7.0 or later
- An API key for at least one supported model provider
- Accessibility Service permission, or Shizuku configured on the device
- Floating window permission if you want to use the floating UI
- A VLM that can understand screenshots and follow structured action prompts

### Installation

1. Download the latest APK from the [Releases](https://github.com/dascard/Open-AutoGLM-App/releases) page.
2. Install the APK on your Android device.
3. Enable the required execution permission:
   - Android Accessibility Service, or
   - Shizuku-based execution.
4. Open the app and configure your model provider:
   - API key
   - Base URL
   - Model name
   - Provider type
5. Enter a task and start execution.

## Usage

### Supported Model Providers

| Provider | Example Model | Notes |
| --- | --- | --- |
| OpenAI | `gpt-4o`, `gpt-4o-mini`, or newer VLM-capable models | Official API or compatible endpoints |
| Claude | Claude vision-capable models | Anthropic API or compatible gateways |
| Gemini | Gemini vision-capable models | Google AI Studio or compatible endpoints |
| Other | Any VLM with an OpenAI-compatible API | Requires compatible request and response behavior |

The actual automation quality depends heavily on the selected model's visual understanding, instruction following, and ability to return valid actions.

### Debug Commands

Commands starting with `#` can be used to test low-level actions directly:

```text
#tap 500,800
#swipe 500,1500,500,500
#type hello world
#enter
#back
#home
#launch Chrome
#screenshot
```

### Launch Command

`#launch` uses strict matching by default:

- Exact app names are preferred.
- Package names are supported, such as `#launch com.android.chrome`.
- If no exact match is found, the app may show similar app names for manual selection.

## Architecture

```text
Open-AutoGLM-App/
├── app/                          # Android module
│   ├── src/main/java/com/autoglm/app/
│   │   ├── MainActivity.kt        # Hybrid WebView container
│   │   ├── core/
│   │   │   ├── AIClient.kt        # Multi-provider AI API client
│   │   │   ├── TaskExecutor.kt    # Task execution engine
│   │   │   └── Action.kt          # Action definitions
│   │   ├── service/
│   │   │   └── AutoGLMAccessibilityService.kt
│   │   └── ui/
│   │       └── WebAppInterface.kt # JavaScript-Kotlin bridge
│   └── src/main/res/
├── frontend/                     # Vue 3 frontend
│   ├── src/
│   │   ├── App.vue
│   │   └── Bridge.ts
│   └── ...
└── docs/                         # Documentation assets
```

### Core Flow

```mermaid
graph TD
    A[User enters a task] --> B[Capture screen]
    B --> C[Build marked screenshot and context]
    C --> D[Send request to VLM]
    D --> E{Model returns action}
    E -->|Tap / Swipe / Input| F[Execute UI action]
    E -->|Launch| G[Open target app]
    E -->|Back / Home| H[System navigation]
    E -->|AskUser| I[Request user intervention]
    E -->|Done| J[Task completed]
    F --> B
    G --> B
    H --> B
    I --> B
```

## Development

### Build from Source

```bash
git clone https://github.com/dascard/Open-AutoGLM-App.git
cd Open-AutoGLM-App

cd frontend
npm install
npm run build
cd ..

./gradlew assembleDebug
```

The debug APK will be generated under:

```text
app/build/outputs/apk/debug/app-debug.apk
```

### Tech Stack

Android:

- Kotlin
- Kotlin Coroutines
- OkHttp
- Gson
- AndroidX
- Accessibility Service
- Shizuku integration

Frontend:

- Vue 3
- TypeScript
- Vite
- Tailwind CSS

### Adding a New Model Provider

1. Add a provider entry in `AIClient.kt`.
2. Implement request construction for the provider.
3. Implement response parsing.
4. Validate the provider with screenshot-based tasks.
5. Document any provider-specific limitations.

## Contributing

Issues and pull requests are welcome.

Good contribution areas include:

- Device compatibility reports
- UI automation reliability improvements
- Safer action validation
- Better prompt templates
- Model provider compatibility
- Documentation improvements
- Test cases for command parsing and execution behavior

Before opening a pull request, please make sure your change is focused and explain how it was tested.

## License

This project is licensed under the [Apache License 2.0](LICENSE).

## Acknowledgements

This project is inspired by and references ideas from the following open-source projects:

- [AutoGLM](https://github.com/zai-org/Open-AutoGLM)
- [Open-AutoGLM-Hybrid](https://github.com/xietao778899-rgb/Open-AutoGLM-Hybrid)

Thanks to the open-source community for making mobile AI automation research and experimentation easier.

## Disclaimer

This project is provided for learning, personal experimentation, and open-source research.

Do not use it for:

- Illegal activities
- Violating app or platform rules
- Bypassing security controls
- Payments or irreversible transactions
- Sensitive private-data operations without supervision

You are responsible for the tasks you run and the consequences of using this tool.

---

Made with ❤️ by the Open-AutoGLM community

Open-AutoGLM-App

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README