macos-computer-use-skill

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

macos-computer-use-skill

★ 18 Open GitHub ↗

Standalone MCP server that gives AI agents full GUI control over macOS — screenshots, mouse, keyboard, apps, clipboard, and multi-display. Zero private dependencies.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">
  <img src="./assets/hero.svg" alt="macOS Computer-Use Skill" width="100%" />
  <h1>macOS Computer-Use Skill</h1>
  <p><strong>Standalone MCP server that gives AI agents full GUI control over macOS — screenshots, mouse, keyboard, apps, clipboard, and multi-display — with zero private dependencies.</strong></p>
  <br />
  <p>
    <img src="https://img.shields.io/badge/version-0.2.2-blue?style=flat-square" alt="version" />
    <img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license" />
    <img src="https://img.shields.io/badge/platform-macOS-lightgrey?style=flat-square&logo=apple" alt="platform" />
    <img src="https://img.shields.io/badge/node-%3E%3D20-339933?style=flat-square&logo=node.js" alt="node" />
    <img src="https://img.shields.io/badge/python-3.10%2B-3776AB?style=flat-square&logo=python&logoColor=white" alt="python" />
    <img src="https://img.shields.io/badge/MCP-compatible-8A2BE2?style=flat-square" alt="MCP" />
  </p>
  <p>
    <a href="#quick-start">Quick Start</a> ·
    <a href="#available-tools">Tools</a> ·
    <a href="#mcp-configuration">MCP Config</a> ·
    <a href="https://clawhub.ai/wimi321/computer-use-macos">ClawHub</a>
  </p>
</div>

---

## Features

| | Feature | Description |
|---|---|---|
| **Vision** | Screenshot & Display | Capture any display, enumerate monitors, zoom into regions |
| **Input** | Mouse & Keyboard | Click, drag, scroll, type, key combos, hold keys — with IME-safe clipboard routing |
| **Apps** | Application Control | Launch apps, detect frontmost app, list installed/running apps, tiered permission model |
| **Clipboard** | Read & Write | Full clipboard access for paste-based workflows |
| **Batch** | Action Batching | Chain multiple actions in a single MCP call for speed |
| **Runtime** | Zero-Config Bootstrap | Auto-creates Python virtualenv and installs dependencies on first run |
| **Portable** | Skill Packaging | Ships as a standalone skill — install once, works without the source repo |
| **Public** | No Private Dependencies | Built entirely on public packages: Node.js, Python, pyautogui, mss, Pillow, pyobjc |

## Quick Start

**1. Clone & build**

```bash
git clone https://github.com/wimi321/macos-computer-use-skill.git
cd macos-computer-use-skill
npm install && npm run build
```

**2. Run the MCP server**

```bash
node dist/cli.js
```

On first launch the server automatically creates a Python virtualenv in `.runtime/venv` and installs all runtime dependencies. No Claude desktop app, no private native modules.

**3. Or install from ClawHub**

```bash
clawhub install computer-use-macos
```

> [!NOTE]
> macOS requires **Accessibility** and **Screen Recording** permissions for the host process. The server checks both on startup and reports status through MCP.

## Architecture

```mermaid
flowchart LR
    A[AI Agent / MCP Client] --> B[MCP Server<br/>TypeScript + stdio]
    B --> C[Tool Layer<br/>28 MCP tools]
    B --> D[Python Bridge<br/>auto-bootstrapped venv]
    D --> E[pyautogui]
    D --> F[mss + Pillow]
    D --> G[pyobjc<br/>Cocoa + Quartz]
    E --> H[Mouse / Keyboard]
    F --> I[Screenshots]
    G --> J[Apps / Displays<br/>Clipboard / Windows]
```

## Available Tools

### Vision & Display

| Tool | Description |
|---|---|
| `screenshot` | Capture the current display as a JPEG image |
| `zoom` | Crop and zoom into a region of the last screenshot |
| `switch_display` | Switch the active capture target to a different monitor |

### Input

| Tool | Description |
|---|---|
| `left_click` | Left-click at a coordinate |
| `double_click` | Double-click at a coordinate |
| `triple_click` | Triple-click (select paragraph/line) |
| `right_click` | Right-click (context menu) |
| `middle_click` | Middle-click |
| `left_click_drag` | Click-and-drag between two points |
| `left_mouse_down` | Press and hold the left mouse button |
| `left_mouse_up` | Release the left mouse button |
| `mouse_move` | Move the cursor without clicking |
| `scroll` | Scroll in any direction at a coordinate |
| `type` | Type text (clipboard-routed on macOS to avoid IME corruption) |
| `key` | Press a key combo (e.g. `cmd+c`, `ctrl+shift+t`) |
| `hold_key` | Hold a key for a duration |
| `cursor_position` | Get the current cursor coordinates |

### Application & System

| Tool | Description |
|---|---|
| `open_application` | Launch a macOS application by name |
| `request_access` | Request access to interact with an application |
| `list_granted_applications` | List apps the current session has permission to control |
| `read_clipboard` | Read the system clipboard |
| `write_clipboard` | Write to the system clipboard |
| `wait` | Pause for a specified duration |

### Batch & Teach Mode

| Tool | Description |
|---|---|
| `computer_batch` | Execute multiple actions in a single call |
| `request_teach_access` | Request elevated access for teaching workflows |
| `teach_step` | Single-step action in teach mode |
| `teach_batch` | Batch actions in teach mode |

## MCP Configuration

Add to your MCP client config:

```json
{
  "mcpServers": {
    "computer-use": {
      "command": "node",
      "args": ["/absolute/path/to/macos-computer-use-skill/dist/cli.js"],
      "env": {
        "CLAUDE_COMPUTER_USE_DEBUG": "0",
        "CLAUDE_COMPUTER_USE_COORDINATE_MODE": "pixels"
      }
    }
  }
}
```

See [`examples/mcp-config.json`](./examples/mcp-config.json) for a ready-to-use template.

## Skill Install

This project ships as a self-contained skill at [`skill/computer-use-macos`](./skill/computer-use-macos).

**From ClawHub:**

```bash
clawhub install computer-use-macos
```

**From the repo:**

```bash
bash skill/computer-use-macos/scripts/install.sh
```

The installer copies the full project to `~/.codex/skills/computer-use-macos/project` — the skill keeps working even if the original clone is removed.

## Environment Variables

| Variable | Default | Description |
|---|---|---|
| `CLAUDE_COMPUTER_USE_DEBUG` | `0` | Enable verbose debug logging |
| `CLAUDE_COMPUTER_USE_COORDINATE_MODE` | `pixels` | Coordinate mode: `pixels` or `normalized_0_100` |
| `CLAUDE_COMPUTER_USE_CLIPBOARD_PASTE` | `1` | Prefer clipboard-based typing (IME-safe) |
| `CLAUDE_COMPUTER_USE_MOUSE_ANIMATION` | `0` | Animate mouse movement |
| `CLAUDE_COMPUTER_USE_HIDE_BEFORE_ACTION` | `0` | Hide overlay windows before actions |

## Requirements

| Requirement | Version |
|---|---|
| macOS | 12+ (Monterey or later) |
| Node.js | 20+ |
| Python | 3.10+ (ships with macOS or via Homebrew) |
| Permissions | Accessibility + Screen Recording |

Python dependencies (`pyautogui`, `mss`, `Pillow`, `pyobjc`) are installed automatically into an isolated virtualenv on first run.

## Repository Layout

```
macos-computer-use-skill/
├── src/
│   ├── cli.ts                    # Entry point
│   ├── server.ts                 # MCP server setup
│   ├── session.ts                # Session context factory
│   ├── computer-use/
│   │   ├── executor.ts           # macOS executor (bridges to Python)
│   │   ├── pythonBridge.ts       # Venv bootstrap + Python IPC
│   │   ├── hostAdapter.ts        # Host adapter factory
│   │   └── ...
│   └── vendor/computer-use-mcp/
│       ├── mcpServer.ts          # MCP server factory
│       ├── toolCalls.ts          # Tool dispatch logic
│       ├── tools.ts              # MCP tool schemas
│       └── ...
├── runtime/
│   ├── mac_helper.py             # Python runtime (pyautogui + pyobjc)
│   └── requirements.txt
├── skill/
│   └── computer-use-macos/       # Portable skill package
├── examples/
│   ├── mcp-config.json
│   └── env.sh.example
├── assets/
│   └── hero.svg
├── package.json
└── tsconfig.json
```

## Roadmap

- [ ] App icon extraction without private APIs
- [ ] Stronger nested helper-app filtering
- [ ] Automated MCP integration test suite
- [ ] Pre-built release artifacts for easier distribution

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.

## License

[MIT](./LICENSE)

## Acknowledgments

This project extracts and adapts reusable TypeScript computer-use logic from the Claude Code workflow, replacing the private native runtime with a fully standalone, publicly installable macOS implementation. Built on top of the [Model Context Protocol](https://modelcontextprotocol.io).

macos-computer-use-skill

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README