KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agents #developer-tools #typescript #codex #cli #claude-code #mcp #rust #deepseek-harness #ai-agent

GlassesOpenClaw

★ 19 Open GitHub ↗

Real-time voice + vision on Meta Ray-Ban or iPhone, with Gemini Live. OpenClaw (Alan) runs actions on your Mac — reminders, messages, share to Instagram, and more.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# GlassesOpenClaw

![GlassesOpenClaw](https://raw.githubusercontent.com/zhanghaonan777/GlassesOpenClaw/main/assets/glassesopenclaw_q_banner.png)

Use Meta Ray-Ban glasses or iPhone for real-time voice and camera; OpenClaw (Alan) runs actions on your Mac when needed.

*Example: “Post this video to Instagram”.*

![Park: share to Ins](https://raw.githubusercontent.com/zhanghaonan777/GlassesOpenClaw/main/assets/glassesopenclaw_park_ins.png)

### App screenshots

| iPhone mode | Glass mode |
|-------------|------------|
| ![iPhone mode](https://raw.githubusercontent.com/zhanghaonan777/GlassesOpenClaw/main/assets/screenshots/screenshot_iphone_mode.png) | ![Glass mode](https://raw.githubusercontent.com/zhanghaonan777/GlassesOpenClaw/main/assets/screenshots/screenshot_glass_mode.png) |
| **iPhone mode**: camera + mic, then QA or Streaming Live. | **Glass mode**: resolution (Low/Med/High), then QA or Streaming Live. |

| Live: vision + AI | QA: voice chat + tasks |
|-------------------|-------------------------|
| ![Live vision](https://raw.githubusercontent.com/zhanghaonan777/GlassesOpenClaw/main/assets/screenshots/screenshot_live_vision.png) | ![QA chat](https://raw.githubusercontent.com/zhanghaonan777/GlassesOpenClaw/main/assets/screenshots/screenshot_qa_chat.png) |
| Ask "What's this?" — Gemini sees the scene and answers (e.g. chandelier). | **QA**: multi-turn voice chat; set reminders, tasks; OpenClaw runs actions. |

---

## Installation

Follow these steps to get the app running on your iPhone (with or without glasses).

### Prerequisites

- **macOS** with **Xcode 15+** and **iOS 17+** device
- **Gemini API key** — [create one](https://aistudio.google.com/apikey) at Google AI Studio (free)
- **Meta Ray-Ban glasses** (optional) — the app can run in **iPhone mode** using the phone camera and mic

### Step 1 — Clone the repo

```bash
git clone https://github.com/YOUR_USERNAME/GlassesOpenClaw.git
cd GlassesOpenClaw
```

### Step 2 — Open the iOS project

```bash
cd samples/CameraAccess
open CameraAccess.xcodeproj
```

### Step 3 — Configure the Gemini API key

The app needs a Gemini API key to talk to the Live API.

- **In-app (recommended):** After the first run, open **Settings** (gear icon) and paste your key in the Gemini field.
- **Or before first run:** Copy the secrets template and edit:

```bash
cp CameraAccess/Secrets.swift.example CameraAccess/Secrets.swift
```

Then set `geminiAPIKey = "your-key"` in `Secrets.swift`.

### Step 4 — Build and run

1. In Xcode, choose your **iPhone** as the run destination.
2. Press **Run** (⌘R).
3. On the device, allow **microphone** (and camera if using iPhone mode) when prompted.

### Step 5 — Use the app (iPhone mode)

1. Tap **「Start on iPhone」** to use the phone’s back camera and mic.
2. Tap the **AI button** to start a **Live** voice+vision session.
3. Speak naturally — the assistant sees the camera feed and replies with voice.

No glasses required for this path.

### Step 6 — Using Meta Ray-Ban glasses (optional)

To stream from the glasses camera and use the glasses mic:

1. **Enable Developer Mode** in the **Meta AI** app on your iPhone: open Meta AI → **Settings** → **App Info** → tap **App version** five times → back to Settings → turn on **Developer Mode**.
2. In the app, tap **「Start Streaming」**, then the **AI** button for Live. QA (voice-only) is available from the streaming screen.

---

## Optional: OpenClaw (actions gateway)

Without OpenClaw, the assistant is **voice + vision only**. With OpenClaw on your Mac, it can run actions: send messages, search the web, manage lists, control smart home, etc., via the **execute** tool and **Alan** (艾伦).

### Install and enable the gateway

1. Install and configure [OpenClaw](https://github.com/nichochar/openclaw) on your Mac.
2. In `~/.openclaw/openclaw.json`, enable the gateway and chat completions:

```json
{
  "gateway": {
    "port": 18789,
    "bind": "lan",
    "auth": { "mode": "token", "token": "your-gateway-token" },
    "http": {
      "endpoints": { "chatCompletions": { "enabled": true } }
    }
  }
}
```

3. Start the gateway: `openclaw gateway restart`  
   Check: `curl http://localhost:18789/health`

### Point the app at OpenClaw

- **In-app:** Settings → OpenClaw: set **Host** (e.g. `http://Your-Mac.local`), **Port** `18789`, **Gateway token** (same as in `openclaw.json`).
- **Or** in `Secrets.swift`: `openClawHost`, `openClawPort`, `openClawGatewayToken`.

Mac hostname: **System Settings → General → Sharing** (e.g. `Johns-MacBook-Pro.local`).

### Remote access (ngrok)

When the iPhone is not on the same Wi‑Fi as the Mac:

1. Install [ngrok](https://ngrok.com), create a free domain at [dashboard.ngrok.com/domains](https://dashboard.ngrok.com/domains).
2. Run: `ngrok http 18789 --domain=your-name.ngrok-free.app` (if you use a proxy, run `unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY` first).
3. In app Settings → OpenClaw: **Host** = `https://your-name.ngrok-free.app`, **Port** = empty, **Gateway token** unchanged.
4. Save and tap **Test OpenClaw**. Keep the ngrok process running while using the app.

---

## What you can do

- Describe what’s in front of you (glasses or phone camera).
- Lists, reminders, messages — handled by OpenClaw/Alan on your Mac.
- Messaging (WhatsApp, Telegram, iMessage, etc.) via OpenClaw.
- General or search: the model answers; heavier tasks use OpenClaw.

**Live** is real-time voice and vision over WebSocket; **QA** is voice-only chat (Gemini REST, OpenAI, or custom endpoint). You can toggle video in Live and use Chinese-friendly ASR/TTS in Settings.

---

## How it works

![How it works](https://raw.githubusercontent.com/zhanghaonan777/GlassesOpenClaw/main/assets/how_it_works.png)

- **Glasses or iPhone** send video (~1 fps) and mic audio to the **app**.
- The app talks to the **Gemini Live API** (WebSocket): sends frames + audio, receives **audio** and optional **tool calls**.
- When the model calls the **execute** tool, the app forwards the task to **OpenClaw** on your Mac; results go back as **toolResponse** and are also injected as text so the model can announce them. See [Live tool flow](samples/CameraAccess/docs/LIVE_TOOL_FLOW.md) for details.

Stack: [Meta Wearables DAT SDK](https://github.com/facebook/meta-wearables-dat-ios), [Gemini Live API](https://ai.google.dev/gemini-api/docs/live), [OpenClaw](https://github.com/nichochar/openclaw) (optional).

---

## Requirements

| Item | Requirement |
|------|-------------|
| OS | iOS 17.0+ |
| IDE | Xcode 15.0+ |
| API | Gemini API key |
| Hardware | iPhone (required); Meta Ray-Ban glasses (optional); OpenClaw on Mac (optional) |

---

## Troubleshooting

| Issue | What to do |
|-------|------------|
| **「Gemini API key not configured」** | Add the key in **Settings** or in `Secrets.swift` (from `Secrets.swift.example`). |
| **OpenClaw connection timeout** | Same Wi‑Fi as Mac; gateway running (`openclaw gateway restart`); host = Mac Bonjour name or ngrok URL; port empty for HTTPS ngrok. |
| **ngrok ERR_NGROK_9009** | Unset proxy: `unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY` before `ngrok`. |
| **ngrok domain error** | Create a free domain at [dashboard.ngrok.com/domains](https://dashboard.ngrok.com/domains) and use `--domain=…`. |
| **Echo in iPhone mode** | App mutes mic while AI speaks; lower volume if needed. |
| **Model doesn’t hear me** | Grant microphone access; speak clearly (aggressive VAD). |
| **OpenClaw duplicate tabs** | Known upstream [OpenClaw CDP behavior](https://github.com/nichochar/openclaw/issues/13851); try `profile: "openclaw"` for Chrome. |

DAT / glasses: [Meta Wearables docs](https://wearables.developer.meta.com/docs/develop/) and [DAT discussions](https://github.com/facebook/meta-wearables-dat-ios/discussions).

---

## License and origin

This project is based on [VisionClaw](https://github.com/sseanliu/VisionClaw) (Copyright (c) Meta Platforms, Inc. and affiliates). It is licensed under the [LICENSE](LICENSE) in this repo. Use of the Wearables Device Access Toolkit is subject to [Meta Wearables Developer Terms](https://wearables.developer.meta.com/terms) and [Acceptable Use Policy](https://wearables.developer.meta.com/acceptable-use-policy).

GlassesOpenClaw

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README