Open source MCP server

Draw. Speak. Gesture.
Your agent ships it.

Annotate localhost in the browser, capture your screen with OpenCV, add voice instructions, or control drawing with hand gestures — all over MCP for Cursor and compatible agents.

Add MCP Server View on GitHub

vision2agent

$ ▌

Features

More than draw-on-canvas

vision2agent extends visual MCP feedback with voice, gestures, and desktop capture — one toolkit for browser and native workflows.

Browser overlay

Proxy your dev server at 127.0.0.1:9742. Draw on the live UI with html2canvas capture and DOM hints for the agent.

Voice instructions

Speak after you annotate — transcript ships with the screenshot in one MCP response.

Hand gestures

Pinch to draw, palm to change color, thumbs up to submit — in browser or desktop OpenCV.

Screen canvas

Annotate any monitor with OpenCV — not limited to browser tabs.

Multimodal sessions

Combine screen draw + voice in a single submission via launch_multimodal_session.

Webcam aware

Integrated camera picker skips virtual cams when possible. check-camera CLI for diagnostics.

Cursor-ready

stdio MCP tools block until you submit — agent gets .last-session.json + image.

CLI testing

npm run browser-gesture and npm run gesture work without invoking MCP from chat.

Workflow

How it works

Three steps from sketch to shipped code.

Install & connect

Clone vision2agent, run npm install && npm run build, then add the MCP server to Cursor via .cursor/mcp.json.

Annotate your app

Launch browser overlay on localhost:3000 (open 127.0.0.1:9742). Draw, speak, or use gestures — then submit.

Agent implements

MCP returns screenshot, DOM hints, and transcript. Your agent edits code to match your visual intent.

Getting started

Add to your workspace

Set up vision2agent in Cursor in under a minute.

1. Build the server

terminal

git clone https://github.com/notzabir/vision2agent.git
cd vision2agent
npm install
npm run setup:python
npm run build

2. Add to .cursor/mcp.json

Use forward slashes and an absolute path to your dist/index.js.

.cursor/mcp.json

{
  "mcpServers": {
    "vision2agent": {
      "command": "node",
      "args": ["C:/path/to/vision2agent/dist/index.js"]
    }
  }
}

3. Try it in chat

prompt

Use vision2agent launch_browser_gesture_canvas on http://localhost:3000

Draw. Speak. Gesture.Your agent ships it.