Z-ComputerUse V1.0.0

A production-grade Model Context Protocol (MCP) server that hands Windsurf IDE (or any MCP-compatible agent) total vision, mouse, and keyboard control of your Windows, macOS, or Linux machine — with hard-interrupt safety you can trigger at any moment.

Built for the Director. Controlled by the Director. You interrupt, it stops.

Highlights

• 38 MCP tools covering screenshots, live recording, mouse, keyboard, zoom, session control, runtime interrupt toggles, and hot-reloadable config
• Anthropic computer-use schema compatible — tool names match so Cascade / Claude recognize them natively
• Two independent kill switches:
  • Press Escape anywhere on your system
  • Move your mouse more than N pixels (configurable threshold)
• Live screen recording with a circular memory buffer and auto-save on interrupt
• Multi-monitor fast capture via mss
• Massive config.json — 9 sections, 50+ knobs, hot-reloadable
• Human-like input — configurable jitter, easing, timing variance
• Safety guards — rate limits, blocked regions, dangerous-key confirmation, session duration watchdog
• DPI-aware on Windows out of the box
• stdio transport — zero port conflicts, spawns directly from Windsurf

Installation

Option A: pip (simplest)

git clone https://github.com/ZannyTornadoCoding/Z-ComputerUse-MCP-Server
cd Z-ComputerUse-MCP-Server
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e .
copy config.example.json config.json

With optional extras (OCR + template matching + interrupt sound):

pip install -e ".[full]"

Option B: uv (fast, recommended)

git clone https://github.com/ZannyTornadoCoding/Z-ComputerUse-MCP-Server
cd Z-ComputerUse-MCP-Server
uv venv
.\.venv\Scripts\Activate.ps1
uv pip install -e ".[full]"
copy config.example.json config.json

Option C: one-shot installer

.\scripts\install.ps1

Windsurf Integration

Add this entry to ~/.codeium/windsurf/mcp_config.json (or ~/.codeium/windsurf-next/mcp_config.json for Windsurf Next):

{
    "mcpServers": {
        "z-computeruse": {
            "command": "python",
            "args": ["-m", "z_computeruse"],
            "env": {
                "Z_COMPUTERUSE_CONFIG": "<ABSOLUTE_PATH_TO_REPO>/config.json"
            }
        }
    }
}

Replace <ABSOLUTE_PATH_TO_REPO> with the full path to your cloned repository (e.g. C:/Users/you/code/Z-ComputerUse-MCP-Server on Windows, /home/you/code/Z-ComputerUse-MCP-Server on Linux).

Then press the refresh button in Windsurf's MCP panel. Cascade should pick up 38 tools under the z-computeruse server.

Tools Exposed

Screen & Vision

| Tool | Purpose | |------|---------| | screenshot | Full-screen capture with auto-scaling | | screenshot_region | Capture a rectangle | | zoom_region | Full-resolution crop for fine inspection | | get_screen_info | Monitor list, resolutions, active primary | | find_text_on_screen | OCR-based text locator (requires vision extras) | | find_image_on_screen | Template-matching locator (requires vision extras) |

Recording

| Tool | Purpose | |------|---------| | start_recording | Begin live frame capture to ring buffer | | stop_recording | Stop + optionally save to MP4 | | get_recent_frames | Retrieve N most recent frames | | get_recording_status | Running state, frame count, duration |

Mouse

| Tool | Purpose | |------|---------| | mouse_move | Move cursor to (x, y) | | mouse_position | Get current cursor position | | left_click / right_click / middle_click / double_click / triple_click | Click variants | | left_click_drag | Click-drag between two points | | scroll | Wheel scroll with direction + clicks | | mouse_down / mouse_up | Fine-grained click control |

Keyboard

| Tool | Purpose | |------|---------| | type_text | Type a string with human-like variance | | key | Press key or combo (ctrl+s, alt+tab) | | hold_key | Hold a key for N seconds | | key_down / key_up | Fine-grained key control |

Control & Session

| Tool | Purpose | |------|---------| | wait | Pause for N seconds (interruptible) | | get_status | Interrupt state, session info, safety stats | | reset_interrupt | Clear the interrupt flag so the agent can resume | | pause_session / resume_session | Temporarily halt all actions | | end_session | Finalize session, optionally save recording |

Config & Runtime Toggles

| Tool | Purpose | |------|---------| | get_config | Fetch current config (whole or by section) | | reload_config | Hot-reload config.json without restart; propagates interrupt changes live | | save_current_config | Persist in-memory config back to disk | | set_mouse_interrupt | Toggle mouse-movement kill switch on/off at runtime | | set_escape_interrupt | Toggle escape-key kill switch on/off at runtime | | set_interrupts_enabled | Master switch for the entire interrupt manager |

The Interrupt System (Total Control)

Z-ComputerUse runs two independent global listeners from the moment it starts:

1. Keyboard listener — watches for a single press of Esc (configurable) anywhere on the system.
2. Mouse listener — watches for any movement more than mouse_movement_threshold_px pixels away from where the bot expected the cursor to be.

When either fires:

1. A shared threading.Event gets set.
2. The current tool call aborts (raises InterruptedError caught at tool boundary).
3. All subsequent tool calls return {"interrupted": true, "reason": "escape_key" | "mouse_movement"} until you explicitly call reset_interrupt.
4. If save_on_interrupt is on, the last buffer_seconds of screen recording is flushed to disk so you can review what it was doing.
5. If sound_on_interrupt is on, a sound plays.

To resume: tell Cascade something like "I interrupted because X. The situation is now Y. Call reset_interrupt and continue with Z." Cascade then calls the tool, the flag clears, and it proceeds.

The bot can't self-reset without Cascade calling the tool, and Cascade can't call the tool unless you tell it to — so you always hold the kill switch.

Config Highlights

All options in config.json. Sections: server, display, recording, mouse, keyboard, interrupt, safety, vision, session, advanced.

Key knobs to tune:

| Key | Effect | |-----|--------| | interrupt.mouse_movement_threshold_px | Lower = more sensitive (default 35 px) | | interrupt.mouse_movement_grace_ms | Time window where programmatic moves don't trigger interrupt | | mouse.move_duration_seconds | Slower = more human-looking | | mouse.human_like_movement | Adds easing + jitter | | display.screenshot_scale_max_edge_px | 1568 is LLM-safe; higher = sharper but bigger tokens | | recording.fps | 5 is a good balance of quality vs. memory | | safety.max_actions_per_minute | Rate limiter | | safety.max_session_duration_minutes | Hard session cap | | safety.blocked_regions | List of [x, y, w, h] rectangles the bot can't click |

Environment variable Z_COMPUTERUSE_CONFIG overrides the default config path.

Running Standalone

For testing outside Windsurf:

python -m z_computeruse

It will listen on stdio. Pair with MCP Inspector:

npx -y @modelcontextprotocol/inspector python -m z_computeruse

Requirements

• Python 3.11+
• Windows 10/11, macOS 12+, or Linux with X11
• mss, pyautogui, pynput (installed automatically)
• Optional: Tesseract OCR binary (for find_text_on_screen), ffmpeg-bundled automatically via imageio-ffmpeg

Safety & Disclaimer

This tool gives an AI direct, unchecked control of your mouse, keyboard, and screen. It is designed for power users who understand the risks. Always:

• Keep one hand near the mouse / Escape key
• Start with safety.max_actions_per_minute low until you trust the agent
• Review recordings after every session
• Never run on a machine with sensitive credentials exposed to the agent

License

MIT — see LICENSE.