Computer Use Agent Behavior Cloning: Collect and Replay

Open-source infrastructure for collecting, labeling, and replaying computer-use agent trajectories across desktops and multimodal models.

CUA Collector is built for community-driven Computer Use Agent research. It records human desktop demonstrations, turns them into labeled multimodal trajectories, and replays them through any OpenAI-compatible vision model. The project is designed to avoid platform lock-in, model lock-in, and closed data pipelines.

🔌 Bring your own model. You are no longer limited to Codex or Claude. Point the collector at any OpenAI-compatible vision endpoint — a hosted API, a local server, or your own gateway — right from the web panel. Add your BASE_URL, MODEL, and API_KEY, save, and switch between providers from a dropdown.

A multimodal agent applying to real ML jobs on LinkedIn — reference demo on the left, the live desktop on the right.

🤝 Open to collaboration. This is a community project — we want your demonstrations, model adapters, platform backends, and ideas. Star the repo, open an issue to share a direction, or send a PR. See Contributing & Collaboration and reach out at puma122707@gmail.com to build the open CUA data stack together.

Why This Project

Open Community Collaboration

The project is intended to be a shared foundation for collecting and improving computer-use behavior data. Researchers, builders, and agent developers can use the same pipeline to contribute demonstrations, compare models, label tasks, and turn successful agent runs into future references.

Bring Your Own Model

Not limited to Codex or Claude anymore. CUA Collector talks to any vision-capable model through an OpenAI-compatible API, so you can run it on the model you choose — a frontier hosted API, an open-weight model behind vLLM / Ollama / LM Studio, or your own custom gateway.

Configure it two ways:

• From the web panel (recommended): the Model provider card lets you add a provider (BASE_URL, MODEL, API_KEY), save it, and switch between saved providers from a dropdown. Keys are stored locally and shown masked.
• From a file: set BASE_URL, MODEL, and API_KEY in Extension_AgentCollector/.env.

The agent loop sends the current screenshot plus text trajectory memory, receives a structured next action, executes it, and records the result.

Platform-Independent Design

The collector separates desktop capture/input control from model reasoning. That means the same data and agent workflow can run across different desktop stacks and model backends.

Current launchers include:

Platform	Launcher	Backend
Ubuntu Wayland / GNOME	./run.sh	Native PipeWire + libevdev capture engine
Linux X11	./run_x11.sh	Python mss + pynput backend
Windows	.\run_win.ps1	Native Win32 backend when built, fallback backend otherwise
macOS	./run_mac.sh	Python cross-platform backend

Workflow

CUA Collector is organized around three steps:

1. Record a human demonstration.
2. Label the demonstration into a reusable trajectory reference.
3. Replay / run a task with a multimodal model using optional reference guidance and trajectory memory.

1. Record

Start the collector, open the web panel, describe the task, then perform the task manually. The collector captures actions automatically while the task is active — use it as a lightweight, always-on recorder for any desktop workflow.

Just press Ctrl+F8 and work: every click and keystroke is captured to data/<task_id>/ alongside pre/post screenshots.

2. Label

Select a recorded demo and label it. The labeling pass asks the configured model to describe each action and its expected outcome, producing a reusable reference under data_labeled/<TITLE>/.

3. Replay / Run

Choose a task, optionally select a labeled reference, set trajectory memory and step budget, then start an agent run. The model sees the current screenshot, recent trajectory memory, and optional reference steps, then returns one structured action at a time.

Each run is saved as data_labeled/BOT_<TITLE>/, so successful agent behavior can become training or guidance data for future runs.

Pick a labeled reference (here, APPLY_MACHINE_LEARNING_WORK), give a task, and watch the model drive the real desktop one action at a time.

Web Control Panel

The recommended interface is the local web control panel:

• Left side: agent control panel for choosing your model provider, recording, labeling, references, memory, step budget, and live trajectory.
• Right side: the real desktop environment where the task is executed.

This layout makes it easy to supervise an agent while seeing both its decisions and the actual UI operations — as shown in the demo at the top of this page.

Quick Start

Ubuntu Wayland / GNOME

git clone https://github.com/Zdong104/CUA_Collector.git
cd CUA_Collector
bash setup.sh
.venv/bin/pip install -r Extension_AgentCollector/requirements.txt

# Log out and log back in once so input permissions and the GNOME extension load.

./run.sh

In another terminal, start the web control panel:

cd Extension_AgentCollector
../.venv/bin/python control_panel.py

Then open:

http://127.0.0.1:8300

The collector exposes its local control API at http://127.0.0.1:8321.

Configure A Model

Bring your own model — you are not tied to Codex or Claude. Any OpenAI-compatible vision endpoint works.

The easiest way is directly in the web panel: open the Model provider card, fill in your API key, base URL, and model, then click Save. Configurations are written to a local .env for you, and you can keep several providers and switch between them from the dropdown (keys stay hidden).

For example:

API_KEY=sk-api
BASE_URL=https://api.minimaxi.com/v1
MODEL=MiniMax-M3

Prefer a file? Edit Extension_AgentCollector/.env directly:

BASE_URL=http://localhost:8000/v1
MODEL=default
API_KEY=
COLLECTOR_URL=http://127.0.0.1:8321

Linux X11

git clone https://github.com/Zdong104/CUA_Collector.git
cd CUA_Collector
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

./run_x11.sh

Windows

git clone https://github.com/Zdong104/CUA_Collector.git
cd CUA_Collector
python -m venv .venv
.venv/Scripts/python.exe -m pip install -r requirements.txt

cmake -S . -B build -DPython3_EXECUTABLE="$PWD\.venv\Scripts\python.exe"
cmake --build build --config Release
.\run_win.ps1

If PowerShell blocks script execution:

powershell -ExecutionPolicy Bypass -File .\run_win.ps1

macOS

git clone https://github.com/Zdong104/CUA_Collector.git
cd CUA_Collector
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

./run_mac.sh

Grant Screen Recording and Accessibility permissions to the terminal app or Python executable, then restart the launcher.

CLI Usage

The web panel is recommended, but each stage also has a CLI equivalent.

Label a recorded human demo:

cd Extension_AgentCollector
python annotate_reference.py --task-id 20260706_204844_5ceecc8c

Run an agent task with a labeled reference:

python model_runner.py \
  --task "Search Google for the stock price today" \
  --reference GO_CHROME_SEARCH_STOCK \
  --lookback 10 \
  --max-steps 20

Auto-label an example demo first, then run against it:

python model_runner.py \
  --task "Search Google for the stock price today" \
  --example 20260706_204844_5ceecc8c \
  --lookback 10

Data Layout

Raw recordings are saved under data/:

data/
  index.json
  <task_id>/
    task.json
    screenshots/
      action_0001_before.png
      action_0001_after.png

Labeled human references and agent runs are saved under data_labeled/:

data_labeled/
  index.json
  <TITLE>/
    task.json
    screenshots/
  BOT_<TITLE>/
    task.json
    screenshots/

Each labeled step includes:

• description: what the human or model is doing.
• action: the executable GUI action, usually represented as a pyautogui call.
• expected: what should visibly happen after the action.

Hotkeys

Hotkey	Action
Ctrl+F8	Start a new task
Ctrl+F12	End the current task
Ctrl+C	Quit the collector

The current collector captures actions automatically while a task is active.

Architecture

Human / Agent Action
        |
        v
Desktop Capture + Input Backend
        |
        v
Raw trajectory in data/<task_id>/
        |
        v
Model labeling
        |
        v
Reusable reference in data_labeled/<TITLE>/
        |
        v
Model replay / agent run
        |
        v
New trajectory in data_labeled/BOT_<TITLE>/

The native Linux Wayland collector uses a C++ capture engine:

• PipeWire screenshot stream
• in-memory ring buffer
• libevdev input monitoring
• pre-action and post-action screenshot correlation
• Python task management and local HTTP control API

The agent extension handles:

• web control panel
• model configuration
• reference labeling
• task execution
• live trajectory display
• bot trajectory persistence

Repository Layout

.
├── collector.py                    # automated collector and local API
├── agent_client.py                 # client for collector action API
├── cross_platform_capture.py       # fallback capture/input backend
├── run.sh                          # Ubuntu Wayland/GNOME launcher
├── run_x11.sh                      # Linux X11 launcher
├── run_win.ps1                     # Windows launcher
├── run_mac.sh                      # macOS launcher
├── setup.sh                        # Ubuntu Wayland/GNOME setup
├── setup_extension.sh              # GNOME cursor tracker extension setup
├── CMakeLists.txt                  # native capture module build
├── include/ src/ tests/            # C++ capture engine
├── Extension_AgentCollector/
│   ├── control_panel.py            # local web panel
│   ├── model_runner.py             # model-controlled replay/run loop
│   ├── annotate_reference.py       # trajectory labeling
│   ├── agent_common.py             # shared model/data helpers
│   └── panel/index.html            # web panel UI
├── data/                           # raw recordings
└── data_labeled/                   # labeled references and bot runs

Contributing & Collaboration

We are actively looking for collaborators. CUA Collector is meant to be a shared, open foundation for computer-use agent research, and it gets better with every demonstration, adapter, and idea the community brings.

Ways to get involved:

• ⭐ Star and watch the repo to follow progress.
• 💡 Share an idea or direction by opening an issue or a discussion — early feedback shapes the roadmap.
• 🔧 Send a pull request. High-value areas include:
  • new platform backends
  • model adapter examples
  • better labeling prompts and schemas
  • dataset tools and validators
  • replay evaluation metrics
  • documentation and demo trajectories
• 📬 Reach out directly at puma122707@gmail.com to collaborate on research, datasets, or integrations.

The goal is to make computer-use data collection and replay reproducible, inspectable, and open — and to build it together.

More Demos

Longer end-to-end screen recordings of the collector in action:

• assets/Demo.gif — full Linux Wayland session
• assets/WindowsDemo.gif — Windows session

Citation

@misc{dong2026cuacollector,
  author = {Zihan Dong},
  title = {ComputerUseAgent\_Collector},
  year = {2026},
  url = {https://github.com/Zdong104/Computer-Use-Agent_Behavior_Clone},
  note = {Computer Use Agent behavior cloning: collect and replay}
}

License

Research and non-commercial use are free.

For commercial use, contact: puma122707@gmail.com.