每日备份 2026-06-10 12:57:13

config/openclaw/skills/desktop-control/AI_AGENT_GUIDE.md

@@ -0,0 +1,448 @@
+# AI Desktop Agent - Cognitive Automation Guide
+
+## 🤖 What Is This?
+
+The **AI Desktop Agent** is an intelligent layer on top of the basic desktop control that **understands** what you want and figures out how to do it autonomously.
+
+Unlike basic automation that requires exact instructions, the AI Agent:
+- **Understands natural language** ("Draw a cat in Paint")
+- **Plans the steps** automatically
+- **Executes autonomously** 
+- **Adapts** based on what it sees
+
+---
+
+## 🎯 What Can It Do?
+
+### ✅ Autonomous Drawing
+```python
+from skills.desktop_control.ai_agent import AIDesktopAgent
+
+agent = AIDesktopAgent()
+
+# Just describe what you want!
+agent.execute_task("Draw a circle in Paint")
+agent.execute_task("Draw a star in MS Paint")
+agent.execute_task("Draw a house with a sun")
+```
+
+**What it does:**
+1. Opens MS Paint
+2. Selects pencil tool
+3. Figures out how to draw the requested shape
+4. Draws it autonomously
+5. Takes a screenshot of the result
+
+### ✅ Autonomous Text Entry
+```python
+# It figures out where to type
+agent.execute_task("Type 'Hello World' in Notepad")
+agent.execute_task("Write an email saying thank you")
+```
+
+**What it does:**
+1. Opens Notepad (or finds active text editor)
+2. Types the text naturally
+3. Formats if needed
+
+### ✅ Autonomous Application Control
+```python
+# It knows how to open apps
+agent.execute_task("Open Calculator")
+agent.execute_task("Launch Microsoft Paint")
+agent.execute_task("Open File Explorer")
+```
+
+### ✅ Autonomous Game Playing (Advanced)
+```python
+# It will try to play the game!
+agent.execute_task("Play Solitaire for me")
+agent.execute_task("Play Minesweeper")
+```
+
+**What it does:**
+1. Analyzes the game screen
+2. Detects game state (cards, mines, etc.)
+3. Decides best move
+4. Executes the move
+5. Repeats until win/lose
+
+---
+
+## 🏗️ How It Works
+
+### Architecture
+
+```
+User Request ("Draw a cat")
+    ↓
+Natural Language Understanding
+    ↓
+Task Planning (Step-by-step plan)
+    ↓
+Step Execution Loop:
+    - Observe Screen (Computer Vision)
+    - Decide Action (AI Reasoning)
+    - Execute Action (Desktop Control)
+    - Verify Result
+    ↓
+Task Complete!
+```
+
+### Key Components
+
+1. **Task Planner** - Breaks down high-level tasks into steps
+2. **Vision System** - Understands what's on screen (screenshots, OCR, object detection)
+3. **Reasoning Engine** - Decides what to do next
+4. **Action Executor** - Performsthe actual mouse/keyboard actions
+5. **Feedback Loop** - Verifies actions succeeded
+
+---
+
+## 📋 Supported Tasks (Current)
+
+### Tier 1: Fully Automated ✅
+
+| Task Pattern | Example | Status |
+|-------------|---------|---------|
+| Draw shapes in Paint | "Draw a circle" | ✅ Working |
+| Basic text entry | "Type Hello" | ✅ Working |
+| Launch applications | "Open Paint" | ✅ Working |
+
+### Tier 2: Partially Automated 🔨
+
+| Task Pattern | Example | Status |
+|-------------|---------|---------|
+| Form filling | "Fill out this form" | 🔨 In Progress |
+| File operations | "Copy these files" | 🔨 In Progress |
+| Web navigation | "Find on Google" | 🔨 Planned |
+
+### Tier 3: Experimental 🧪
+
+| Task Pattern | Example | Status |
+|-------------|---------|---------|
+| Game playing | "Play Solitaire" | 🧪 Experimental |
+| Image editing | "Resize this photo" | 🧪 Planned |
+| Code editing | "Fix this bug" | 🧪 Research |
+
+---
+
+## 🎨 Example: Drawing in Paint
+
+### Simple Request
+```python
+agent = AIDesktopAgent()
+result = agent.execute_task("Draw a circle in Paint")
+
+# Check result
+print(f"Status: {result['status']}")
+print(f"Steps taken: {len(result['steps'])}")
+```
+
+### What Happens Behind the Scenes
+
+**1. Planning Phase:**
+```
+Plan generated:
+  Step 1: Launch MS Paint
+  Step 2: Wait 2s for Paint to load
+  Step 3: Activate Paint window
+  Step 4: Select pencil tool (press 'P')
+  Step 5: Draw circle at canvas center
+  Step 6: Screenshot the result
+```
+
+**2. Execution Phase:**
+```
+[✓] Launched Paint via Win+R → mspaint
+[✓] Waited 2.0s
+[✓] Activated window "Paint"
+[✓] Pressed 'P' to select pencil
+[✓] Drew circle with 72 points
+[✓] Screenshot saved: drawing_result.png
+```
+
+**3. Result:**
+```python
+{
+    "task": "Draw a circle in Paint",
+    "status": "completed",
+    "success": True,
+    "steps": [... 6 steps ...],
+    "screenshots": [... 6 screenshots ...],
+}
+```
+
+---
+
+## 🎮 Example: Game Playing
+
+```python
+agent = AIDesktopAgent()
+
+# Play a simple game
+result = agent.execute_task("Play Solitaire for me")
+```
+
+### Game Playing Loop
+
+```
+1. Analyze screen → Detect cards, positions
+2. Identify valid moves → Find legal plays
+3. Evaluate moves → Which is best?
+4. Execute move → Click and drag card
+5. Repeat until game ends
+```
+
+### Game-Specific Intelligence
+
+The agent can learn patterns for:
+- **Solitaire**: Card stacking rules, suit matching
+- **Minesweeper**: Probability calculations, safe clicks
+- **2048**: Tile merging strategy
+- **Chess** (if integrated with engine): Move evaluation
+
+---
+
+## 🧠 Enhancing the AI
+
+### Adding Application Knowledge
+
+```python
+# In ai_agent.py, add to app_knowledge:
+
+self.app_knowledge = {
+    "photoshop": {
+        "name": "Adobe Photoshop",
+        "launch_command": "photoshop",
+        "common_actions": {
+            "new_layer": {"hotkey": ["ctrl", "shift", "n"]},
+            "brush_tool": {"hotkey": ["b"]},
+            "eraser": {"hotkey": ["e"]},
+        }
+    }
+}
+```
+
+### Adding Custom Task Patterns
+
+```python
+# Add a custom planning method
+def _plan_photo_edit(self, task: str) -> List[Dict]:
+    """Plan for photo editing tasks."""
+    return [
+        {"type": "launch_app", "app": "photoshop"},
+        {"type": "wait", "duration": 3.0},
+        {"type": "open_file", "path": extracted_path},
+        {"type": "apply_filter", "filter": extracted_filter},
+        {"type": "save_file"},
+    ]
+```
+
+---
+
+## 🔥 Advanced: Vision + Reasoning
+
+### Screen Analysis
+
+The agent can analyze screenshots to:
+- **Detect UI elements** (buttons, text fields, menus)
+- **Read text** (OCR for labels, instructions)
+- **Identify objects** (icons, images, game pieces)
+- **Understand layout** (where things are)
+
+```python
+# Analyze what's on screen
+analysis = agent._analyze_screen()
+
+print(analysis)
+# Output:
+# {
+#     "active_window": "Untitled - Paint",
+#     "mouse_position": (640, 480),
+#     "detected_elements": [...],
+#     "text_found": [...],
+# }
+```
+
+### Integration with OpenClaw LLM
+
+```python
+# Future: Use OpenClaw's LLM for reasoning
+agent = AIDesktopAgent(llm_client=openclaw_llm)
+
+# The agent can now:
+# - Reason about complex tasks
+# - Understand context better
+# - Plan more sophisticated workflows
+# - Learn from feedback
+```
+
+---
+
+## 🛠️ Extending for Your Needs
+
+### Add Support for New Apps
+
+1. **Identify the app**
+2. **Document common actions**
+3. **Add to knowledge base**
+4. **Create planning method**
+
+Example: Adding Excel support
+
+```python
+# Step 1: Add to app_knowledge
+"excel": {
+    "name": "Microsoft Excel",
+    "launch_command": "excel",
+    "common_actions": {
+        "new_sheet": {"hotkey": ["shift", "f11"]},
+        "sum_formula": {"action": "type", "text": "=SUM()"},
+    }
+}
+
+# Step 2: Create planner
+def _plan_excel_task(self, task: str) -> List[Dict]:
+    return [
+        {"type": "launch_app", "app": "excel"},
+        {"type": "wait", "duration": 2.0},
+        # ... specific Excel steps
+    ]
+
+# Step 3: Hook into main planner
+if "excel" in task_lower or "spreadsheet" in task_lower:
+    return self._plan_excel_task(task)
+```
+
+---
+
+## 🎯 Real-World Use Cases
+
+### 1. Automated Form Filling
+```python
+agent.execute_task("Fill out the job application with my resume data")
+```
+
+### 2. Batch Image Processing
+```python
+agent.execute_task("Resize all images in this folder to 800x600")
+```
+
+### 3. Social Media Posting
+```python
+agent.execute_task("Post this image to Instagram with caption 'Beautiful sunset'")
+```
+
+### 4. Data Entry
+```python
+agent.execute_task("Copy data from this PDF to Excel spreadsheet")
+```
+
+### 5. Testing
+```python
+agent.execute_task("Test the login form with invalid credentials")
+```
+
+---
+
+## ⚙️ Configuration
+
+### Enable/Disable Failsafe
+```python
+# Safe mode (default)
+agent = AIDesktopAgent(failsafe=True)
+
+# Fast mode (no failsafe)
+agent = AIDesktopAgent(failsafe=False)
+```
+
+### Set Max Steps
+```python
+# Prevent infinite loops
+result = agent.execute_task("Play game", max_steps=100)
+```
+
+### Access Action History
+```python
+# Review what the agent did
+print(agent.action_history)
+```
+
+---
+
+## 🐛 Debugging
+
+### View Step-by-Step Execution
+```python
+result = agent.execute_task("Draw a star in Paint")
+
+for i, step in enumerate(result['steps'], 1):
+    print(f"Step {i}: {step['step']['description']}")
+    print(f"  Success: {step['success']}")
+    if 'error' in step:
+        print(f"  Error: {step['error']}")
+```
+
+### View Screenshots
+```python
+# Each step captures before/after screenshots
+for screenshot_pair in result['screenshots']:
+    before = screenshot_pair['before']
+    after = screenshot_pair['after']
+    
+    # Display or save for analysis
+    before.save(f"step_{screenshot_pair['step']}_before.png")
+    after.save(f"step_{screenshot_pair['step']}_after.png")
+```
+
+---
+
+## 🚀 Future Enhancements
+
+Planned features:
+
+- [ ] **Computer Vision**: OCR, object detection, UI element recognition
+- [ ] **LLM Integration**: Natural language understanding with OpenClaw LLM
+- [ ] **Learning**: Remember successful patterns, improve over time
+- [ ] **Multi-App Workflows**: "Get data from Chrome and put in Excel"
+- [ ] **Voice Control**: "Alexa, draw a cat in Paint"
+- [ ] **Autonomous Debugging**: Fix errors automatically
+- [ ] **Game AI**: Reinforcement learning for game playing
+- [ ] **Web Automation**: Full browser control with understanding
+
+---
+
+## 📚 Full API
+
+### Main Methods
+
+```python
+# Execute a task
+result = agent.execute_task(task: str, max_steps: int = 50)
+
+# Analyze screen
+analysis = agent._analyze_screen()
+
+# Manual mode: Execute individual steps
+step = {"type": "launch_app", "app": "paint"}
+result = agent._execute_step(step)
+```
+
+### Result Structure
+
+```python
+{
+    "task": str,                    # Original task
+    "status": str,                  # "completed", "failed", "error"
+    "success": bool,                # Overall success
+    "steps": List[Dict],            # All steps executed
+    "screenshots": List[Dict],      # Before/after screenshots
+    "failed_at_step": int,          # If failed, which step
+    "error": str,                   # Error message if failed
+}
+```
+
+---
+
+**🦞 Built for OpenClaw - The future of desktop automation!**

config/openclaw/skills/desktop-control/QUICK_REFERENCE.md

@@ -0,0 +1,269 @@
+# Desktop Control - Quick Reference Card
+
+## 🚀 Instant Start
+
+```python
+from skills.desktop_control import DesktopController
+
+dc = DesktopController()
+```
+
+## 🖱️ Mouse Control (Top 10)
+
+```python
+# 1. Move mouse
+dc.move_mouse(500, 300, duration=0.5)
+
+# 2. Click
+dc.click(500, 300)  # Left click at position
+dc.click()           # Click at current position
+
+# 3. Right click
+dc.right_click(500, 300)
+
+# 4. Double click
+dc.double_click(500, 300)
+
+# 5. Drag & drop
+dc.drag(100, 100, 500, 500, duration=1.0)
+
+# 6. Scroll
+dc.scroll(-5)  # Scroll down 5 clicks
+
+# 7. Get position
+x, y = dc.get_mouse_position()
+
+# 8. Move relative
+dc.move_relative(100, 50)  # Move 100px right, 50px down
+
+# 9. Smooth movement
+dc.move_mouse(1000, 500, duration=1.0, smooth=True)
+
+# 10. Middle click
+dc.middle_click()
+```
+
+## ⌨️ Keyboard Control (Top 10)
+
+```python
+# 1. Type text (instant)
+dc.type_text("Hello World")
+
+# 2. Type text (human-like, 60 WPM)
+dc.type_text("Hello World", wpm=60)
+
+# 3. Press key
+dc.press('enter')
+dc.press('tab')
+dc.press('escape')
+
+# 4. Hotkeys (shortcuts)
+dc.hotkey('ctrl', 'c')      # Copy
+dc.hotkey('ctrl', 'v')      # Paste  
+dc.hotkey('ctrl', 's')      # Save
+dc.hotkey('win', 'r')       # Run dialog
+dc.hotkey('alt', 'tab')     # Switch window
+
+# 5. Hold & release
+dc.key_down('shift')
+dc.type_text("hello")  # Types "HELLO"
+dc.key_up('shift')
+
+# 6. Arrow keys
+dc.press('up')
+dc.press('down')
+dc.press('left')
+dc.press('right')
+
+# 7. Function keys
+dc.press('f5')  # Refresh
+
+# 8. Multiple presses
+dc.press('backspace', presses=5)
+
+# 9. Special keys
+dc.press('home')
+dc.press('end')
+dc.press('pagedown')
+dc.press('delete')
+
+# 10. Fast combo
+dc.hotkey('ctrl', 'alt', 'delete')
+```
+
+## 📸 Screen Operations (Top 5)
+
+```python
+# 1. Screenshot (full screen)
+img = dc.screenshot()
+dc.screenshot(filename="screen.png")
+
+# 2. Screenshot (region)
+img = dc.screenshot(region=(100, 100, 800, 600))
+
+# 3. Get pixel color
+r, g, b = dc.get_pixel_color(500, 300)
+
+# 4. Find image on screen
+location = dc.find_on_screen("button.png")
+
+# 5. Get screen size
+width, height = dc.get_screen_size()
+```
+
+## 🪟 Window Management (Top 5)
+
+```python
+# 1. Get all windows
+windows = dc.get_all_windows()
+
+# 2. Activate window
+dc.activate_window("Chrome")
+
+# 3. Get active window
+active = dc.get_active_window()
+
+# 4. List windows
+for title in dc.get_all_windows():
+    print(title)
+
+# 5. Switch to app
+dc.activate_window("Visual Studio Code")
+```
+
+## 📋 Clipboard (Top 2)
+
+```python
+# 1. Copy to clipboard
+dc.copy_to_clipboard("Hello!")
+
+# 2. Get from clipboard
+text = dc.get_from_clipboard()
+```
+
+## 🔥 Real-World Examples
+
+### Example 1: Auto-fill Form
+```python
+dc.click(300, 200)  # Name field
+dc.type_text("John Doe", wpm=80)
+dc.press('tab')
+dc.type_text("john@email.com", wpm=80)
+dc.press('tab')
+dc.type_text("Password123", wpm=60)
+dc.press('enter')
+```
+
+### Example 2: Copy-Paste Automation
+```python
+# Select all
+dc.hotkey('ctrl', 'a')
+# Copy
+dc.hotkey('ctrl', 'c')
+# Wait
+dc.pause(0.5)
+# Switch window
+dc.hotkey('alt', 'tab')
+# Paste
+dc.hotkey('ctrl', 'v')
+```
+
+### Example 3: File Operations
+```python
+# Select multiple files
+dc.key_down('ctrl')
+dc.click(100, 200)
+dc.click(100, 250)
+dc.click(100, 300)
+dc.key_up('ctrl')
+# Copy
+dc.hotkey('ctrl', 'c')
+```
+
+### Example 4: Screenshot Workflow
+```python
+# Take screenshot
+dc.screenshot(filename=f"capture_{time.time()}.png")
+# Open in Paint
+dc.hotkey('win', 'r')
+dc.pause(0.5)
+dc.type_text('mspaint')
+dc.press('enter')
+```
+
+### Example 5: Search & Replace
+```python
+# Open Find & Replace
+dc.hotkey('ctrl', 'h')
+dc.pause(0.3)
+# Type find text
+dc.type_text("old_text")
+dc.press('tab')
+# Type replace text
+dc.type_text("new_text")
+# Replace all
+dc.hotkey('alt', 'a')
+```
+
+## ⚙️ Configuration
+
+```python
+# With failsafe (move to corner to abort)
+dc = DesktopController(failsafe=True)
+
+# With approval mode (ask before each action)
+dc = DesktopController(require_approval=True)
+
+# Maximum speed (no safety checks)
+dc = DesktopController(failsafe=False)
+```
+
+## 🛡️ Safety
+
+```python
+# Check if safe to continue
+if dc.is_safe():
+    dc.click(500, 500)
+
+# Pause execution
+dc.pause(2.0)  # Wait 2 seconds
+
+# Emergency abort: Move mouse to any screen corner
+```
+
+## 🎯 Pro Tips
+
+1. **Instant typing**: `interval=0` or `wpm=None`
+2. **Human typing**: `wpm=60` (60 words/min)
+3. **Smooth mouse**: `duration=0.5, smooth=True`
+4. **Instant mouse**: `duration=0`
+5. **Wait for UI**: `dc.pause(0.5)` between actions
+6. **Failsafe**: Always enable for safety
+7. **Test first**: Use `demo.py` to test features
+8. **Coordinates**: Use `get_mouse_position()` to find them
+9. **Screenshots**: Capture before/after for verification
+10. **Hotkeys > Menus**: Faster and more reliable
+
+## 📦 Dependencies
+
+```bash
+pip install pyautogui pillow opencv-python pygetwindow pyperclip
+```
+
+## 🚨 Common Issues
+
+**Mouse not moving correctly?**
+- Check DPI scaling in Windows settings
+- Verify coordinates with `get_mouse_position()`
+
+**Keyboard not working?**
+- Ensure target app has focus
+- Some apps block automation (games, secure apps)
+
+**Failsafe triggering?**
+- Keep mouse away from screen corners
+- Disable if needed: `failsafe=False`
+
+---
+
+**Built for OpenClaw** 🦞 - Desktop automation made easy!

config/openclaw/skills/desktop-control/SKILL.md

@@ -0,0 +1,650 @@
+---
+description: Advanced desktop automation with mouse, keyboard, and screen control
+---
+
+# Desktop Control Skill
+
+**The most advanced desktop automation skill for OpenClaw.** Provides pixel-perfect mouse control, lightning-fast keyboard input, screen capture, window management, and clipboard operations.
+
+## 🎯 Features
+
+### Mouse Control
+- ✅ **Absolute positioning** - Move to exact coordinates
+- ✅ **Relative movement** - Move from current position
+- ✅ **Smooth movement** - Natural, human-like mouse paths
+- ✅ **Click types** - Left, right, middle, double, triple clicks
+- ✅ **Drag & drop** - Drag from point A to point B
+- ✅ **Scroll** - Vertical and horizontal scrolling
+- ✅ **Position tracking** - Get current mouse coordinates
+
+### Keyboard Control
+- ✅ **Text typing** - Fast, accurate text input
+- ✅ **Hotkeys** - Execute keyboard shortcuts (Ctrl+C, Win+R, etc.)
+- ✅ **Special keys** - Enter, Tab, Escape, Arrow keys, F-keys
+- ✅ **Key combinations** - Multi-key press combinations
+- ✅ **Hold & release** - Manual key state control
+- ✅ **Typing speed** - Configurable WPM (instant to human-like)
+
+### Screen Operations
+- ✅ **Screenshot** - Capture entire screen or regions
+- ✅ **Image recognition** - Find elements on screen (via OpenCV)
+- ✅ **Color detection** - Get pixel colors at coordinates
+- ✅ **Multi-monitor** - Support for multiple displays
+
+### Window Management
+- ✅ **Window list** - Get all open windows
+- ✅ **Activate window** - Bring window to front
+- ✅ **Window info** - Get position, size, title
+- ✅ **Minimize/Maximize** - Control window states
+
+### Safety Features
+- ✅ **Failsafe** - Move mouse to corner to abort
+- ✅ **Pause control** - Emergency stop mechanism
+- ✅ **Approval mode** - Require confirmation for actions
+- ✅ **Bounds checking** - Prevent out-of-screen operations
+- ✅ **Logging** - Track all automation actions
+
+---
+
+## 🚀 Quick Start
+
+### Installation
+
+First, install required dependencies:
+
+```bash
+pip install pyautogui pillow opencv-python pygetwindow
+```
+
+### Basic Usage
+
+```python
+from skills.desktop_control import DesktopController
+
+# Initialize controller
+dc = DesktopController(failsafe=True)
+
+# Mouse operations
+dc.move_mouse(500, 300)  # Move to coordinates
+dc.click()  # Left click at current position
+dc.click(100, 200, button="right")  # Right click at position
+
+# Keyboard operations
+dc.type_text("Hello from OpenClaw!")
+dc.hotkey("ctrl", "c")  # Copy
+dc.press("enter")
+
+# Screen operations
+screenshot = dc.screenshot()
+position = dc.get_mouse_position()
+```
+
+---
+
+## 📋 Complete API Reference
+
+### Mouse Functions
+
+#### `move_mouse(x, y, duration=0, smooth=True)`
+Move mouse to absolute screen coordinates.
+
+**Parameters:**
+- `x` (int): X coordinate (pixels from left)
+- `y` (int): Y coordinate (pixels from top)
+- `duration` (float): Movement time in seconds (0 = instant, 0.5 = smooth)
+- `smooth` (bool): Use bezier curve for natural movement
+
+**Example:**
+```python
+# Instant movement
+dc.move_mouse(1000, 500)
+
+# Smooth 1-second movement
+dc.move_mouse(1000, 500, duration=1.0)
+```
+
+#### `move_relative(x_offset, y_offset, duration=0)`
+Move mouse relative to current position.
+
+**Parameters:**
+- `x_offset` (int): Pixels to move horizontally (positive = right)
+- `y_offset` (int): Pixels to move vertically (positive = down)
+- `duration` (float): Movement time in seconds
+
+**Example:**
+```python
+# Move 100px right, 50px down
+dc.move_relative(100, 50, duration=0.3)
+```
+
+#### `click(x=None, y=None, button='left', clicks=1, interval=0.1)`
+Perform mouse click.
+
+**Parameters:**
+- `x, y` (int, optional): Coordinates to click (None = current position)
+- `button` (str): 'left', 'right', 'middle'
+- `clicks` (int): Number of clicks (1 = single, 2 = double)
+- `interval` (float): Delay between multiple clicks
+
+**Example:**
+```python
+# Simple left click
+dc.click()
+
+# Double-click at specific position
+dc.click(500, 300, clicks=2)
+
+# Right-click
+dc.click(button='right')
+```
+
+#### `drag(start_x, start_y, end_x, end_y, duration=0.5, button='left')`
+Drag and drop operation.
+
+**Parameters:**
+- `start_x, start_y` (int): Starting coordinates
+- `end_x, end_y` (int): Ending coordinates
+- `duration` (float): Drag duration
+- `button` (str): Mouse button to use
+
+**Example:**
+```python
+# Drag file from desktop to folder
+dc.drag(100, 100, 500, 500, duration=1.0)
+```
+
+#### `scroll(clicks, direction='vertical', x=None, y=None)`
+Scroll mouse wheel.
+
+**Parameters:**
+- `clicks` (int): Scroll amount (positive = up/left, negative = down/right)
+- `direction` (str): 'vertical' or 'horizontal'
+- `x, y` (int, optional): Position to scroll at
+
+**Example:**
+```python
+# Scroll down 5 clicks
+dc.scroll(-5)
+
+# Scroll up 10 clicks
+dc.scroll(10)
+
+# Horizontal scroll
+dc.scroll(5, direction='horizontal')
+```
+
+#### `get_mouse_position()`
+Get current mouse coordinates.
+
+**Returns:** `(x, y)` tuple
+
+**Example:**
+```python
+x, y = dc.get_mouse_position()
+print(f"Mouse is at: {x}, {y}")
+```
+
+---
+
+### Keyboard Functions
+
+#### `type_text(text, interval=0, wpm=None)`
+Type text with configurable speed.
+
+**Parameters:**
+- `text` (str): Text to type
+- `interval` (float): Delay between keystrokes (0 = instant)
+- `wpm` (int, optional): Words per minute (overrides interval)
+
+**Example:**
+```python
+# Instant typing
+dc.type_text("Hello World")
+
+# Human-like typing at 60 WPM
+dc.type_text("Hello World", wpm=60)
+
+# Slow typing with 0.1s between keys
+dc.type_text("Hello World", interval=0.1)
+```
+
+#### `press(key, presses=1, interval=0.1)`
+Press and release a key.
+
+**Parameters:**
+- `key` (str): Key name (see Key Names section)
+- `presses` (int): Number of times to press
+- `interval` (float): Delay between presses
+
+**Example:**
+```python
+# Press Enter
+dc.press('enter')
+
+# Press Space 3 times
+dc.press('space', presses=3)
+
+# Press Down arrow
+dc.press('down')
+```
+
+#### `hotkey(*keys, interval=0.05)`
+Execute keyboard shortcut.
+
+**Parameters:**
+- `*keys` (str): Keys to press together
+- `interval` (float): Delay between key presses
+
+**Example:**
+```python
+# Copy (Ctrl+C)
+dc.hotkey('ctrl', 'c')
+
+# Paste (Ctrl+V)
+dc.hotkey('ctrl', 'v')
+
+# Open Run dialog (Win+R)
+dc.hotkey('win', 'r')
+
+# Save (Ctrl+S)
+dc.hotkey('ctrl', 's')
+
+# Select All (Ctrl+A)
+dc.hotkey('ctrl', 'a')
+```
+
+#### `key_down(key)` / `key_up(key)`
+Manually control key state.
+
+**Example:**
+```python
+# Hold Shift
+dc.key_down('shift')
+dc.type_text("hello")  # Types "HELLO"
+dc.key_up('shift')
+
+# Hold Ctrl and click (for multi-select)
+dc.key_down('ctrl')
+dc.click(100, 100)
+dc.click(200, 100)
+dc.key_up('ctrl')
+```
+
+---
+
+### Screen Functions
+
+#### `screenshot(region=None, filename=None)`
+Capture screen or region.
+
+**Parameters:**
+- `region` (tuple, optional): (left, top, width, height) for partial capture
+- `filename` (str, optional): Path to save image
+
+**Returns:** PIL Image object
+
+**Example:**
+```python
+# Full screen
+img = dc.screenshot()
+
+# Save to file
+dc.screenshot(filename="screenshot.png")
+
+# Capture specific region
+img = dc.screenshot(region=(100, 100, 500, 300))
+```
+
+#### `get_pixel_color(x, y)`
+Get color of pixel at coordinates.
+
+**Returns:** RGB tuple `(r, g, b)`
+
+**Example:**
+```python
+r, g, b = dc.get_pixel_color(500, 300)
+print(f"Color at (500, 300): RGB({r}, {g}, {b})")
+```
+
+#### `find_on_screen(image_path, confidence=0.8)`
+Find image on screen (requires OpenCV).
+
+**Parameters:**
+- `image_path` (str): Path to template image
+- `confidence` (float): Match threshold (0-1)
+
+**Returns:** `(x, y, width, height)` or None
+
+**Example:**
+```python
+# Find button on screen
+location = dc.find_on_screen("button.png")
+if location:
+    x, y, w, h = location
+    # Click center of found image
+    dc.click(x + w//2, y + h//2)
+```
+
+#### `get_screen_size()`
+Get screen resolution.
+
+**Returns:** `(width, height)` tuple
+
+**Example:**
+```python
+width, height = dc.get_screen_size()
+print(f"Screen: {width}x{height}")
+```
+
+---
+
+### Window Functions
+
+#### `get_all_windows()`
+List all open windows.
+
+**Returns:** List of window titles
+
+**Example:**
+```python
+windows = dc.get_all_windows()
+for title in windows:
+    print(f"Window: {title}")
+```
+
+#### `activate_window(title_substring)`
+Bring window to front by title.
+
+**Parameters:**
+- `title_substring` (str): Part of window title to match
+
+**Example:**
+```python
+# Activate Chrome
+dc.activate_window("Chrome")
+
+# Activate VS Code
+dc.activate_window("Visual Studio Code")
+```
+
+#### `get_active_window()`
+Get currently focused window.
+
+**Returns:** Window title (str)
+
+**Example:**
+```python
+active = dc.get_active_window()
+print(f"Active window: {active}")
+```
+
+---
+
+### Clipboard Functions
+
+#### `copy_to_clipboard(text)`
+Copy text to clipboard.
+
+**Example:**
+```python
+dc.copy_to_clipboard("Hello from OpenClaw!")
+```
+
+#### `get_from_clipboard()`
+Get text from clipboard.
+
+**Returns:** str
+
+**Example:**
+```python
+text = dc.get_from_clipboard()
+print(f"Clipboard: {text}")
+```
+
+---
+
+## ⌨️ Key Names Reference
+
+### Alphabet Keys
+`'a'` through `'z'`
+
+### Number Keys
+`'0'` through `'9'`
+
+### Function Keys
+`'f1'` through `'f24'`
+
+### Special Keys
+- `'enter'` / `'return'`
+- `'esc'` / `'escape'`
+- `'space'` / `'spacebar'`
+- `'tab'`
+- `'backspace'`
+- `'delete'` / `'del'`
+- `'insert'`
+- `'home'`
+- `'end'`
+- `'pageup'` / `'pgup'`
+- `'pagedown'` / `'pgdn'`
+
+### Arrow Keys
+- `'up'` / `'down'` / `'left'` / `'right'`
+
+### Modifier Keys
+- `'ctrl'` / `'control'`
+- `'shift'`
+- `'alt'`
+- `'win'` / `'winleft'` / `'winright'`
+- `'cmd'` / `'command'` (Mac)
+
+### Lock Keys
+- `'capslock'`
+- `'numlock'`
+- `'scrolllock'`
+
+### Punctuation
+- `'.'` / `','` / `'?'` / `'!'` / `';'` / `':'`
+- `'['` / `']'` / `'{'` / `'}'`
+- `'('` / `')'`
+- `'+'` / `'-'` / `'*'` / `'/'` / `'='`
+
+---
+
+## 🛡️ Safety Features
+
+### Failsafe Mode
+
+Move mouse to **any corner** of the screen to abort all automation.
+
+```python
+# Enable failsafe (enabled by default)
+dc = DesktopController(failsafe=True)
+```
+
+### Pause Control
+
+```python
+# Pause all automation for 2 seconds
+dc.pause(2.0)
+
+# Check if automation is safe to proceed
+if dc.is_safe():
+    dc.click(500, 500)
+```
+
+### Approval Mode
+
+Require user confirmation before actions:
+
+```python
+dc = DesktopController(require_approval=True)
+
+# This will ask for confirmation
+dc.click(500, 500)  # Prompt: "Allow click at (500, 500)? [y/n]"
+```
+
+---
+
+## 🎨 Advanced Examples
+
+### Example 1: Automated Form Filling
+
+```python
+dc = DesktopController()
+
+# Click name field
+dc.click(300, 200)
+dc.type_text("John Doe", wpm=80)
+
+# Tab to next field
+dc.press('tab')
+dc.type_text("john@example.com", wpm=80)
+
+# Tab to password
+dc.press('tab')
+dc.type_text("SecurePassword123", wpm=60)
+
+# Submit form
+dc.press('enter')
+```
+
+### Example 2: Screenshot Region and Save
+
+```python
+# Capture specific area
+region = (100, 100, 800, 600)  # left, top, width, height
+img = dc.screenshot(region=region)
+
+# Save with timestamp
+import datetime
+timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
+img.save(f"capture_{timestamp}.png")
+```
+
+### Example 3: Multi-File Selection
+
+```python
+# Hold Ctrl and click multiple files
+dc.key_down('ctrl')
+dc.click(100, 200)  # First file
+dc.click(100, 250)  # Second file
+dc.click(100, 300)  # Third file
+dc.key_up('ctrl')
+
+# Copy selected files
+dc.hotkey('ctrl', 'c')
+```
+
+### Example 4: Window Automation
+
+```python
+# Activate Calculator
+dc.activate_window("Calculator")
+time.sleep(0.5)
+
+# Type calculation
+dc.type_text("5+3=", interval=0.2)
+time.sleep(0.5)
+
+# Take screenshot of result
+dc.screenshot(filename="calculation_result.png")
+```
+
+### Example 5: Drag & Drop File
+
+```python
+# Drag file from source to destination
+dc.drag(
+    start_x=200, start_y=300,  # File location
+    end_x=800, end_y=500,       # Folder location
+    duration=1.0                 # Smooth 1-second drag
+)
+```
+
+---
+
+## ⚡ Performance Tips
+
+1. **Use instant movements** for speed: `duration=0`
+2. **Batch operations** instead of individual calls
+3. **Cache screen positions** instead of recalculating
+4. **Disable failsafe** for maximum performance (use with caution)
+5. **Use hotkeys** instead of menu navigation
+
+---
+
+## ⚠️ Important Notes
+
+- **Screen coordinates** start at (0, 0) in top-left corner
+- **Multi-monitor setups** may have negative coordinates for secondary displays
+- **Windows DPI scaling** may affect coordinate accuracy
+- **Failsafe corners** are: (0,0), (width-1, 0), (0, height-1), (width-1, height-1)
+- **Some applications** may block simulated input (games, secure apps)
+
+---
+
+## 🔧 Troubleshooting
+
+### 🔒 锁屏检测与处理 (重要!)
+
+**在使用真实桌面操作前，必须先检测锁屏状态:**
+
+```python
+import subprocess
+
+def is_screen_locked():
+    """检测屏幕是否锁屏"""
+    result = subprocess.run(
+        ["loginctl", "show-session", "-p", "LockedHint", "14"],
+        capture_output=True, text=True
+    )
+    return "yes" in result.stdout.lower()
+```
+
+**检测流程:**
+1. 先截屏查看画面状态
+2. 如果显示锁屏界面 → 输入密码解锁
+3. 如果显示登录页面 → 输入账号密码
+4. 如果是正常桌面 → 可以开始操作
+
+**重要提示:**
+- 长时间不操作会导致自动锁屏
+- 操作前先检查锁屏状态
+- Ubuntu 锁屏密码: a12345678
+
+### Mouse not moving to correct position
+- Check DPI scaling settings
+- Verify screen resolution matches expectations
+- Use `get_screen_size()` to confirm dimensions
+
+### Keyboard input not working
+- Ensure target application has focus
+- Some apps require admin privileges
+- Try increasing `interval` for reliability
+
+### Failsafe triggering accidentally
+- Increase screen border tolerance
+- Move mouse away from corners during normal use
+- Disable if needed: `DesktopController(failsafe=False)`
+
+### Permission errors
+- Run Python with administrator privileges for some operations
+- Some secure applications block automation
+
+---
+
+## 📦 Dependencies
+
+- **PyAutoGUI** - Core automation engine
+- **Pillow** - Image processing
+- **OpenCV** (optional) - Image recognition
+- **PyGetWindow** - Window management
+
+Install all:
+```bash
+pip install pyautogui pillow opencv-python pygetwindow
+```
+
+---
+
+**Built for OpenClaw** - The ultimate desktop automation companion 🦞

config/openclaw/skills/desktop-control/_meta.json

@@ -0,0 +1,11 @@
+{
+  "owner": "matagul",
+  "slug": "desktop-control",
+  "displayName": "Desktop Control",
+  "latest": {
+    "version": "1.0.0",
+    "publishedAt": 1770255200863,
+    "commit": "https://github.com/clawdbot/skills/commit/66090236946d6a2ab565cf9d425ae7102f881b0a"
+  },
+  "history": []
+}