VsCode SimpleBrowser extension to automate seemless copilot ui testing and QA

[kwkau]

💡 [DISCUSSION] Proposal: Enable Screenshot Capture & DOM Interaction in the Simple Browser

Hey everyone 👋 — I'm working on an AI-powered VS Code extension (SIBA AI Simple Browser Extensions) that needs deeper interaction with the built-in Simple Browser. Right now, we’re hitting some limitations that make it hard to create truly seamless automation and AI workflows inside VS Code.

🔍 The Problem

We want our extension to:

• Take high-quality screenshots of what's shown in the Simple Browser
• Read and interact with DOM content (like clicking, typing, scrolling)
• Do all of this from within VS Code, not using external tools like Playwright

Currently, there’s no official API to access or control the Simple Browser, so we’ve been forced to use workarounds like:

• Injecting html2canvas (which breaks on cross-origin or CSP-restricted pages)
• Proposed APIs like editorInsets (which aren’t marketplace-friendly)
• External tools like Playwright (which are slow and not truly integrated)

📦 What We’re Proposing

We’d love to see some first-class API support for the Simple Browser that enables:

1. vscode.simpleBrowser.getActivePanel()

Get a handle to the active Simple Browser instance.

const browser = vscode.simpleBrowser.getActivePanel();

2. browser.captureScreenshot()

Capture a pixel-perfect image of what the user sees.

const img = await browser.captureScreenshot({ format: 'png' });

Internally this could call Electron’s webContents.capturePage().

3. browser.evaluate<T>(script: string): Thenable<T>

Run scripts inside the page context, safely.

const heading = await browser.evaluate(`document.querySelector("h1")?.textContent`);

4. browser.simulateEvent(selector: string, event: string, payload?: any)

Simulate clicks, typing, scrolling, etc.

await browser.simulateEvent("#submit", "click");
await browser.simulateEvent("#name", "input", { value: "Kwaku" });

🧠 Use Cases

• AI Assistants that interact with web apps directly inside VS Code
• Docs automation: Capture screenshots and extract content on the fly
• Accessibility audits: DOM traversal from inside the IDE
• QA workflows: Lightweight end-to-end flows without external browsers

🛡️ Security Thoughts

All of this would only apply to:

• Pages rendered inside Simple Browser tabs
• Scripts that run in the page's sandbox (no Node access)
• Pages the user explicitly navigates to

Optionally, a vscode.experimental.enableSimpleBrowserInterop setting could gate these APIs for early adopters.

🔄 Alternatives Considered

Option	Why It Doesn’t Work Well
editorInsets + JS injection	Fragile, not stable for marketplace
Playwright / Puppeteer	External, slow, no access to Simple Browser
Screen capture tools	OS-specific, not ideal in cross-platform editor

📣 Why This Matters

With more AI tooling, test automation, and web-based workflows shifting into the IDE, the Simple Browser has the potential to be more than just a viewer. It could become an automation-ready interface — where intelligent agents, scripts, and extensions can interact with live web content in a secure, developer-friendly way.

---

I’d love your thoughts, feedback, or alternatives.

Would you use something like this in your extensions or workflows?

Let’s discuss 💬

[Windsage63]

What I would like to see here is a Chrome or Firefox style developer tool window that would give the agents access to the browser console from within vs-code. Whenever I have issues in the front-end currently I either troubleshooting myself with the console or copy and paste the console info to the chat. If simple-browser had a console the agents could fix most UI issues during automated testing. To me this seems like something that will be required to close the loop on vibe development.