API to determine whether integrated terminal has focus

[pokey]

As part of implementing improved voice coding support, we are developing a VSCode extension that enables improved communication with talon. One thing we need is to be able to determine that the terminal pane currently has focus, so that we can activate commands for interacting with the terminal. To do so, we'd basically need two things:

• A function to check whether the terminal has focus
• An event that is fired when the terminal gains / loses focus

[pokey]

Fwiw I think half of this issue could be solved with #118722 by using the terminalFocus context

[Tyriar]

Right now our API doesn't really have the concept of focus I think by design. We'll see how #118722 turns out but I think the right thing to do here is to hook into the operating system's accessibility APIs and check what is being focused in the accessibility tree.

[pokey]

@TylerLeonhardt thanks for your response. Not sure I understand, though. How come your design encourages exposing focus as a when-clause context, but not as an API? Thanks for your help

[Tyriar]

This is what I was told:

the API doesn’t really have the concept of focus, because focus if often in elements that have no API representation

[pokey]

Ok, well if your source has a response to my question, I'd be interested to understand the answer, but I guess hopefully #118722 will get merged and at least solve half the issue here

[daanzu]

@Tyriar Having this would be tremendously useful for accessibility. It is quite disappointing to hear that there is push back to implementing it.

[daanzu]

@pokey I hope such an extension could be made general enough to be used by voice software other than talon. I have been wanting to work on something similar for my own setup using dragonfly/kaldi-active-grammar. Specifically support for: executing VSCode commands, retrieving VSCode context, and retrieving VSCode's buffer text around the cursor for priming the dictation language model. Is your work being done on github?

[pokey]

Hi @daanzu glad to hear the interest! None of my extensions are talon-specific. I have a few extensions that I've developed for voice coding, but you'll probably be most interested in https://marketplace.visualstudio.com/items?itemName=pokey.command-server. Lmk if you have any troubles getting it set up / linked with kaldi and I'd be happy to hop on a zoom and pair up!

[Tyriar]

Why can't you use the OS' a11y APIs to do this? I would have expected that's possible as it's how screen readers get all their information.

[daanzu]

@Tyriar It should be possible, but the OS accessibility API tends to be quite complicated, slow, and prone to breaking. If I am understanding it correctly, the VSCode API already has the concept for key bindings, which has the benefit of being easily discoverable for users. It seems silly to then make it unavailable as one simple API query.

@pokey Thanks for the info! I'll take a look.

[lunixbochs]

Why can't you use the OS' a11y APIs to do this? I would have expected that's possible as it's how screen readers get all their information.

On Mac, VSCode appears to the accessibility subsystem by default as an opaque rectangle. In Electron, you can manually set a flag on the window to enable enhanced accessibility, but that doesn't work for everything, and when it does work it sort of exports the whole DOM which typically comes with a huge performance and resource hit (which is why it's disabled by default in Chromium and Electron - they aren't exactly light on resources to begin).

The simplest way to surface context information (e.g. which part of the app is active) to voice control systems is to put a simple string in the window title (e.g. "terminal" "file picker" "command bar"). The most robust way is to have some kind of plugin api for querying context, and events when context changes.