https://github.com/csabakecskemeti/ai-interact

aihub project intends to change the way how do we interact with LLMs. Today many tools offer integration to all different models, and various chat applications are available online and locally. This provides a very scattered picture for end users and applications without AI integration require extra effort to get help with. aihub offers a more natural way to interface with generative AI models that are app agnostic, by sharing a screen portion with the model where the user seeks help.

How it works

A small Python application with a minimal GUI runs in the background. The application is API-integrated with an LLM of your choice (our recommendation for local inference is LMStudio) and running a keyboard listener. With the [SHIFT][F1] keyboard shortcut the user initiates the capture mode. By defining an imaginary rectangle with 2 mouse clicks (define 2 diagonal corners of the rectangle) the code captures an image from anywhere on the screen. Then these images are processed by a locally running text extraction model: Tesseract, and the result text will be sent to the LLM with the preconfigured prefix. We’ve found that LLMs can handle the not perfect text extraction of Tesseract.

Examples

Help with code

Summary