Runtime API

To begin using the runtime API, install

The NPM package primarily provides you with types, the actual API is injected at runtime by the Highlight app. Thus, these methods will throw errors if your web app attempts to access them outside of Highlight.

​

How it works

Your Highlight app runs in its own isolated web view. Before your app loads, we inject a set of APIs into the web view to allow your app to interact with the user’s computer. This is similar to how a regular web browser exposes APIs. However, Highlight APIs are designed to bridge the gap, allowing you to build AI apps without needing to start from scratch with MacOS/Windows exclusive APIs. Instead, with Highlight, you get a uniform API that works across all major operating systems.

Because your app is still a typical web app, you can still use all the standard web APIs you’re used to. This includes things like fetch, localStorage, and more.

​

Types

​

HighlightContext

The HighlightContext type allows your application to access all the context from the user’s computer to help assist them with their request. This object is created when the user activates the Highlight hotkey or clicks the Highlight icon.

See the type definition here.

​

Inference

Run local inference through Highlight without needing your own backend.

​

Get Text Prediction

Generate text predictions using an LLM model based on provided messages.

​

Model Information

The current LLM inference model is Llama 3.1, featuring:

• 405 billion parameters
• Context size of up to 64,000 tokens

​

Usage

import Highlight from "@highlight-ai/app-runtime";

const textPrediction = Highlight.inference.getTextPrediction(messages, options);

for await (const chunk of textPrediction) {
  console.log("Incoming chunk:", chunk);
}

Parameters

1. messages: An array of message objects, each containing:

   • role: string (“system”, “user”, or “assistant”)
   • content: string

2. options: (Optional) Configuration object

   {
     temperature?: number;
     topP?: number;
     frequencyPenalty?: number;
     presencePenalty?: number;
     n?: number;
     logitBias?: Record<string, number>;
     stop?: string[];
     stopSequences?: string[];
     maxTokens?: number;
     logprobs?: boolean;
     topLogprobs?: number;
   }
   

​

Example

const messages = [
  {
    role: 'system',
    content: 'You are a helpful assistant that can answer questions and help with tasks.',
  },
  {
    role: 'user',
    content: 'What is the capital of France?',
  }
];

const options = {
  temperature: 0.1,
};

const textPrediction = Highlight.inference.getTextPrediction(messages, options);

for await (const chunk of textPrediction) {
  console.log("Incoming chunk:", chunk);
}

This function returns an async iterable, allowing you to process the prediction in chunks as they arrive.

​

User

APIs that involve user data.

​

Get context

Asynchronously fetches the context of the user’s computer. Similar to the behavior of the onContext event listener. However, you may poll this method to get repeated updates.

import Highlight from "@highlight-ai/app-runtime";

const context = await Highlight.user.getContext();

// Console log the title of the focused window (ignoring Highlight).
console.log(context.application.focusedWindow.title);

​

Get facts

Returns a list of facts about the user. These facts are set by the user during Highlight onboarding and can later be adjusted in Highlight settings.

import Highlight from "@highlight-ai/app-runtime";

const facts = await Highlight.user.getFacts();

​

Get audio

Returns the latest buffer of audio data from the user’s computer.

import Highlight from "@highlight-ai/app-runtime";

const audio = await Highlight.user.getAudio();

​

Get audio by duration

Returns the audio transcript for the previous duration seconds of time

import Highlight from "@highlight-ai/app-runtime";

// Get audio transcript for the last 2 minutes (120 seconds)
const audioTranscript = await Highlight.user.getAudioForDuration(120);

​

Get email

Returns a privacy-protected email address of the user (for example: abcdefghij@highlight.email). This email address forwards to the user’s main email address.

You can use this for implementing auth on top of an existing system you already have.

import Highlight from "@highlight-ai/app-runtime";

const email = await Highlight.user.getEmail();

​

Get screenshot

Returns a Base64 encoded string with the image data from the user’s screen.

import Highlight from "@highlight-ai/app-runtime";

const screenshot = await Highlight.user.getScreenshot();

​

Get display screenshots

Returns a list of desktop image objects, one corresponding to each of the user’s displays. Each object includes a Base64 encoded string with the image data for that display. Your app must be granted screenshot permissions by calling requestScreenshotPermission before you can use this function.

import Highlight from "@highlight-ai/app-runtime";

const windows = await Highlight.user.getDisplayScreenshots();

​

Get windows

Returns a list of the user’s open windows as well as an app icon for that window if available.

import Highlight from "@highlight-ai/app-runtime";

const windows = await Highlight.user.getWindows();

​

Get window screenshot

Returns a Base64 encoded string of an image of the window with the provided title. Window titles can be accessed by calling getWindows. Your app must be granted screenshot permissions by calling requestScreenshotPermission before you can use this function.

import Highlight from "@highlight-ai/app-runtime";

const windows = await Highlight.user.getWindows();
const windowScreenshot = await Highlight.user.getWindowScreenshot(windows[0].windowTitle);

​

App Storage

App storage is a secure, isolated place to keep your app’s data safe in Highlight.

We plan on later supporting secure cloud sync (with user opt-in required). Along with other upcoming features (like a unified way to share documents/attachments with other apps), we recommend using app storage over localStorage.

​

Loading Data

When your app first loads in Highlight, it will asynchronously request data stored on disk via appStorage. This typically happens within milliseconds, but if you want to ensure the data is ready, you can use:

Highlight.appStorage.isHydrated();
// or
await Highlight.appStorage.whenHydrated();

​

Get Data

To access data, use the get method:

const data = Highlight.appStorage.get("key");

​

Set Data

To set data, use the set method:

Highlight.appStorage.set("key", "value");

​

Delete Data

To delete data, use the delete method:

Highlight.appStorage.delete("key");

​

Auth

​

Sign the user in with Highlight

Signs the user into your app automatically. Returns an access token (JWT) and a refresh token that can be used to validate the session on your backend and keep the user signed in. This access token is also used to make requests to the Highlight web API (coming soon).

The JWT can be decoded to fetch the sub or subscriber ID (a unique ID that remains consistent across all Highlight apps).

For more information on using Highlight auth, read the authentication article.

​

Permissions

Request permissions for your Highlight app to grant it additional capabilities.

​

Request background permission

Requests permission for your Highlight app to run in the background. After requesting permission, you will need to call setBackgroundStatus to actually set the background status.

import Highlight from "@highlight-ai/app-runtime";

const granted = await Highlight.permissions.requestBackgroundPermission();

// If the user grants the permission, the promise resolves to true.
// If the user declines the permission, the promise resolves to false.
console.log(granted);

​

Request screenshot permission

Requests permission for your Highlight app to capture screenshots of the user’s displays and windows. This permission must be granted in order to receive responses from getDisplayScreenshots and getWindowScreenshot.

import Highlight from "@highlight-ai/app-runtime";

const granted = await Highlight.permissions.requestScreenshotPermission();

// If the user grants the permission, the promise resolves to true.
// If the user declines the permission, the promise resolves to false.
console.log(granted);

​

Request clipboard read permission

Requests permission for your Highlight app to read the contents of the user’s clipboard. This permission must be granted in order to receive a response from getClipboardContents.

import Highlight from "@highlight-ai/app-runtime";

const granted = await Highlight.permissions.requestClipboardReadPermission();

// If the user grants the permission, the promise resolves to true.
// If the user declines the permission, the promise resolves to false.
console.log(granted);

​

App

Methods that help your app feel native to Highlight. This category deals with app to Highlight communication.

​

Determine if your app is running inside of Highlight

You can check to see if your app is running inside of Highlight by using the isRunningInHighlight method.

import Highlight from "@highlight-ai/app-runtime";

const isRunningInHighlight = Highlight.isRunningInHighlight();

​

Add event listener

Registers an event listener for your app. When this event occurs, the callback function will be called. This function returns a destroy function that you can call to remove the listener when you are done with it.

import Highlight from "@highlight-ai/app-runtime";

const destroyListener = Highlight.app.addListener("onContext", (context) => {
  console.log("onContext", context);
});

// Call this when you want to remove the listener
destroyListener();

​

Desktop Shortcuts

You can check if the user has created a desktop shortcut for you app, and if not you can prompt them to create one.

import Highlight from "@highlight-ai/app-runtime";

const hasDesktopShortcut = await Highlight.app.hasDesktopShortcut()
if (!hasDesktopShortcut) {
  await Highlight.app.requestCreateDesktopShortcut()
}

​

Get hotkey

Get the hotkey the user should use to bring up the Highlight overlay window. This is helpful when making an onboard presentation for your users.

import Highlight from "@highlight-ai/app-runtime";

const hotkey = await Highlight.app.getHotkey();
// hotKey is a string like "CMD + ."

​

Open App

Opens a different app by its ID in Highlight.

import Highlight from "@highlight-ai/app-runtime";

await Highlight.app.openApp("hlchat");

​

Set background status

Tells Highlight whether your app should run in the background. You will need to request background permission first.

If set to true, your app will start when Highlight starts and continue running until Highlight is closed.

Calling setBackgroundStatus(true) will throw an error if you don’t have permission to run in the background.

import Highlight from "@highlight-ai/app-runtime";

await Highlight.app.setBackgroundStatus(true);