How-To

How To Wire A Web App

Mount the relay in Astro, Express, or any Web-standard runtime, and consume the stream from the browser.

This guide shows how to expose a relay from server routes and consume it with the reconnecting client.

Build A Shared Relay

Create one relay instance for the process:

The provider factories come from @inbrowser/model; the relay only routes to them. Each cloud factory already matches ModelClientFactory (its config is { apiKey?, model }), so it registers directly:

import { createRelay, type ModelEvent } from '@inbrowser/relay';
import { geminiModelClient } from '@inbrowser/model/providers/gemini';
import { openrouterModelClient } from '@inbrowser/model/providers/openrouter';
import {
  createRtdbJobStore,
  serviceAccountTokenProvider,
} from '@inbrowser/resumable';

export const relay = createRelay({
  store: createRtdbJobStore<ModelEvent>({
    url: process.env.RTDB_URL!,
    auth: serviceAccountTokenProvider({
      keyFile: process.env.SERVICE_ACCOUNT_FILE!,
    }),
    rootPath: 'inference_jobs',
    defaultTtlMs: 7 * 24 * 60 * 60 * 1000,
  }),
  providers: {
    gemini: geminiModelClient,
    openrouter: openrouterModelClient,
  },
  sweep: {
    intervalMs: 60 * 60 * 1000,
  },
});

If a provider needs construction options beyond { apiKey, model } — for example pinning a baseUrl or a temperature, or passing a Claude subscription token — wrap the factory so the relay-supplied { apiKey, model } is merged with your extras:

import { ollamaModelClient } from '@inbrowser/model/providers/ollama';

providers: {
  // Pin Ollama's base URL instead of carrying it on the wire via apiKey.
  ollama: (c) => ollamaModelClient({ ...c, baseUrl: 'http://localhost:11434' }),
}

Use a durable store in production. The memory store is useful for tests and local demos, but it cannot survive process restart.

Wire Web-Standard Runtimes Directly

For runtimes that already use Web Request and Response, call the relay handlers directly:

export async function POST(request: Request): Promise<Response> {
  return relay.handleStart(request);
}

export async function GET(
  request: Request,
  params: { id: string },
): Promise<Response> {
  return relay.handleStream(request, { jobId: params.id });
}

The route shape is up to the host. The client helpers assume this common shape:

POST /api/inference/job
GET /api/inference/job/:id/stream?from=N

Wire Astro

import { createAstroRoutes } from '@inbrowser/relay/adapters/astro';
import { relay } from './relay';

export const { start, stream } = createAstroRoutes(relay);

Then re-export the routes:

// src/pages/api/inference/job.ts
export { start as POST } from '~/server/relay';

// src/pages/api/inference/job/[id]/stream.ts
export { stream as GET } from '~/server/relay';

Wire Express Or Cloud Functions

import express from 'express';
import { createExpressHandlers } from '@inbrowser/relay/adapters/express';
import { relay } from './relay';

const app = express();
const { start, stream } = createExpressHandlers(relay, {
  cors: true,
});

app.use(express.json());
app.post('/api/inference/job', start);
app.get('/api/inference/job/:id/stream', stream);

Use cors: true when the browser calls a different origin directly, such as a raw Cloud Run or Cloud Functions URL.

Consume From The Browser

import {
  createResumableClient,
  installBrowserLifecycle,
} from '@inbrowser/relay/client';

const client = createResumableClient({
  startUrl: '/api/inference/job',
  streamUrl: (jobId, from) =>
    `/api/inference/job/${encodeURIComponent(jobId)}/stream?from=${from}`,
  installLifecycle: installBrowserLifecycle(),
  onReconnect: (info) => {
    console.debug('relay reconnect', info);
  },
});

for await (const event of client.stream({
  provider: 'gemini',
  model: 'gemini-3-flash-preview',
  messages: [{ role: 'user', text: 'Plan my next step' }],
  tools: [],
  apiKey: userApiKey,
})) {
  if (event.kind === 'text') {
    appendText(event.text);
  }
}

installBrowserLifecycle() is optional but useful in browsers. When a tab returns to the foreground, it aborts the current connection so the client reconnects immediately instead of waiting for a stale socket to time out.

Handle Stream Buffering

SSE must reach the browser as a live stream. If a proxy buffers the response, the relay can still run the job, but the user will not see incremental output.

For Firebase Hosting plus Cloud Run or Cloud Functions, prefer calling the raw function URL from the browser and enabling CORS on the Express adapter. Hosting rewrites may buffer SSE end to end.

Shut Down Cleanly

Call relay.stop() during graceful shutdown:

process.on('SIGTERM', () => {
  void relay.stop().finally(() => process.exit(0));
});

This stops scheduled sweeps and waits for in-flight producers to settle.