Voice Realtime

Streaming voice assistant pre-bound to a user's calendar. WebSocket protocol modeled on the OpenAI Realtime API.

Endpoints

Connect

Open a WebSocket to:

wss://api.chronapilot.com/v1/voice/realtime?session=<jwt>

The session JWT is scoped to one user. It carries the user ID, the granted scopes, and a short TTL (5 min by default, max 30 min). If you call from a Connect platform key, the JWT additionally carries the cpa_… connected-account ID and inherits that account's restrictions.

Mint the JWT server-side from your sk_live_… key — never embed your secret key in a client:

POST /v1/voice/sessions

{
  "user_id": "usr_1abc2def3ghi",
  "scopes": ["events:read","events:write","alerts:read"],
  "ttl_seconds": 300
}

Response:

{
  "id": "hub_9stu0vwx1yza",
  "object": "voice_session",
  "session_token": "eyJhbGciOi…",
  "expires_at": "2026-05-22T13:05:00-07:00"
}

Message protocol

Modeled on Azure OpenAI Realtime, with ChronaPilot extensions.

Client → server

• session.update — set model, voice, tools, output format.
• audio.append — append PCM16 audio chunk (base64).
• text.append — append a text user turn.
• response.cancel — abort current response (barge-in).

Server → client

• transcript.partial — partial user transcript (low-latency).
• transcript.final — final user transcript.
• intent.detected — { name, confidence, parameters }.
• action.proposed — AI wants to call a tool; client surface for user confirmation.
• action.executed — tool ran; payload includes the tool result and the affected resource ID.
• audio.delta — TTS audio chunk to play (base64 PCM16).
• response.done — turn complete.

Available tools

The model has 26 pre-bound tools — calendar reads, mutations, travel queries, alert management. The full schema mirrors the production tool catalog (CalendarToolDefinitions.cs):

• create_event, update_event, cancel_event, bulk_cancel_events
• get_day_overview, get_day_summary, get_weekly_summary
• get_next_event, search_events
• get_departure_info, get_travel_time, get_alternate_routes, start_navigation
• check_if_late, acknowledge_late, report_arrival
• get_alerts, dismiss_alert, snooze_alert
• create_reminder
• notify_attendees
• update_preferences, set_quiet_hours, set_alert_profile, set_global_buffer
• check_availability

Confirmation flow

By default, mutating tools (anything that writes to a calendar) emit action.proposed first. Your client renders a confirmation (or auto-confirms based on user preference); on user confirmation, send action.confirm. Read-only tools execute without prompting.

Sample flow

const ws = new WebSocket(`wss://api.chronapilot.com/v1/voice/realtime?session=${sessionToken}`);

ws.onopen = () => {
  ws.send(JSON.stringify({
    type: 'session.update',
    session: { voice: 'sage', output_format: 'pcm16' }
  }));
};

ws.onmessage = (m) => {
  const ev = JSON.parse(m.data);
  switch (ev.type) {
    case 'transcript.final':
      console.log('user said:', ev.text);
      break;
    case 'action.proposed':
      ui.showConfirm(ev.action, () => ws.send(JSON.stringify({ type: 'action.confirm', id: ev.id })));
      break;
    case 'audio.delta':
      audioPlayer.enqueue(base64ToPCM(ev.audio));
      break;
  }
};

Reference implementation

• VoiceHub.cs — SignalR hub
• OpenAIRealtimeService.cs — backend bridge
• FunctionCallOrchestrator.cs — tool dispatch
• CalendarToolDefinitions.cs — tool schemas