sip_realtime_ai

Overview

sip_realtime_ai is a standalone Linux SIP-to-realtime-AI gateway. It accepts SIP calls, establishes RTP audio sessions, selects an AI profile from the SIP Request-URI user and exchanges audio with the selected provider.

Installation

Officially supported platform: Debian 13 (Trixie), amd64. The gateway is distributed as a Debian package and runs independently of FreeSWITCH. Builds for other Linux distributions may be available upon request.

sudo apt install ./sip-realtime-ai_*.deb

The package installs a sample XML gateway configuration and an OpenAI session JSON example in the following locations:

/etc/sip_realtime_ai/
├── sip_realtime_ai.conf
└── realtime_ai/
    └── sales.json

The main .conf file contains XML. Update the gateway settings and AI profile credentials, then customize the OpenAI session settings:

sudo nano /etc/sip_realtime_ai/sip_realtime_ai.conf
sudo nano /etc/sip_realtime_ai/realtime_ai/sales.json

Additional provider profiles can reference their own JSON files placed in /etc/sip_realtime_ai/realtime_ai/. The package includes the OpenAI example; other provider configurations can be added as needed.

Configuration

The gateway uses an XML configuration with three top-level sections: <license>, <settings> and <profiles>. The example below reflects the tested local configuration; replace the credentials before deployment. The license example is for trial/commercial use; Community Edition needs no license.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <license>
    <param name="product" value="sip_realtime_ai"/>
    <param name="type" value="trial"/>
    <param name="issued_to" value="YOUR_COMPANY"/>
    <param name="expires_at" value="YOUR_EXPIRY"/>
    <param name="sig" value="YOUR_LICENSE_SIGNATURE"/>
  </license>
  <settings>
    <param name="sip-bind" value="127.0.0.1"/>
    <param name="sip-port" value="5070"/>
    <param name="rtp-bind" value="127.0.0.1"/>
    <param name="rtp-port-start" value="10000"/>
    <param name="rtp-port-end" value="19998"/>
  </settings>
  <profiles>
    <profile name="sales">
      <param name="provider" value="openai"/>
      <param name="api-key" value="YOUR_API_KEY"/>
      <param name="model" value="gpt-realtime-2.1"/>
      <param name="config" value="sales.json"/>
    </profile>
  </profiles>
</configuration>

Licensing

Community Edition: free for up to 10 concurrent channels, with no license required. Trial and commercial licenses are available for deployments that need higher capacity.

Trial and commercial licenses use the <license> section. The product value is sip_realtime_ai; the license supplies the remaining signed fields.

sip_realtime_ai license status
sip_realtime_ai license request

license status displays license information; license request prepares a request for a trial or commercial license. Community Edition users do not need to run either command.

Do not publish license signatures or provider API keys.

SIP / RTP Settings

The <settings> section defines local SIP and RTP interfaces and ports. The values below are from the local test configuration, not recommended public bind addresses.

<settings>
  <param name="sip-bind" value="127.0.0.1"/>
  <param name="sip-port" value="5070"/>
  <param name="rtp-bind" value="127.0.0.1"/>
  <param name="rtp-port-start" value="10000"/>
  <param name="rtp-port-end" value="19998"/>
</settings>

In this configuration, SIP signaling uses UDP port 5070 and RTP uses UDP ports 10000–19998. Binding to 127.0.0.1 limits traffic to the local host; remote endpoints require an appropriate reachable interface and firewall rules.

SIP Routing

The gateway selects a named AI profile using the user portion of the SIP Request-URI. For example, a call addressed to sip:sales@gateway.example selects the sales profile.

Each destination used for routing must have a corresponding profile configured under <profiles>. The example hostname is illustrative, not a required gateway hostname.

SDES-SRTP

The gateway supports SDES-SRTP for encrypted RTP media. Key material is negotiated in the SIP/SDP exchange using an a=crypto attribute with an inline: key. The supported crypto suite is AES_CM_128_HMAC_SHA1_80 (AES-128 counter-mode encryption with an 80-bit HMAC-SHA1 authentication tag).

m=audio 10000 RTP/SAVP 0 8
 a=rtpmap:0 PCMU/8000
 a=rtpmap:8 PCMA/8000
 a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:<BASE64_MASTER_KEY_AND_SALT>

For an accepted secure-media offer, the gateway generates an RTP/SAVP answer with the selected a=crypto tag, suite and its own inline key material. Both PCMU and PCMA can be offered as RTP payload formats, subject to codec negotiation.

Audio Codecs

The gateway RTP media path supports PCMU (G.711 μ-law) and PCMA (G.711 A-law) for 8 kHz telephony audio. Provider-side formats are configured separately and may differ from the SIP media codec.

Codec support on a given call also depends on the endpoint SDP offer and the codec selected during negotiation.

Audio Processing

Caller audio is streamed to the selected realtime AI provider, while AI-generated speech is returned over RTP. The gateway handles media conversion and resampling independently in each direction when the provider requires a different audio format or sample rate. SIP-side PCMU/PCMA and provider-side audio formats do not need to match.

For example, the tested OpenAI profile uses 24 kHz PCM audio while the SIP side uses G.711 telephony media. The tested Azure profile receives G.711 μ-law at 8 kHz and sends PCM16 at 8 kHz. Provider-specific turn detection and response interruption settings remain in the provider-native JSON configuration. Barge-in behavior depends on provider support and session settings.

SIP INFO Commands

During an active SIP call, send an in-dialog INFO request with Content-Type: application/json to interact with the selected AI provider.

Send a text message to the active AI session. The gateway translates the message into the selected provider's native protocol.

{
  "command": "send_text",
  "text": "Hello from SIP INFO!"
}

Forward a provider-native JSON object to the active AI session. The data object is passed through without translating provider-specific fields. Supported updates depend on the provider.

{
  "command": "update",
  "data": {
    "type": "session.update",
    "session": {
      "instructions": "Always respond in English."
    }
  }
}

Profiles

Define one or more named profiles in the <profiles> section. A profile selects the provider, API key and provider-specific model or configuration. Profiles can be named according to their SIP destination, such as sales or support.

Provider JSON files contain native provider session settings. They are not interchangeable across providers. The gateway forwards provider-specific session configuration without introducing a proprietary schema for provider options. Supported native options may include voices, instructions, VAD, interruption behavior, audio formats and tool/function declarations. Tool execution and external integrations still require the corresponding provider and application-side implementation; declaring a function alone does not implement it.

OpenAI

Configure an OpenAI gateway profile with your API key, realtime model and a provider-native JSON configuration file. The tested example uses PCM16 at 24 kHz in both directions.

<profile name="sales">
    <param name="provider" value="openai"/>
    <param name="api-key" value="YOUR_API_KEY"/>
    <param name="config" value="sales.json"/>
    <param name="model" value="gpt-realtime-2.1"/>
</profile>

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "You are a helpful voice assistant.",
    "output_modalities": [
      "audio"
    ],
    "audio": {
      "input": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.7,
          "prefix_padding_ms": 300,
          "silence_duration_ms": 500,
          "create_response": true,
          "interrupt_response": true
        }
      },
      "output": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "voice": "marin"
      }
    }
  }
}

Google Gemini

Gemini profiles use an API key and a provider-native JSON configuration file.

<profile name="support">
    <param name="provider" value="gemini"/>
    <param name="api-key" value="YOUR_API_KEY"/>
    <param name="config" value="support.json"/>
</profile>

{
  "setup": {
    "model": "models/gemini-3.8-live",
    "generationConfig": {
      "responseModalities": [
        "AUDIO"
      ]
    },
    "tools": [
      {
        "functionDeclarations": [
          {
            "name": "get_current_time",
            "description": "Get the current local time.",
            "parameters": {
              "type": "OBJECT",
              "properties": {}
            }
          }
        ]
      }
    ]
  }
}

ElevenLabs

ElevenLabs uses an existing Agent configuration. Specify the ElevenLabs Agent ID through the profile's model parameter.

<profile name="elevenlabs">
    <param name="provider" value="elevenlabs"/>
    <param name="api-key" value="YOUR_API_KEY"/>
    <param name="model" value="YOUR_AGENT_ID"/>
</profile>

Azure Voice Live

Azure Voice Live requires an Azure endpoint in addition to the API key, model and provider-native JSON session configuration.

<profile name="azure">
    <param name="provider" value="azure"/>
    <param name="api-key" value="YOUR_API_KEY"/>
    <param name="config" value="azure.json"/>
    <param name="model" value="gpt-realtime"/>
    <param name="endpoint" value="https://your-resource.services.ai.azure.com"/>
</profile>

{
  "type": "session.update",
  "session": {
    "modalities": [
      "text",
      "audio"
    ],
    "voice": {
      "type": "openai",
      "name": "alloy"
    },
    "instructions": "You are a helpful voice assistant. Respond briefly and naturally.",
    "input_audio_format": "g711_ulaw",
    "output_audio_format": "pcm16_8000hz",
    "input_audio_sampling_rate": 8000,
    "input_audio_transcription": {
      "model": "gpt-4o-mini-transcribe"
    },
    "turn_detection": {
      "type": "azure_semantic_vad",
      "threshold": 0.5,
      "prefix_padding_ms": 420,
      "silence_duration_ms": 500
    }
  }
}

Troubleshooting

Check the XML syntax and required <license> section. If licensing fails, review the license type, expiration and signature supplied for this gateway.

Verify sip-bind, sip-port, local firewall and SIP routing. A service bound to 127.0.0.1 cannot receive direct traffic from another machine.

Check the user part of the SIP Request-URI and confirm a matching <profile name="..."> exists.

Verify the negotiated PCMU/PCMA codec, RTP port range, firewall/NAT path and the provider's native input/output audio format.

Verify the provider API key, model or Agent ID, endpoint where required, and the provider-native JSON configuration. Review gateway logs for the reported failure.