MetaHuman AI Chat 1.0.0 documentation

Servers

socketio://localhost/socketioproduction

Handshake

object

session

required

string

Session ID

type

required

string

Sender or Receiver type

"sender"
"receiver"

Additional properties are allowed.

Operations

SEND session.update

A client emits the event to update the session

Operation IDsession.update.subscribe

Available only on servers:

production

object

address

session.updated

object

contentType

application/json

object

description

The server emits the event to the Sender when session is updated

#Sender

Accepts the following message:

subscribe.message

application/json

Message IDsubscribe.message

object

required

object

provider

required

string

{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"google","title":"Google"}
{"name":"mistral","title":"Mistral"}

required

oneOf

Adheres to:

string

"whisper-1"
"gpt-4o-transcribe"
"gpt_4o_mini_transcribe"

Or to:

string

No available models (used a default model)

Or to:

string

No available models (used a default model)

Or to:

string

"voxtral-mini-latest"
"voxtral-small-latest"

mediaFormat

required

string

languageCode

string

sentimentAnalysis

boolean

Default value:false

prompt

string

Additional properties are NOT allowed.

required

object

model

required

string

prompt

required

string

temperature

required

number

[ 0 .. 1 ]

array<string>

maxToken

number

provider

string

{"name":"aigames","title":"AI_Games"}
{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"azure","title":"Azure"}
{"name":"clarity","title":"Clarity"}
{"name":"bayer","title":"Bayer"}

name

string

stream

boolean

Default value:false

Additional properties are NOT allowed.

required

object

provider

required

string

{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"azure","title":"Azure"}
{"name":"google","title":"Google"}

required

oneOf

Adheres to:

string

"tts-1"
"tts-1-hd"
"gpt-4o-mini-tts"

Or to:

string

No available models (used a default model)

Or to:

string

"gpt-4"
"gpt-4o"
"gpt-4-turbo"
"gpt-3.5-turbo"
"gpt-5-pro"
"gpt-5-codex"

responseType

string

{"name":"single","title":"Single"}
{"name":"chain","title":"Chain"}
{"name":"stream","title":"Stream"}

languageCode

required

string

voice

string

speed

number

[ 0 .. 4 ]

ssmlGender

string

{"name":"unspecified","title":"Unspecified"}
{"name":"neutral","title":"Neutral"}
{"name":"male","title":"Male"}
{"name":"female","title":"Female"}

stream

boolean

Default value:false

isBase64

boolean

Default value:true

isOrdered

boolean

Experimental field

Default value:false

Additional properties are NOT allowed.

Examples

{
  "STT": {
    "provider": {
      "name": "openai",
      "title": "OpenAI"
    },
    "model": "whisper-1",
    "mediaFormat": "mp3",
    "languageCode": "en-US",
    "sentimentAnalysis": false,
    "prompt": "string"
  },
  "LLM": {
    "model": "gpt-4o",
    "prompt": "You are useful MetaHuman assistant",
    "temperature": 1,
    "stop": [
      "string"
    ],
    "maxToken": 0,
    "provider": {
      "name": "aigames",
      "title": "AI_Games"
    },
    "name": "string",
    "stream": false
  },
  "TTS": {
    "provider": {
      "name": "openai",
      "title": "OpenAI"
    },
    "model": "tts-1",
    "responseType": {
      "name": "single",
      "title": "Single"
    },
    "languageCode": "en-US",
    "voice": "string",
    "speed": 4,
    "ssmlGender": {
      "name": "unspecified",
      "title": "Unspecified"
    },
    "stream": false,
    "isBase64": true,
    "isOrdered": false
  }
}

This example has been generated automatically.

RECEIVE session.updated

The server emits the event to the Sender when session is updated

Operation IDsession.updated.publish

Available only on servers:

production

#Sender

Accepts the following message:

publish.message

application/json

Message IDpublish.message

object

Examples

{
  "sessionId": "32a5e96a-c35d-4e97-8ee0-a2166057feac",
  "success": true
}

This example has been generated automatically.

RECEIVE input.buffer.initialized

The Server sends to the Sender when a first audio chunk is received

Operation IDinput.buffer.initialized.publish

Available only on servers:

production

#Sender

Accepts the following message:

publish.message

application/json

Message IDpublish.message

object

Examples

{
  "sessionId": "32a5e96a-c35d-4e97-8ee0-a2166057feac"
}

This example has been generated automatically.

SEND input.buffer.commit

The Sender signalize the process of streaming audio is completed

Operation IDinput.buffer.commit.subscribe

Available only on servers:

production

object

description

The Server sends to the Sender when an audio is started processing

address

input.buffer.commited

object

contentType

application/json

object

address

request.transcribe.done

object

contentType

application/json

object

text

required

string

sentiment

string

language

string

Additional properties are NOT allowed.

description

The Server sends the transcribed text to the Sender

object

address

response.text

object

contentType

application/json

object

description

The Server sends the LLM response to the Receiver

object

address

response.audio.delta

object

contentType

application/json

object

required

number

The index of the audio chunk

text

required

string

The whole text response

required

restricted any oneOf

isBase64

required

boolean

done

required

boolean

Indicates if the whole response is done

streamPartDone

required

boolean

Indicates if the stream part is done. Useful for TtsResponseType.Chain when text is split into multiple requests

totalNumber

number

Additional properties are NOT allowed.

description

The Server sends audio response to the Receiver

#Sender

Accepts one of the following messages:

#0subscribe.message

Message IDsubscribe.message

No payload

Examples
#1connect_error
- application/json
Message IDconnect_error

object

name
required

string

message
required

string

required

object

message
required

string

statusCode
required

number

Additional properties are allowed.

Additional properties are allowed.
Examples
{ "name": "string", "message": "string", "data": { "message": "string", "statusCode": 0 } }

This example has been generated automatically.
#2custom.error
- application/json
Message IDcustom.error

object

message
required

string

errorId
required

number

Additional properties are allowed.
Examples
{ "message": "string", "errorId": 0 }

This example has been generated automatically.

RECEIVE request.transcribe.done

The Server sends the transcribed text to the Sender

Operation IDrequest.transcribe.done.publish

Available only on servers:

production

#Sender

Accepts the following message:

publish.message

application/json

Message IDpublish.message

object

Examples

{
  "text": "string",
  "sentiment": "string",
  "language": "string"
}

This example has been generated automatically.

RECEIVE response.text

The Server sends the LLM response to the Receiver

Operation IDresponse.text.publish

Available only on servers:

production

#Receiver

Accepts the following message:

publish.message

application/json

Message IDpublish.message

object

Examples

{
  "text": "string"
}

This example has been generated automatically.

RECEIVE response.audio.delta

The Server sends audio response to the Receiver

Operation IDresponse.audio.delta.publish

Available only on servers:

production

#Receiver

Accepts the following message:

publish.message

application/json

Message IDpublish.message

object

required

number

The index of the audio chunk

text

required

string

The whole text response

required

restricted any oneOf

isBase64

required

boolean

done

required

boolean

Indicates if the whole response is done

streamPartDone

required

boolean

Indicates if the stream part is done. Useful for TtsResponseType.Chain when text is split into multiple requests

totalNumber

number

Additional properties are NOT allowed.

Examples

{
  "id": 0,
  "text": "string",
  "audio": "string",
  "isBase64": true,
  "done": true,
  "streamPartDone": true,
  "totalNumber": 0
}

This example has been generated automatically.

Schemas

object

required

object

provider

required

string

{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"google","title":"Google"}
{"name":"mistral","title":"Mistral"}

required

oneOf

Adheres to:

string

"whisper-1"
"gpt-4o-transcribe"
"gpt_4o_mini_transcribe"

Or to:

string

No available models (used a default model)

Or to:

string

No available models (used a default model)

Or to:

string

"voxtral-mini-latest"
"voxtral-small-latest"

mediaFormat

required

string

languageCode

string

sentimentAnalysis

boolean

Default value:false

prompt

string

Additional properties are NOT allowed.

required

object

model

required

string

prompt

required

string

temperature

required

number

[ 0 .. 1 ]

array<string>

maxToken

number

provider

string

{"name":"aigames","title":"AI_Games"}
{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"azure","title":"Azure"}
{"name":"clarity","title":"Clarity"}
{"name":"bayer","title":"Bayer"}

name

string

stream

boolean

Default value:false

Additional properties are NOT allowed.

required

object

provider

required

string

{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"azure","title":"Azure"}
{"name":"google","title":"Google"}

required

oneOf

Adheres to:

string

"tts-1"
"tts-1-hd"
"gpt-4o-mini-tts"

Or to:

string

No available models (used a default model)

Or to:

string

"gpt-4"
"gpt-4o"
"gpt-4-turbo"
"gpt-3.5-turbo"
"gpt-5-pro"
"gpt-5-codex"

responseType

string

{"name":"single","title":"Single"}
{"name":"chain","title":"Chain"}
{"name":"stream","title":"Stream"}

languageCode

required

string

voice

string

speed

number

[ 0 .. 4 ]

ssmlGender

string

{"name":"unspecified","title":"Unspecified"}
{"name":"neutral","title":"Neutral"}
{"name":"male","title":"Male"}
{"name":"female","title":"Female"}

stream

boolean

Default value:false

isBase64

boolean

Default value:true

isOrdered

boolean

Experimental field

Default value:false

Additional properties are NOT allowed.

object

provider

required

string

{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"google","title":"Google"}
{"name":"mistral","title":"Mistral"}

required

oneOf

Adheres to:

string

"whisper-1"
"gpt-4o-transcribe"
"gpt_4o_mini_transcribe"

Or to:

string

No available models (used a default model)

Or to:

string

No available models (used a default model)

Or to:

string

"voxtral-mini-latest"
"voxtral-small-latest"

mediaFormat

required

string

languageCode

string

sentimentAnalysis

boolean

Default value:false

prompt

string

Additional properties are NOT allowed.

object

model

required

string

prompt

required

string

temperature

required

number

[ 0 .. 1 ]

array<string>

maxToken

number

provider

string

{"name":"aigames","title":"AI_Games"}
{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"azure","title":"Azure"}
{"name":"clarity","title":"Clarity"}
{"name":"bayer","title":"Bayer"}

name

string

stream

boolean

Default value:false

Additional properties are NOT allowed.

object

provider

required

string

{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"azure","title":"Azure"}
{"name":"google","title":"Google"}

required

oneOf

Adheres to:

string

"tts-1"
"tts-1-hd"
"gpt-4o-mini-tts"

Or to:

string

No available models (used a default model)

Or to:

string

"gpt-4"
"gpt-4o"
"gpt-4-turbo"
"gpt-3.5-turbo"
"gpt-5-pro"
"gpt-5-codex"

responseType

string

{"name":"single","title":"Single"}
{"name":"chain","title":"Chain"}
{"name":"stream","title":"Stream"}

languageCode

required

string

voice

string

speed

number

[ 0 .. 4 ]

ssmlGender

string

{"name":"unspecified","title":"Unspecified"}
{"name":"neutral","title":"Neutral"}
{"name":"male","title":"Male"}
{"name":"female","title":"Female"}

stream

boolean

Default value:false

isBase64

boolean

Default value:true

isOrdered

boolean

Experimental field

Default value:false

Additional properties are NOT allowed.

object

restricted any oneOf

object

required

number

The index of the audio chunk

text

required

string

The whole text response

required

restricted any oneOf

isBase64

required

boolean

done

required

boolean

Indicates if the whole response is done

streamPartDone

required

boolean

Indicates if the stream part is done. Useful for TtsResponseType.Chain when text is split into multiple requests

totalNumber

number

Additional properties are NOT allowed.

STT_Provider

string

{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"google","title":"Google"}
{"name":"mistral","title":"Mistral"}

LLM_Provider

string

{"name":"aigames","title":"AI_Games"}
{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"azure","title":"Azure"}
{"name":"clarity","title":"Clarity"}
{"name":"bayer","title":"Bayer"}

TTS_Provider

string

{"name":"openai","title":"OpenAI"}
{"name":"aws","title":"AWS"}
{"name":"azure","title":"Azure"}
{"name":"google","title":"Google"}

TtsResponseType

string

{"name":"single","title":"Single"}
{"name":"chain","title":"Chain"}
{"name":"stream","title":"Stream"}

SsmlGender

string

{"name":"unspecified","title":"Unspecified"}
{"name":"neutral","title":"Neutral"}
{"name":"male","title":"Male"}
{"name":"female","title":"Female"}

restricted any oneOf

sttOpenAiModel

string

"whisper-1"
"gpt-4o-transcribe"
"gpt_4o_mini_transcribe"

sttAwsModel

string

No available models (used a default model)

sttGoogleModel

string

No available models (used a default model)

sttMistralModel

string

"voxtral-mini-latest"
"voxtral-small-latest"

llmAwsModel

string

Default value:"anthropic.claude-instant-v1"

{"name":"anthropic.claude-v2","title":"Anthropic_Claude_v2"}
{"name":"anthropic.claude-v1","title":"Anthropic_claude_v1"}
{"name":"anthropic.claude-instant-v1","title":"Anthropic_claude_instant_v1"}

llmClarityModel

string

No available models (used a default model)

llmOpenAiModel

string

"gpt-4.1"
"gpt-4.1-mini"
"gpt-4.1-nano"
"gpt-4.1-2025-04-14"
"gpt-4.1-mini-2025-04-14"
"gpt-4.1-nano-2025-04-14"
"o4-mini"
"o4-mini-2025-04-16"
"o3"
"o3-2025-04-16"
"o3-mini"
"o3-mini-2025-01-31"
"o1"
"o1-2024-12-17"
"o1-preview"
"o1-preview-2024-09-12"
"o1-mini"
"o1-mini-2024-09-12"
"gpt-4o"
"gpt-4o-2024-11-20"
"gpt-4o-2024-08-06"
"gpt-4o-2024-05-13"
"gpt-4o-audio-preview"
"gpt-4o-audio-preview-2024-10-01"
"gpt-4o-audio-preview-2024-12-17"
"gpt-4o-mini-audio-preview"
"gpt-4o-mini-audio-preview-2024-12-17"
"gpt-4o-search-preview"
"gpt-4o-mini-search-preview"
"gpt-4o-search-preview-2025-03-11"
"gpt-4o-mini-search-preview-2025-03-11"
"chatgpt-4o-latest"
"codex-mini-latest"
"gpt-4o-mini"
"gpt-4o-mini-2024-07-18"
"gpt-4-turbo"
"gpt-4-turbo-2024-04-09"
"gpt-4-0125-preview"
"gpt-4-turbo-preview"
"gpt-4-1106-preview"
"gpt-4-vision-preview"
"gpt-4"
"gpt-4-0314"
"gpt-4-0613"
"gpt-4-32k"
"gpt-4-32k-0314"
"gpt-4-32k-0613"
"gpt-3.5-turbo"
"gpt-3.5-turbo-16k"
"gpt-3.5-turbo-0301"
"gpt-3.5-turbo-0613"
"gpt-3.5-turbo-1106"
"gpt-3.5-turbo-0125"
"gpt-3.5-turbo-16k-0613"

llmAzureModel

string

"gpt-35-turbo"
"gpt-35-turbo-16k"
"gpt-4"
"gpt-4-32k"
"amazon.titan-tg1-large"
"anthropic.claude"
"anthropic.claude-v3-opus"
"anthropic.claude-v3-sonnet"
"anthropic.claude-v3-5-sonnet"
"anthropic.claude-v3-haiku"
"gpt-4-turbo"
"gpt-4o"
"gpt-4o-mini-2024-07-18"
"gpt-4.1-2025-04-14"
"gpt-4.1-nano-2025-04-14"
"gpt-4.1-mini-2025-04-14"
"anthropic.claude-v3-5-haiku"
"anthropic.claude-3-7-sonnet-20250219-v1:0"
"anthropic.claude-sonnet-4-20250514-v1:0"
"anthropic.claude-opus-4-20250514-v1:0"

llmAiGamesModel

string

"ATLAS"
"ORION"
"NOVA"
"RHEA"

llmBayerModel

string

"o4-mini"
"grok-3"
"OpenAI Text Embedding 3 (small)"
"OpenAI Text Embedding 3 (large)"
"Claude Sonnet 4"
"Cohere Rerank v3.5"
"Gemini 2.5 Pro"
"Gemini 2.5 Flash"
"Gemini 2.0 Flash"
"GPT 4 Turbo"
"bge-m3"
"Llama 3.1 405B Instruct"
"GPT-5"
"GPT-5 Nano"
"GPT-4o mini"
"BGE Reranker v2 M3"
"Claude 3.5 Sonnet"
"gpt-4o-2024-08-06"
"GPT-5 Mini"
"Claude Opus 4"
"DeepSeek R1 Reasoning"
"o3-mini"
"o1-mini"
"o1-preview"
"GPT-4"
"Mistral Large"
"Command-R+"
"GPT-3.5"
"GPT-4 Turbo Vision"
"BioLlama-3 70B"
"Llama 3.1 70B Instruct"
"Llama 3.1 8B Instruct"
"GPT-4 (0125-preview)"
"Llama-3 8B"
"Llama-3 70B"
"Claude Opus 4.1"
"GPT‑OSS‑120B"
"Claude Sonnet 4.5"
"GPT 4o"
"Claude 3.7 Sonnet"
"MedCPT Reranker"
"GPT 4o mini Batch"
"GPT 4o Batch"
"o3 mini Batch"
"GPT-4o"
"gpt-4.1"

ttsAzureModel

string

"gpt-4"
"gpt-4o"
"gpt-4-turbo"
"gpt-3.5-turbo"
"gpt-5-pro"
"gpt-5-codex"

ttsOpenAiModel

string

"tts-1"
"tts-1-hd"
"gpt-4o-mini-tts"

ttsAwsModel

string

No available models (used a default model)

ttsAzureVoice

string

https://westus2.tts.speech.microsoft.com/cognitiveservices/voices/list

"en-US-JennyNeural"
"en-US-SaraNeural"
"en-US-TonyNeural"
"en-US-NancyNeural"

ttsOpenAiVoice

string

"alloy"
"ash"
"ballad"
"coral"
"echo"
"fable"
"onyx"
"nova"
"sage"
"shimmer"
"verse"

ttsAwsVoice

string

"Aditi"
"Adriano"
"Amy"
"Andres"
"Aria"
"Arlet"
"Arthur"
"Astrid"
"Ayanda"
"Bianca"
"Brian"
"Camila"
"Carla"
"Carmen"
"Celine"
"Chantal"
"Conchita"
"Cristiano"
"Daniel"
"Danielle"
"Dora"
"Elin"
"Emma"
"Enrique"
"Ewa"
"Filiz"
"Gabrielle"
"Geraint"
"Giorgio"
"Gregory"
"Gwyneth"
"Hala"
"Hannah"
"Hans"
"Hiujin"
"Ida"
"Ines"
"Isabelle"
"Ivy"
"Jacek"
"Jan"
"Joanna"
"Joey"
"Justin"
"Kajal"
"Karl"
"Kazuha"
"Kendra"
"Kevin"
"Kimberly"
"Laura"
"Lea"
"Liam"
"Lisa"
"Liv"
"Lotte"
"Lucia"
"Lupe"
"Mads"
"Maja"
"Marlene"
"Mathieu"
"Matthew"
"Maxim"
"Mia"
"Miguel"
"Mizuki"
"Naja"
"Niamh"
"Nicole"
"Ola"
"Olivia"
"Pedro"
"Penelope"
"Raveena"
"Remi"
"Ricardo"
"Ruben"
"Russell"
"Ruth"
"Salli"
"Seoyeon"
"Sergio"
"Sofie"
"Stephen"
"Suvi"
"Takumi"
"Tatyana"
"Thiago"
"Tomoko"
"Vicki"
"Vitoria"
"Zayd"
"Zeina"
"Zhiyu"

object

name

required

string

message

required

string

required

object

Additional properties are allowed.

object

MetaHuman AI Chat 1.0.0

Servers

Operations

SEND session.update

Examples

This example has been generated automatically.

RECEIVE session.updated

Examples

This example has been generated automatically.

SEND input.buffer.append

Examples

This example has been generated automatically.

RECEIVE input.buffer.initialized

Examples

This example has been generated automatically.

SEND input.buffer.commit

Examples

Examples

This example has been generated automatically.

Examples

This example has been generated automatically.

RECEIVE request.transcribe.done

Examples

This example has been generated automatically.

RECEIVE response.text

Examples

This example has been generated automatically.

RECEIVE response.audio.delta

Examples

This example has been generated automatically.

Schemas