目录

背景

分析

解决方案

---

背景

正常情况下，会从Openai Realtime api Sever收到正常的json数据,但是当返回音频数据时，总会返回非json数据。这是什么问题呢？

分析

期望的完整响应数据如下：

{
  "session": {
    "input_audio_format": "pcm16",
    "instructions": "Role: 1.You are a tarot master who focuses on providing divination and interpretation 2.Your name is Luna 3.Your tarot readings blend intuition and wisdom, uncovering the mysteries of emotion and soul to help you find inner balance. \\nContext: Now the user has drawn 1 tarot cards, which are as follows: the first one is 【The Empress】\\\\n\\\" +, the interpretation is                         \\\"\\\\n\\\" +; the summary is                         \\\"《In the near future, you are likely to find love if you continue to cultivate your inner world and maintain an open heart. Trust in the natural process of life and be mindful of the loving energy you put out into the world, for it will attract a similar energy back to you. Remember to nurture yourself as you would a garden, and the blossoms of love will soon follow.》.\\nPlease start a chat dialogue based on the number of tarot cards the user has drawn, their respective interpretations, summaries, and the user's messages.\\nNote: 1. Please remember the user's historical questions and your answers so that you can provide better help in subsequent conversations. 2.The output characters should be less than 150.",
    "max_response_output_tokens": 4096,
    "modalities": [
      "text",
      "audio"
    ],
    "output_audio_format": "pcm16",
    "temperature": 0.8,
    "tool_choice": "auto",
    "tools": [
      
    ],
    "turn_detection": {
      "prefix_padding_ms": 300,
      "silence_duration_ms": 500,
      "threshold": 0.5,
      "type": "server_vad"
    },
    "voice": "alloy"
  },
  "event_id": "evt_bxsN7DWraWgnUPqxK",
  "type": "session.update"
}

实际收到的数据类似如下：

eart. Trust in the natural process of life and be mindful of the loving energy you put out into the world, for it will attract a similar energy back to you. Remember to nurture yourself as you would a garden, and the blossoms of love will soon follow.》.\\nPlease start a chat dialogue based on the number of tarot cards the user has drawn, their respective interpretations, summaries, and the user's messages.\\nNote: 1. Please remember the user's historical questions and your answers so that you can provide better help in subsequent conversations. 2.The output characters should be less than 150.",
    "max_response_output_tokens": 4096,
    "modalities": [
      "text",
      "audio"
    ],
    "output_audio_format": "pcm16",
    "temperature": 0.8,
    "tool_choice": "auto",
    "tools": [
      
    ],
    "turn_detection": {
      "prefix_padding_ms": 300,
      "silence_duration_ms": 500,
      "threshold": 0.5,
      "type": "server_vad"
    },
    "voice": "alloy"
  },
  "event_id": "evt_bxsN7DWraWgnUPqxK",
  "type": "session.update"
}

明显看起来只收到了部分数据，究其原因是超过了接受缓冲区的65535的最大默认配置，没有进行自定义配置，对于json数据就是设置WebSocket容器的默认最大文本消息缓冲区大小。

解决方案

设置最大文本消息缓冲区大小，具体代码如下：

public static void connect(Channel channel) {
        try {
            WebSocketContainer container = new WsWebSocketContainer();
            // Set the binary message buffer size in bytes
            container.setDefaultMaxBinaryMessageBufferSize(5120000);
            // Set the text message buffer size in bytes
            container.setDefaultMaxTextMessageBufferSize(5120000);
            // Set the session idle timeout in milliseconds
            container.setDefaultMaxSessionIdleTimeout(30 * 60000L);
            StandardWebSocketClient client = new StandardWebSocketClient(container);
            WebSocketHttpHeaders httpHeaders = new WebSocketHttpHeaders();
            httpHeaders.add("Authorization", "Bearer sk-***");
            httpHeaders.add("OpenAI-Beta", "realtime=v1");
            WebSocketSession session = client.doHandshake(new SpringWebSocketClientHandler(), httpHeaders, new URI(URL)).get();
            if (session.isOpen()) {
                log.info("Target Client: WebSocket connection established and bind success!");
                log.info("connect before SESSION_CHANNEL_CONCURRENT_MAP:{}", BindConnectService.SESSION_CHANNEL_CONCURRENT_MAP);
                BindConnectService.safeBindChannelSession(session, channel);
                log.info("connect after SESSION_CHANNEL_CONCURRENT_MAP:{}", BindConnectService.SESSION_CHANNEL_CONCURRENT_MAP);
            } else {
                log.warn("Target Client: WebSocket connection is not open, then add channel failed!");
                channel.close();
            }
        } catch (Exception e) {
            log.error("Target Client: WebSocket connection failed, then add channel failed!", e);
            channel.close();
        }
    }