Compare commits

...

12 Commits

Author SHA1 Message Date
Terrence
91c7774117 bump to 0.6.2 2024-11-01 15:19:37 +08:00
Terrence
b0bc81b921 add new boards 2024-11-01 14:26:02 +08:00
Terrence
a701d5918e add abort command 2024-10-31 05:57:13 +08:00
Terrence
6f5f5a0642 use protocol 3 2024-10-30 06:58:29 +08:00
Terrence
3e1e576272 update ml307 component version 2024-10-29 00:26:05 +08:00
Terrence
33518dca2b add config files for known boards 2024-10-29 00:22:29 +08:00
Terrence
fe05a039a2 fix v0.4.x board name 2024-10-25 12:46:33 +08:00
Terrence
32a5977fcc update to 0.4.1 2024-10-25 12:29:23 +08:00
Terrence
357cd95693 fix audio data overflow 2024-10-25 12:25:22 +08:00
Terrence
2833142870 update docs 2024-10-25 10:05:01 +08:00
Terrence
a2487f46c8 ES8311 + ES7210 2024-10-24 09:53:08 +08:00
Terrence
4c6da771ec add versions.py for ota upgrade 2024-10-17 18:34:51 +08:00
56 changed files with 2328 additions and 1007 deletions

1
.gitignore vendored
View File

@@ -8,3 +8,4 @@ sdkconfig.old
sdkconfig
dependencies.lock
.env
releases/

View File

@@ -4,7 +4,7 @@
# CMakeLists in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.16)
set(PROJECT_VER "0.3.3")
set(PROJECT_VER "0.6.2")
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(xiaozhi)

View File

@@ -1,9 +1,11 @@
# 小智 AI 聊天机器人
BiliBili 视频介绍 [【ESP32+SenseVoice+Qwen72B打造你的AI聊天伴侣](https://www.bilibili.com/video/BV11msTenEH3/?share_source=copy_web&vd_source=ee1aafe19d6e60cf22e60a93881faeba)
这是虾哥的第一个硬件作品。
[ESP32+SenseVoice+Qwen72B打造你的AI聊天伴侣【bilibili】](https://www.bilibili.com/video/BV11msTenEH3/?share_source=copy_web&vd_source=ee1aafe19d6e60cf22e60a93881faeba)
[手工打造你的 AI 女友新手入门教程【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
## 项目目的
本项目基于乐鑫的 ESP-IDF 进行开发。
@@ -18,16 +20,16 @@ BiliBili 视频介绍 [【ESP32+SenseVoice+Qwen72B打造你的AI聊天伴侣
- Wi-Fi 配网
- 支持 BOOT 键唤醒和打断
- 离线语音唤醒(使用乐鑫方案)
- 离线语音唤醒(乐鑫方案)
- 流式语音对话WebSocket 协议)
- 支持国语、粤语、英语、日语、韩语 5 种语言识别(使用 SenseVoice 方案)
- 支持国语、粤语、英语、日语、韩语 5 种语言识别SenseVoice 方案)
- 声纹识别(识别是谁在喊 AI 的名字,[3D Speaker 项目](https://github.com/modelscope/3D-Speaker)
- 使用大模型 TTS火山引擎方案,阿里云接入中
- 使用大模型 TTS火山引擎与 CosyVoice 方案)
- 支持可配置的提示词和音色(自定义角色)
- 免费提供 Qwen2.5 72B 豆包模型(受限于性能和额度,人多后可能会限额)
- Qwen2.5 72B 豆包 API
- 支持每轮对话后自我总结,生成记忆体
- 扩展液晶显示屏,显示信号强弱(后面可以显示中文字幕)
- 支持 ML307 Cat.1 4G 模块(可选)
- 扩展液晶显示屏,显示信号强弱
- 支持 ML307 Cat.1 4G 模块
## 硬件部分
@@ -35,60 +37,29 @@ BiliBili 视频介绍 [【ESP32+SenseVoice+Qwen72B打造你的AI聊天伴侣
[《小智 AI 聊天机器人百科全书》](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb?from=from_copylink)
第二版接线图如下:
面包板接线图如下:
![第二版接线图](docs/wiring2.jpg)
![面包板接线图](docs/wiring2.jpg)
## 固件部分
### 免开发环境烧录
新手第一次操作建议先不要搭建开发环境,直接使用免开发环境烧录的固件。
新手第一次操作建议先不要搭建开发环境,直接使用免开发环境烧录的固件。固件使用的是作者友情提供的测试服,目前开放免费使用,请勿用于商业用途。
点击 [这里](https://github.com/78/xiaozhi-esp32/releases) 下载最新版固件。
[Flash烧录固件无IDF开发环境](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)
固件使用的是作者友情提供的测试服,目前开放免费使用,请勿用于商业用途。
### 搭建开发环境
### 开发环境
- Cursor 或 VSCode
- 安装 ESP-IDF 插件,选择 SDK 版本 5.3 或以上
- Ubuntu 比 Windows 更好,编译速度快,也免去驱动问题的困扰
### 配置项目与编译固件
- 目前只支持 ESP32 S3Flash 至少 8MB, PSRAM 至少 2MB注意默认配置只兼容 8MB PSRAM如果你使用 2MB PSRAM需要修改配置否则无法识别
- 配置 OTA Version URL 为 `https://api.tenclass.net/xiaozhi/ota/`
- 配置 WebSocket URL 为 `wss://api.tenclass.net/xiaozhi/v1/`
- 配置 WebSocket Access Token 为 `test-token`
- 如果 INMP441 和 MAX98357 接线跟默认配置不一样,需要修改 GPIO 配置
- 配置完成后,编译固件
## AI 角色配置
如果你已经拥有一个小智 AI 聊天机器人,可以参考 [后台操作视频教程](https://www.bilibili.com/video/BV1jUCUY2EKM/)
## 配置 Wi-Fi 4G 版本跳过)
按照上述接线,烧录固件,设备上电后,开发板上的 RGB 会闪烁蓝灯(部分开发板需要焊接 RGB 灯的开关才会亮),进入配网状态。
打开手机 Wi-Fi连接上设备热点 `Xiaozhi-xxxx` 后,使用浏览器访问 `http://192.168.4.1`,进入配网页面。
选择你的路由器 WiFi输入密码点击连接设备会在 3 秒后自动重启,之后设备会自动连接到路由器。
## 测试设备是否连接成功
设备连接上路由器后,闪烁一下绿灯。此时,喊一声“你好,小智”,设备会先亮蓝灯(表示连接服务器),然后再亮绿灯,播放语音。
如果没有亮蓝灯,说明麦克风有问题,请检查接线是否正确。
如果没有亮绿灯,或者蓝灯常亮,说明设备没有连接到服务器,请检查 WiFi 连接是否正常。
如果设备已经连接 Wi-Fi但是没有声音请检查是否接线正确。
在 v0.2.1 版本之后的固件,也可以按下连接 GPIO 1 的按钮(低电平有效),进行录音测试。
## 配置设备
如果上述步骤测试成功,设备会播报你的设备 ID你需要到 [小智测试服的控制面板](https://xiaozhi.tenclass.net/) 页面,添加设备。
详细的使用说明以及测试服的注意事项,请参考 [小智测试服的帮助说明](https://xiaozhi.tenclass.net/help)。
详细的使用说明以及测试服的注意事项,请参考 [小智测试服的帮助说明](https://xiaozhi.me/help)。

40
convert_audio_to_p3.py Normal file
View File

@@ -0,0 +1,40 @@
# convert audio files to protocol v3 stream
import librosa
import opuslib
import struct
import sys
import tqdm
import numpy as np
def encode_audio_to_opus(input_file, output_file):
# Load audio file using librosa
audio, sample_rate = librosa.load(input_file, sr=None, mono=False, dtype=np.int16)
# Get left channel if stereo
if audio.ndim == 2:
audio = audio[0]
# Initialize Opus encoder
encoder = opuslib.Encoder(sample_rate, 1, opuslib.APPLICATION_VOIP)
# Encode audio data to Opus packets
# Save encoded data to file
with open(output_file, 'wb') as f:
sample_rate = 16000 # 16000Hz
duration = 60 # 60ms every frame
frame_size = int(sample_rate * duration / 1000)
for i in tqdm.tqdm(range(0, len(audio) - frame_size, frame_size)):
frame = audio[i:i + frame_size]
opus_data = encoder.encode(frame.tobytes(), frame_size=frame_size)
# protocol format, [1u type, 1u reserved, 2u len, data]
packet = struct.pack('>BBH', 0, 0, len(opus_data)) + opus_data
f.write(packet)
# Example usage
if len(sys.argv) != 3:
print('Usage: python convert.py <input_file> <output_file>')
sys.exit(1)
input_file = sys.argv[1]
output_file = sys.argv[2]
encode_audio_to_opus(input_file, output_file)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

After

Width:  |  Height:  |  Size: 57 KiB

View File

@@ -1,6 +1,4 @@
#include <BuiltinLed.h>
#include <TcpTransport.h>
#include <TlsTransport.h>
#include <Ml307SslTransport.h>
#include <WifiConfigurationAp.h>
#include <WifiStation.h>
@@ -10,46 +8,60 @@
#include <esp_log.h>
#include <cJSON.h>
#include <driver/gpio.h>
#include <arpa/inet.h>
#include "Application.h"
#define TAG "Application"
extern const char p3_err_reg_start[] asm("_binary_err_reg_p3_start");
extern const char p3_err_reg_end[] asm("_binary_err_reg_p3_end");
extern const char p3_err_pin_start[] asm("_binary_err_pin_p3_start");
extern const char p3_err_pin_end[] asm("_binary_err_pin_p3_end");
extern const char p3_err_wificonfig_start[] asm("_binary_err_wificonfig_p3_start");
extern const char p3_err_wificonfig_end[] asm("_binary_err_wificonfig_p3_end");
Application::Application()
: boot_button_((gpio_num_t)CONFIG_BOOT_BUTTON_GPIO),
volume_up_button_((gpio_num_t)CONFIG_VOLUME_UP_BUTTON_GPIO)
#ifdef CONFIG_USE_ML307
, ml307_at_modem_(CONFIG_ML307_TX_PIN, CONFIG_ML307_RX_PIN, 4096),
http_(ml307_at_modem_),
firmware_upgrade_(http_)
#else
, http_(),
firmware_upgrade_(http_)
#endif
#ifdef CONFIG_USE_DISPLAY
, display_(CONFIG_DISPLAY_SDA_PIN, CONFIG_DISPLAY_SCL_PIN)
#endif
: boot_button_((gpio_num_t)BOOT_BUTTON_GPIO),
volume_up_button_((gpio_num_t)VOLUME_UP_BUTTON_GPIO),
volume_down_button_((gpio_num_t)VOLUME_DOWN_BUTTON_GPIO),
display_(DISPLAY_SDA_PIN, DISPLAY_SCL_PIN)
{
event_group_ = xEventGroupCreate();
opus_encoder_.Configure(CONFIG_AUDIO_INPUT_SAMPLE_RATE, 1);
opus_encoder_.Configure(16000, 1);
opus_decoder_ = opus_decoder_create(opus_decode_sample_rate_, 1, NULL);
if (opus_decode_sample_rate_ != CONFIG_AUDIO_OUTPUT_SAMPLE_RATE) {
opus_resampler_.Configure(opus_decode_sample_rate_, CONFIG_AUDIO_OUTPUT_SAMPLE_RATE);
if (opus_decode_sample_rate_ != AUDIO_OUTPUT_SAMPLE_RATE) {
output_resampler_.Configure(AUDIO_OUTPUT_SAMPLE_RATE, opus_decode_sample_rate_);
}
if (16000 != AUDIO_INPUT_SAMPLE_RATE) {
input_resampler_.Configure(AUDIO_INPUT_SAMPLE_RATE, 16000);
}
firmware_upgrade_.SetCheckVersionUrl(CONFIG_OTA_VERSION_URL);
firmware_upgrade_.SetHeader("Device-Id", SystemInfo::GetMacAddress().c_str());
firmware_upgrade_.SetPostData(SystemInfo::GetJsonString());
}
Application::~Application() {
if (update_display_timer_ != nullptr) {
esp_timer_stop(update_display_timer_);
esp_timer_delete(update_display_timer_);
}
if (ws_client_ != nullptr) {
delete ws_client_;
}
if (opus_decoder_ != nullptr) {
opus_decoder_destroy(opus_decoder_);
}
if (audio_encode_task_stack_ != nullptr) {
free(audio_encode_task_stack_);
heap_caps_free(audio_encode_task_stack_);
}
if (main_loop_task_stack_ != nullptr) {
heap_caps_free(main_loop_task_stack_);
}
if (audio_device_ != nullptr) {
delete audio_device_;
}
vEventGroupDelete(event_group_);
@@ -57,6 +69,7 @@ Application::~Application() {
void Application::CheckNewVersion() {
// Check if there is a new firmware version available
firmware_upgrade_.SetBoardJson(Board::GetInstance().GetJson());
firmware_upgrade_.CheckVersion();
if (firmware_upgrade_.HasNewVersion()) {
// Wait for the chat state to be idle
@@ -65,11 +78,9 @@ void Application::CheckNewVersion() {
}
SetChatState(kChatStateUpgrading);
firmware_upgrade_.StartUpgrade([this](int progress, size_t speed) {
#ifdef CONFIG_USE_DISPLAY
char buffer[64];
snprintf(buffer, sizeof(buffer), "Upgrading...\n %d%% %zuKB/s", progress, speed / 1024);
display_.SetText(buffer);
#endif
});
// If upgrade success, the device will reboot and never reach here
ESP_LOGI(TAG, "Firmware upgrade failed...");
@@ -79,138 +90,98 @@ void Application::CheckNewVersion() {
}
}
#ifdef CONFIG_USE_DISPLAY
void Application::Alert(const std::string&& title, const std::string&& message) {
ESP_LOGE(TAG, "Alert: %s, %s", title.c_str(), message.c_str());
display_.ShowNotification(std::string(title + "\n" + message));
#ifdef CONFIG_USE_ML307
static std::string csq_to_string(int csq) {
if (csq == -1) {
return "No network";
} else if (csq >= 0 && csq <= 9) {
return "Very bad";
} else if (csq >= 10 && csq <= 14) {
return "Bad";
} else if (csq >= 15 && csq <= 19) {
return "Fair";
} else if (csq >= 20 && csq <= 24) {
return "Good";
} else if (csq >= 25 && csq <= 31) {
return "Very good";
}
return "Invalid";
}
#else
static std::string rssi_to_string(int rssi) {
if (rssi >= -55) {
return "Very good";
} else if (rssi >= -65) {
return "Good";
} else if (rssi >= -75) {
return "Fair";
} else if (rssi >= -85) {
return "Poor";
} else {
return "No network";
if (message == "PIN is not ready") {
PlayLocalFile(p3_err_pin_start, p3_err_pin_end - p3_err_pin_start);
} else if (message == "Configuring WiFi") {
PlayLocalFile(p3_err_wificonfig_start, p3_err_wificonfig_end - p3_err_wificonfig_start);
} else if (message == "Registration denied") {
PlayLocalFile(p3_err_reg_start, p3_err_reg_end - p3_err_reg_start);
}
}
#endif
void Application::UpdateDisplay() {
while (true) {
if (chat_state_ == kChatStateIdle) {
#ifdef CONFIG_USE_ML307
std::string network_name = ml307_at_modem_.GetCarrierName();
int signal_quality = ml307_at_modem_.GetCsq();
if (signal_quality == -1) {
network_name = "No network";
} else {
ESP_LOGI(TAG, "%s CSQ: %d", network_name.c_str(), signal_quality);
display_.SetText(network_name + "\n" + csq_to_string(signal_quality) + " (" + std::to_string(signal_quality) + ")");
}
#else
auto& wifi_station = WifiStation::GetInstance();
int8_t rssi = wifi_station.GetRssi();
display_.SetText(wifi_station.GetSsid() + "\n" + rssi_to_string(rssi) + " (" + std::to_string(rssi) + ")");
#endif
}
vTaskDelay(pdMS_TO_TICKS(10 * 1000));
void Application::PlayLocalFile(const char* data, size_t size) {
ESP_LOGI(TAG, "PlayLocalFile: %zu bytes", size);
SetDecodeSampleRate(16000);
{
std::lock_guard<std::mutex> lock(mutex_);
auto packet = new AudioPacket();
packet->type = kAudioPacketTypeStart;
audio_decode_queue_.push_back(packet);
}
ParseBinaryProtocol3(data, size);
{
std::lock_guard<std::mutex> lock(mutex_);
auto packet = new AudioPacket();
packet->type = kAudioPacketTypeStop;
audio_decode_queue_.push_back(packet);
cv_.notify_all();
}
}
#endif
void Application::Start() {
auto& builtin_led = BuiltinLed::GetInstance();
#ifdef CONFIG_USE_ML307
builtin_led.SetBlue();
builtin_led.StartContinuousBlink(100);
ml307_at_modem_.SetDebug(false);
ml307_at_modem_.SetBaudRate(921600);
// Print the ML307 modem information
std::string module_name = ml307_at_modem_.GetModuleName();
ESP_LOGI(TAG, "ML307 Module: %s", module_name.c_str());
#ifdef CONFIG_USE_DISPLAY
display_.SetText(std::string("Wait for network\n") + module_name);
#endif
ml307_at_modem_.ResetConnections();
ml307_at_modem_.WaitForNetworkReady();
ESP_LOGI(TAG, "ML307 IMEI: %s", ml307_at_modem_.GetImei().c_str());
ESP_LOGI(TAG, "ML307 ICCID: %s", ml307_at_modem_.GetIccid().c_str());
auto& board = Board::GetInstance();
board.Initialize();
// If low power, the material ready event will be triggered by the modem because of a reset
ml307_at_modem_.OnMaterialReady([this]() {
ESP_LOGI(TAG, "ML307 material ready");
Schedule([this]() {
SetChatState(kChatStateIdle);
});
});
#else
// Try to connect to WiFi, if failed, launch the WiFi configuration AP
auto& wifi_station = WifiStation::GetInstance();
#ifdef CONFIG_USE_DISPLAY
display_.SetText(std::string("Connect to WiFi\n") + wifi_station.GetSsid());
#endif
builtin_led.SetBlue();
builtin_led.StartContinuousBlink(100);
wifi_station.Start();
if (!wifi_station.IsConnected()) {
builtin_led.SetBlue();
builtin_led.Blink(1000, 500);
auto& wifi_ap = WifiConfigurationAp::GetInstance();
wifi_ap.SetSsidPrefix("Xiaozhi");
#ifdef CONFIG_USE_DISPLAY
display_.SetText(wifi_ap.GetSsid() + "\n" + wifi_ap.GetWebServerUrl());
#endif
wifi_ap.Start();
return;
}
#endif
audio_device_.OnInputData([this](const int16_t* data, int size) {
audio_device_ = board.CreateAudioDevice();
audio_device_->Initialize();
audio_device_->EnableOutput(true);
audio_device_->EnableInput(true);
audio_device_->OnInputData([this](std::vector<int16_t>&& data) {
if (16000 != AUDIO_INPUT_SAMPLE_RATE) {
if (audio_device_->input_channels() == 2) {
auto left_channel = std::vector<int16_t>(data.size() / 2);
auto right_channel = std::vector<int16_t>(data.size() / 2);
for (size_t i = 0, j = 0; i < left_channel.size(); ++i, j += 2) {
left_channel[i] = data[j];
right_channel[i] = data[j + 1];
}
auto resampled_left = std::vector<int16_t>(input_resampler_.GetOutputSamples(left_channel.size()));
auto resampled_right = std::vector<int16_t>(input_resampler_.GetOutputSamples(right_channel.size()));
input_resampler_.Process(left_channel.data(), left_channel.size(), resampled_left.data());
input_resampler_.Process(right_channel.data(), right_channel.size(), resampled_right.data());
data.resize(resampled_left.size() + resampled_right.size());
for (size_t i = 0, j = 0; i < resampled_left.size(); ++i, j += 2) {
data[j] = resampled_left[i];
data[j + 1] = resampled_right[i];
}
} else {
auto resampled = std::vector<int16_t>(input_resampler_.GetOutputSamples(data.size()));
input_resampler_.Process(data.data(), data.size(), resampled.data());
data = std::move(resampled);
}
}
#ifdef CONFIG_USE_AFE_SR
if (audio_processor_.IsRunning()) {
audio_processor_.Input(data, size);
audio_processor_.Input(data);
}
if (wake_word_detect_.IsDetectionRunning()) {
wake_word_detect_.Feed(data, size);
wake_word_detect_.Feed(data);
}
#else
std::vector<int16_t> pcm(data, data + size);
Schedule([this, pcm = std::move(pcm)]() {
Schedule([this, data = std::move(data)]() {
if (chat_state_ == kChatStateListening) {
std::lock_guard<std::mutex> lock(mutex_);
audio_encode_queue_.emplace_back(std::move(pcm));
audio_encode_queue_.emplace_back(std::move(data));
cv_.notify_all();
}
});
#endif
});
// Initialize the audio device
audio_device_.Start(CONFIG_AUDIO_INPUT_SAMPLE_RATE, CONFIG_AUDIO_OUTPUT_SAMPLE_RATE);
// OPUS encoder / decoder use a lot of stack memory
const size_t opus_stack_size = 4096 * 8;
audio_encode_task_stack_ = (StackType_t*)malloc(opus_stack_size);
audio_encode_task_stack_ = (StackType_t*)heap_caps_malloc(opus_stack_size, MALLOC_CAP_SPIRAM);
audio_encode_task_ = xTaskCreateStatic([](void* arg) {
Application* app = (Application*)arg;
app->AudioEncodeTask();
@@ -221,9 +192,92 @@ void Application::Start() {
Application* app = (Application*)arg;
app->AudioPlayTask();
vTaskDelete(NULL);
}, "play_audio", 4096 * 4, this, 5, NULL);
}, "play_audio", 4096 * 4, this, 4, NULL);
board.StartNetwork();
// Blink the LED to indicate the device is running
builtin_led.SetGreen();
builtin_led.BlinkOnce();
boot_button_.OnClick([this]() {
Schedule([this]() {
if (chat_state_ == kChatStateIdle) {
SetChatState(kChatStateConnecting);
StartWebSocketClient();
if (ws_client_ && ws_client_->IsConnected()) {
opus_encoder_.ResetState();
#ifdef CONFIG_USE_AFE_SR
audio_processor_.Start();
#endif
SetChatState(kChatStateListening);
ESP_LOGI(TAG, "Communication started");
} else {
SetChatState(kChatStateIdle);
}
} else if (chat_state_ == kChatStateSpeaking) {
AbortSpeaking();
} else if (chat_state_ == kChatStateListening) {
if (ws_client_ && ws_client_->IsConnected()) {
ws_client_->Close();
}
}
});
});
volume_up_button_.OnClick([this]() {
Schedule([this]() {
auto volume = audio_device_->output_volume() + 10;
if (volume > 100) {
volume = 100;
}
audio_device_->SetOutputVolume(volume);
display_.ShowNotification("Volume\n" + std::to_string(volume));
});
});
volume_up_button_.OnLongPress([this]() {
Schedule([this]() {
audio_device_->SetOutputVolume(100);
display_.ShowNotification("Volume\n100");
});
});
volume_down_button_.OnClick([this]() {
Schedule([this]() {
auto volume = audio_device_->output_volume() - 10;
if (volume < 0) {
volume = 0;
}
audio_device_->SetOutputVolume(volume);
display_.ShowNotification("Volume\n" + std::to_string(volume));
});
});
volume_down_button_.OnLongPress([this]() {
Schedule([this]() {
audio_device_->SetOutputVolume(0);
display_.ShowNotification("Volume\n0");
});
});
const size_t main_loop_stack_size = 4096 * 2;
main_loop_task_stack_ = (StackType_t*)heap_caps_malloc(main_loop_stack_size, MALLOC_CAP_SPIRAM);
xTaskCreateStatic([](void* arg) {
Application* app = (Application*)arg;
app->MainLoop();
vTaskDelete(NULL);
}, "main_loop", main_loop_stack_size, this, 1, main_loop_task_stack_, &main_loop_task_buffer_);
// Launch a task to check for new firmware version
xTaskCreate([](void* arg) {
Application* app = (Application*)arg;
app->CheckNewVersion();
vTaskDelete(NULL);
}, "check_new_version", 4096 * 2, this, 1, NULL);
#ifdef CONFIG_USE_AFE_SR
wake_word_detect_.Initialize(audio_device_->input_channels(), audio_device_->input_reference());
wake_word_detect_.OnVadStateChange([this](bool speaking) {
Schedule([this, speaking]() {
auto& builtin_led = BuiltinLed::GetInstance();
@@ -263,7 +317,7 @@ void Application::Start() {
SetChatState(kChatStateIdle);
}
} else if (chat_state_ == kChatStateSpeaking) {
break_speaking_ = true;
AbortSpeaking();
}
// Resume detection
@@ -272,6 +326,7 @@ void Application::Start() {
});
wake_word_detect_.StartDetection();
audio_processor_.Initialize(audio_device_->input_channels(), audio_device_->input_reference());
audio_processor_.OnOutput([this](std::vector<int16_t>&& data) {
Schedule([this, data = std::move(data)]() {
if (chat_state_ == kChatStateListening) {
@@ -283,79 +338,8 @@ void Application::Start() {
});
#endif
// Blink the LED to indicate the device is running
builtin_led.SetGreen();
builtin_led.BlinkOnce();
boot_button_.OnClick([this]() {
Schedule([this]() {
if (chat_state_ == kChatStateIdle) {
SetChatState(kChatStateConnecting);
StartWebSocketClient();
if (ws_client_ && ws_client_->IsConnected()) {
opus_encoder_.ResetState();
#ifdef CONFIG_USE_AFE_SR
audio_processor_.Start();
#endif
SetChatState(kChatStateListening);
ESP_LOGI(TAG, "Communication started");
} else {
SetChatState(kChatStateIdle);
}
} else if (chat_state_ == kChatStateSpeaking) {
break_speaking_ = true;
} else if (chat_state_ == kChatStateListening) {
if (ws_client_ && ws_client_->IsConnected()) {
ws_client_->Close();
}
}
});
});
volume_up_button_.OnClick([this]() {
Schedule([this]() {
auto volume = audio_device_.output_volume() + 10;
if (volume > 100) {
volume = 0;
}
audio_device_.SetOutputVolume(volume);
#ifdef CONFIG_USE_DISPLAY
display_.ShowNotification("Volume\n" + std::to_string(volume));
#endif
});
});
volume_up_button_.OnLongPress([this]() {
Schedule([this]() {
audio_device_.SetOutputVolume(0);
#ifdef CONFIG_USE_DISPLAY
display_.ShowNotification("Volume\n0");
#endif
});
});
xTaskCreate([](void* arg) {
Application* app = (Application*)arg;
app->MainLoop();
vTaskDelete(NULL);
}, "main_loop", 4096 * 2, this, 5, NULL);
// Launch a task to check for new firmware version
xTaskCreate([](void* arg) {
Application* app = (Application*)arg;
app->CheckNewVersion();
vTaskDelete(NULL);
}, "check_new_version", 4096 * 2, this, 1, NULL);
#ifdef CONFIG_USE_DISPLAY
// Launch a task to update the display
xTaskCreate([](void* arg) {
Application* app = (Application*)arg;
app->UpdateDisplay();
vTaskDelete(NULL);
}, "update_display", 4096, this, 1, NULL);
#endif
SetChatState(kChatStateIdle);
display_.UpdateDisplay();
}
void Application::Schedule(std::function<void()> callback) {
@@ -380,8 +364,25 @@ void Application::MainLoop() {
}
}
void Application::AbortSpeaking() {
ESP_LOGI(TAG, "Abort speaking");
skip_to_end_ = true;
if (ws_client_ && ws_client_->IsConnected()) {
cJSON* root = cJSON_CreateObject();
cJSON_AddStringToObject(root, "type", "abort");
char* json = cJSON_PrintUnformatted(root);
std::lock_guard<std::mutex> lock(mutex_);
ws_client_->Send(json);
cJSON_Delete(root);
free(json);
}
}
void Application::SetChatState(ChatState state) {
const char* state_str[] = {
"unknown",
"idle",
"connecting",
"listening",
@@ -389,15 +390,21 @@ void Application::SetChatState(ChatState state) {
"wake_word_detected",
"testing",
"upgrading",
"unknown"
"invalid_state"
};
if (chat_state_ == state) {
// No need to update the state
return;
}
chat_state_ = state;
ESP_LOGI(TAG, "STATE: %s", state_str[chat_state_]);
auto& builtin_led = BuiltinLed::GetInstance();
switch (chat_state_) {
case kChatStateUnknown:
case kChatStateIdle:
builtin_led.TurnOff();
audio_device_->EnableOutput(false);
break;
case kChatStateConnecting:
builtin_led.SetBlue();
@@ -410,6 +417,7 @@ void Application::SetChatState(ChatState state) {
case kChatStateSpeaking:
builtin_led.SetGreen();
builtin_led.TurnOn();
audio_device_->EnableOutput(true);
break;
case kChatStateWakeWordDetected:
builtin_led.SetBlue();
@@ -434,25 +442,24 @@ void Application::SetChatState(ChatState state) {
}
}
BinaryProtocol* Application::AllocateBinaryProtocol(const uint8_t* payload, size_t payload_size) {
auto last_timestamp = 0;
auto protocol = (BinaryProtocol*)heap_caps_malloc(sizeof(BinaryProtocol) + payload_size, MALLOC_CAP_SPIRAM);
protocol->version = htons(PROTOCOL_VERSION);
protocol->type = htons(0);
BinaryProtocol3* Application::AllocateBinaryProtocol3(const uint8_t* payload, size_t payload_size) {
auto protocol = (BinaryProtocol3*)heap_caps_malloc(sizeof(BinaryProtocol3) + payload_size, MALLOC_CAP_SPIRAM);
protocol->type = 0;
protocol->reserved = 0;
protocol->timestamp = htonl(last_timestamp);
protocol->payload_size = htonl(payload_size);
assert(sizeof(BinaryProtocol) == 16);
protocol->payload_size = htons(payload_size);
assert(sizeof(BinaryProtocol3) == 4UL);
memcpy(protocol->payload, payload, payload_size);
return protocol;
}
void Application::AudioEncodeTask() {
ESP_LOGI(TAG, "Audio encode task started");
const int max_audio_play_queue_size_ = 2;
while (true) {
std::unique_lock<std::mutex> lock(mutex_);
cv_.wait(lock, [this]() {
return !audio_encode_queue_.empty() || !audio_decode_queue_.empty();
return !audio_encode_queue_.empty() || (!audio_decode_queue_.empty() && audio_play_queue_.size() < max_audio_play_queue_size_);
});
if (!audio_encode_queue_.empty()) {
@@ -462,10 +469,10 @@ void Application::AudioEncodeTask() {
// Encode audio data
opus_encoder_.Encode(pcm, [this](const uint8_t* opus, size_t opus_size) {
auto protocol = AllocateBinaryProtocol(opus, opus_size);
auto protocol = AllocateBinaryProtocol3(opus, opus_size);
Schedule([this, protocol, opus_size]() {
if (ws_client_ && ws_client_->IsConnected()) {
if (!ws_client_->Send(protocol, sizeof(BinaryProtocol) + opus_size, true)) {
if (!ws_client_->Send(protocol, sizeof(BinaryProtocol3) + opus_size, true)) {
ESP_LOGE(TAG, "Failed to send audio data");
}
}
@@ -477,21 +484,23 @@ void Application::AudioEncodeTask() {
audio_decode_queue_.pop_front();
lock.unlock();
int frame_size = opus_decode_sample_rate_ * opus_duration_ms_ / 1000;
packet->pcm.resize(frame_size);
if (packet->type == kAudioPacketTypeData && !skip_to_end_) {
int frame_size = opus_decode_sample_rate_ * opus_duration_ms_ / 1000;
packet->pcm.resize(frame_size);
int ret = opus_decode(opus_decoder_, packet->opus.data(), packet->opus.size(), packet->pcm.data(), frame_size, 0);
if (ret < 0) {
ESP_LOGE(TAG, "Failed to decode audio, error code: %d", ret);
delete packet;
continue;
}
int ret = opus_decode(opus_decoder_, packet->opus.data(), packet->opus.size(), packet->pcm.data(), frame_size, 0);
if (ret < 0) {
ESP_LOGE(TAG, "Failed to decode audio, error code: %d", ret);
delete packet;
continue;
}
if (opus_decode_sample_rate_ != CONFIG_AUDIO_OUTPUT_SAMPLE_RATE) {
int target_size = opus_resampler_.GetOutputSamples(frame_size);
std::vector<int16_t> resampled(target_size);
opus_resampler_.Process(packet->pcm.data(), frame_size, resampled.data());
packet->pcm = std::move(resampled);
if (opus_decode_sample_rate_ != AUDIO_OUTPUT_SAMPLE_RATE) {
int target_size = output_resampler_.GetOutputSamples(frame_size);
std::vector<int16_t> resampled(target_size);
output_resampler_.Process(packet->pcm.data(), frame_size, resampled.data());
packet->pcm = std::move(resampled);
}
}
std::lock_guard<std::mutex> lock(mutex_);
@@ -510,35 +519,32 @@ void Application::HandleAudioPacket(AudioPacket* packet) {
}
// This will block until the audio device has finished playing the audio
audio_device_.OutputData(packet->pcm);
if (break_speaking_) {
break_speaking_ = false;
skip_to_end_ = true;
// Play a silence and skip to the end
int frame_size = opus_decode_sample_rate_ / 1000 * opus_duration_ms_;
std::vector<int16_t> silence(frame_size);
bzero(silence.data(), silence.size() * sizeof(int16_t));
audio_device_.OutputData(silence);
}
audio_device_->OutputData(packet->pcm);
break;
}
case kAudioPacketTypeStart:
break_speaking_ = false;
skip_to_end_ = false;
Schedule([this]() {
SetChatState(kChatStateSpeaking);
});
break;
case kAudioPacketTypeStop:
skip_to_end_ = false;
Schedule([this]() {
SetChatState(kChatStateListening);
if (ws_client_ && ws_client_->IsConnected()) {
SetChatState(kChatStateListening);
} else {
SetChatState(kChatStateIdle);
}
});
break;
case kAudioPacketTypeSentenceStart:
ESP_LOGI(TAG, "<< %s", packet->text.c_str());
break;
case kAudioPacketTypeSentenceEnd:
if (break_speaking_) {
skip_to_end_ = true;
}
break;
default:
ESP_LOGI(TAG, "Unknown packet type: %d", packet->type);
@@ -558,6 +564,7 @@ void Application::AudioPlayTask() {
});
auto packet = std::move(audio_play_queue_.front());
audio_play_queue_.pop_front();
cv_.notify_all();
lock.unlock();
HandleAudioPacket(packet);
@@ -572,9 +579,26 @@ void Application::SetDecodeSampleRate(int sample_rate) {
opus_decoder_destroy(opus_decoder_);
opus_decode_sample_rate_ = sample_rate;
opus_decoder_ = opus_decoder_create(opus_decode_sample_rate_, 1, NULL);
if (opus_decode_sample_rate_ != CONFIG_AUDIO_OUTPUT_SAMPLE_RATE) {
ESP_LOGI(TAG, "Resampling audio from %d to %d", opus_decode_sample_rate_, CONFIG_AUDIO_OUTPUT_SAMPLE_RATE);
opus_resampler_.Configure(opus_decode_sample_rate_, CONFIG_AUDIO_OUTPUT_SAMPLE_RATE);
if (opus_decode_sample_rate_ != AUDIO_OUTPUT_SAMPLE_RATE) {
ESP_LOGI(TAG, "Resampling audio from %d to %d", opus_decode_sample_rate_, AUDIO_OUTPUT_SAMPLE_RATE);
output_resampler_.Configure(opus_decode_sample_rate_, AUDIO_OUTPUT_SAMPLE_RATE);
}
}
void Application::ParseBinaryProtocol3(const char* data, size_t size) {
for (const char* p = data; p < data + size; ) {
auto protocol = (BinaryProtocol3*)p;
p += sizeof(BinaryProtocol3);
auto packet = new AudioPacket();
packet->type = kAudioPacketTypeData;
auto payload_size = ntohs(protocol->payload_size);
packet->opus.resize(payload_size);
memcpy(packet->opus.data(), protocol->payload, payload_size);
p += payload_size;
std::lock_guard<std::mutex> lock(mutex_);
audio_decode_queue_.push_back(packet);
}
}
@@ -586,18 +610,10 @@ void Application::StartWebSocketClient() {
std::string url = CONFIG_WEBSOCKET_URL;
std::string token = "Bearer " + std::string(CONFIG_WEBSOCKET_ACCESS_TOKEN);
#ifdef CONFIG_USE_ML307
ws_client_ = new WebSocket(new Ml307SslTransport(ml307_at_modem_, 0));
#else
if (url.find("wss://") == 0) {
ws_client_ = new WebSocket(new TlsTransport());
} else {
ws_client_ = new WebSocket(new TcpTransport());
}
#endif
ws_client_ = Board::GetInstance().CreateWebSocket();
ws_client_->SetHeader("Authorization", token.c_str());
ws_client_->SetHeader("Device-Id", SystemInfo::GetMacAddress().c_str());
ws_client_->SetHeader("Protocol-Version", std::to_string(PROTOCOL_VERSION).c_str());
ws_client_->SetHeader("Device-Id", SystemInfo::GetMacAddress().c_str());
ws_client_->OnConnected([this]() {
ESP_LOGI(TAG, "Websocket connected");
@@ -607,24 +623,14 @@ void Application::StartWebSocketClient() {
std::string message = "{";
message += "\"type\":\"hello\",";
message += "\"audio_params\":{";
message += "\"format\":\"opus\", \"sample_rate\":" + std::to_string(CONFIG_AUDIO_INPUT_SAMPLE_RATE) + ", \"channels\":1";
message += "\"format\":\"opus\", \"sample_rate\":16000, \"channels\":1";
message += "}}";
ws_client_->Send(message);
});
ws_client_->OnData([this](const char* data, size_t len, bool binary) {
if (binary) {
auto protocol = (BinaryProtocol*)data;
auto packet = new AudioPacket();
packet->type = kAudioPacketTypeData;
packet->timestamp = ntohl(protocol->timestamp);
auto payload_size = ntohl(protocol->payload_size);
packet->opus.resize(payload_size);
memcpy(packet->opus.data(), protocol->payload, payload_size);
std::lock_guard<std::mutex> lock(mutex_);
audio_decode_queue_.push_back(packet);
ParseBinaryProtocol3(data, len);
cv_.notify_all();
} else {
// Parse JSON data
@@ -640,6 +646,9 @@ void Application::StartWebSocketClient() {
if (sample_rate != NULL) {
SetDecodeSampleRate(sample_rate->valueint);
}
// If the device is speaking, we need to skip the last session
skip_to_end_ = true;
} else if (strcmp(state->valuestring, "stop") == 0) {
packet->type = kAudioPacketTypeStop;
} else if (strcmp(state->valuestring, "sentence_end") == 0) {

View File

@@ -1,13 +1,9 @@
#ifndef _APPLICATION_H_
#define _APPLICATION_H_
#include "AudioDevice.h"
#include <OpusEncoder.h>
#include <OpusResampler.h>
#include <WebSocket.h>
#include <Ml307AtModem.h>
#include <Ml307Http.h>
#include <EspHttp.h>
#include <opus.h>
#include <resampler_structs.h>
@@ -17,7 +13,9 @@
#include <list>
#include <condition_variable>
#include "AudioDevice.h"
#include "Display.h"
#include "Board.h"
#include "FirmwareUpgrade.h"
#ifdef CONFIG_USE_AFE_SR
@@ -30,13 +28,11 @@
#define DETECTION_RUNNING 1
#define COMMUNICATION_RUNNING 2
#define PROTOCOL_VERSION 2
struct BinaryProtocol {
uint16_t version;
uint16_t type;
uint32_t reserved;
uint32_t timestamp;
uint32_t payload_size;
#define PROTOCOL_VERSION 3
struct BinaryProtocol3 {
uint8_t type;
uint8_t reserved;
uint16_t payload_size;
uint8_t payload[];
} __attribute__((packed));
@@ -59,6 +55,7 @@ struct AudioPacket {
enum ChatState {
kChatStateUnknown,
kChatStateIdle,
kChatStateConnecting,
kChatStateListening,
@@ -75,7 +72,12 @@ public:
}
void Start();
ChatState GetChatState() const { return chat_state_; }
Display& GetDisplay() { return display_; }
void Schedule(std::function<void()> callback);
void SetChatState(ChatState state);
void Alert(const std::string&& title, const std::string&& message);
void AbortSpeaking();
// 删除拷贝构造函数和赋值运算符
Application(const Application&) = delete;
Application& operator=(const Application&) = delete;
@@ -86,29 +88,23 @@ private:
Button boot_button_;
Button volume_up_button_;
AudioDevice audio_device_;
Button volume_down_button_;
AudioDevice* audio_device_ = nullptr;
Display display_;
#ifdef CONFIG_USE_AFE_SR
WakeWordDetect wake_word_detect_;
AudioProcessor audio_processor_;
#endif
#ifdef CONFIG_USE_ML307
Ml307AtModem ml307_at_modem_;
Ml307Http http_;
#else
EspHttp http_;
#endif
FirmwareUpgrade firmware_upgrade_;
#ifdef CONFIG_USE_DISPLAY
Display display_;
#endif
std::mutex mutex_;
std::condition_variable_any cv_;
std::list<std::function<void()>> main_tasks_;
WebSocket* ws_client_ = nullptr;
EventGroupHandle_t event_group_;
volatile ChatState chat_state_ = kChatStateIdle;
volatile ChatState chat_state_ = kChatStateUnknown;
volatile bool break_speaking_ = false;
bool skip_to_end_ = false;
esp_timer_handle_t update_display_timer_ = nullptr;
// Audio encode / decode
TaskHandle_t audio_encode_task_ = nullptr;
@@ -122,25 +118,25 @@ private:
OpusDecoder* opus_decoder_ = nullptr;
int opus_duration_ms_ = 60;
int opus_decode_sample_rate_ = CONFIG_AUDIO_OUTPUT_SAMPLE_RATE;
OpusResampler opus_resampler_;
int opus_decode_sample_rate_ = AUDIO_OUTPUT_SAMPLE_RATE;
OpusResampler input_resampler_;
OpusResampler output_resampler_;
TaskHandle_t check_new_version_task_ = nullptr;
StaticTask_t check_new_version_task_buffer_;
StackType_t* check_new_version_task_stack_ = nullptr;
TaskHandle_t main_loop_task_ = nullptr;
StaticTask_t main_loop_task_buffer_;
StackType_t* main_loop_task_stack_ = nullptr;
void MainLoop();
void Schedule(std::function<void()> callback);
BinaryProtocol* AllocateBinaryProtocol(const uint8_t* payload, size_t payload_size);
BinaryProtocol3* AllocateBinaryProtocol3(const uint8_t* payload, size_t payload_size);
void ParseBinaryProtocol3(const char* data, size_t size);
void SetDecodeSampleRate(int sample_rate);
void SetChatState(ChatState state);
void StartWebSocketClient();
void CheckNewVersion();
void UpdateDisplay();
void AudioEncodeTask();
void AudioPlayTask();
void HandleAudioPacket(AudioPacket* packet);
void PlayLocalFile(const char* data, size_t size);
};
#endif // _APPLICATION_H_

View File

@@ -1,10 +1,15 @@
#include "AudioDevice.h"
#include "Board.h"
#include <esp_log.h>
#include <cstring>
#include <cmath>
#define TAG "AudioDevice"
AudioDevice::AudioDevice() {
AudioDevice::AudioDevice()
: input_sample_rate_(AUDIO_INPUT_SAMPLE_RATE),
output_sample_rate_(AUDIO_OUTPUT_SAMPLE_RATE) {
}
AudioDevice::~AudioDevice() {
@@ -19,26 +24,16 @@ AudioDevice::~AudioDevice() {
}
}
void AudioDevice::Start(int input_sample_rate, int output_sample_rate) {
input_sample_rate_ = input_sample_rate;
output_sample_rate_ = output_sample_rate;
#ifdef CONFIG_AUDIO_DEVICE_I2S_SIMPLEX
CreateSimplexChannels();
void AudioDevice::Initialize() {
#ifdef AUDIO_I2S_METHOD_SIMPLEX
CreateSimplexChannels();
#else
CreateDuplexChannels();
CreateDuplexChannels();
#endif
ESP_ERROR_CHECK(i2s_channel_enable(tx_handle_));
ESP_ERROR_CHECK(i2s_channel_enable(rx_handle_));
xTaskCreate([](void* arg) {
auto audio_device = (AudioDevice*)arg;
audio_device->InputTask();
}, "audio_input", 4096 * 2, this, 5, &audio_input_task_);
}
void AudioDevice::CreateDuplexChannels() {
#ifndef AUDIO_I2S_METHOD_SIMPLEX
duplex_ = true;
i2s_chan_config_t chan_cfg = {
@@ -73,10 +68,10 @@ void AudioDevice::CreateDuplexChannels() {
},
.gpio_cfg = {
.mclk = I2S_GPIO_UNUSED,
.bclk = (gpio_num_t)CONFIG_AUDIO_DEVICE_I2S_MIC_GPIO_BCLK,
.ws = (gpio_num_t)CONFIG_AUDIO_DEVICE_I2S_MIC_GPIO_WS,
.dout = (gpio_num_t)CONFIG_AUDIO_DEVICE_I2S_SPK_GPIO_DOUT,
.din = (gpio_num_t)CONFIG_AUDIO_DEVICE_I2S_MIC_GPIO_DIN,
.bclk = (gpio_num_t)AUDIO_I2S_GPIO_BCLK,
.ws = (gpio_num_t)AUDIO_I2S_GPIO_LRCK,
.dout = (gpio_num_t)AUDIO_I2S_GPIO_DOUT,
.din = (gpio_num_t)AUDIO_I2S_GPIO_DIN,
.invert_flags = {
.mclk_inv = false,
.bclk_inv = false,
@@ -86,18 +81,21 @@ void AudioDevice::CreateDuplexChannels() {
};
ESP_ERROR_CHECK(i2s_channel_init_std_mode(tx_handle_, &std_cfg));
ESP_ERROR_CHECK(i2s_channel_init_std_mode(rx_handle_, &std_cfg));
ESP_ERROR_CHECK(i2s_channel_enable(tx_handle_));
ESP_ERROR_CHECK(i2s_channel_enable(rx_handle_));
ESP_LOGI(TAG, "Duplex channels created");
#endif
}
#ifdef CONFIG_AUDIO_DEVICE_I2S_SIMPLEX
void AudioDevice::CreateSimplexChannels() {
#ifdef AUDIO_I2S_METHOD_SIMPLEX
// Create a new channel for speaker
i2s_chan_config_t chan_cfg = {
.id = I2S_NUM_0,
.role = I2S_ROLE_MASTER,
.dma_desc_num = 6,
.dma_frame_num = 240,
.auto_clear_after_cb = false,
.auto_clear_after_cb = true,
.auto_clear_before_cb = false,
.intr_priority = 0,
};
@@ -124,9 +122,9 @@ void AudioDevice::CreateSimplexChannels() {
},
.gpio_cfg = {
.mclk = I2S_GPIO_UNUSED,
.bclk = (gpio_num_t)CONFIG_AUDIO_DEVICE_I2S_SPK_GPIO_BCLK,
.ws = (gpio_num_t)CONFIG_AUDIO_DEVICE_I2S_SPK_GPIO_WS,
.dout = (gpio_num_t)CONFIG_AUDIO_DEVICE_I2S_SPK_GPIO_DOUT,
.bclk = (gpio_num_t)AUDIO_I2S_SPK_GPIO_BCLK,
.ws = (gpio_num_t)AUDIO_I2S_SPK_GPIO_LRCK,
.dout = (gpio_num_t)AUDIO_I2S_SPK_GPIO_DOUT,
.din = I2S_GPIO_UNUSED,
.invert_flags = {
.mclk_inv = false,
@@ -141,48 +139,67 @@ void AudioDevice::CreateSimplexChannels() {
chan_cfg.id = I2S_NUM_1;
ESP_ERROR_CHECK(i2s_new_channel(&chan_cfg, nullptr, &rx_handle_));
std_cfg.clk_cfg.sample_rate_hz = (uint32_t)input_sample_rate_;
std_cfg.gpio_cfg.bclk = (gpio_num_t)CONFIG_AUDIO_DEVICE_I2S_MIC_GPIO_BCLK;
std_cfg.gpio_cfg.ws = (gpio_num_t)CONFIG_AUDIO_DEVICE_I2S_MIC_GPIO_WS;
std_cfg.gpio_cfg.bclk = (gpio_num_t)AUDIO_I2S_MIC_GPIO_SCK;
std_cfg.gpio_cfg.ws = (gpio_num_t)AUDIO_I2S_MIC_GPIO_WS;
std_cfg.gpio_cfg.dout = I2S_GPIO_UNUSED;
std_cfg.gpio_cfg.din = (gpio_num_t)CONFIG_AUDIO_DEVICE_I2S_MIC_GPIO_DIN;
std_cfg.gpio_cfg.din = (gpio_num_t)AUDIO_I2S_MIC_GPIO_DIN;
ESP_ERROR_CHECK(i2s_channel_init_std_mode(rx_handle_, &std_cfg));
ESP_LOGI(TAG, "Simplex channels created");
}
#endif
void AudioDevice::Write(const int16_t* data, int samples) {
ESP_ERROR_CHECK(i2s_channel_enable(tx_handle_));
ESP_ERROR_CHECK(i2s_channel_enable(rx_handle_));
ESP_LOGI(TAG, "Simplex channels created");
#endif
}
int AudioDevice::Write(const int16_t* data, int samples) {
int32_t buffer[samples];
// output_volume_: 0-100
// volume_factor_: 0-65536
int32_t volume_factor = pow(double(output_volume_) / 100.0, 2) * 65536;
for (int i = 0; i < samples; i++) {
buffer[i] = int32_t(data[i]) * volume_factor;
int64_t temp = int64_t(data[i]) * volume_factor; // 使用 int64_t 进行乘法运算
if (temp > INT32_MAX) {
buffer[i] = INT32_MAX;
} else if (temp < INT32_MIN) {
buffer[i] = INT32_MIN;
} else {
buffer[i] = static_cast<int32_t>(temp);
}
}
size_t bytes_written;
ESP_ERROR_CHECK(i2s_channel_write(tx_handle_, buffer, samples * sizeof(int32_t), &bytes_written, portMAX_DELAY));
return bytes_written / sizeof(int32_t);
}
int AudioDevice::Read(int16_t* dest, int samples) {
size_t bytes_read;
int32_t bit32_buffer_[samples];
if (i2s_channel_read(rx_handle_, bit32_buffer_, samples * sizeof(int32_t), &bytes_read, portMAX_DELAY) != ESP_OK) {
int32_t bit32_buffer[samples];
if (i2s_channel_read(rx_handle_, bit32_buffer, samples * sizeof(int32_t), &bytes_read, portMAX_DELAY) != ESP_OK) {
ESP_LOGE(TAG, "Read Failed!");
return 0;
}
samples = bytes_read / sizeof(int32_t);
for (int i = 0; i < samples; i++) {
int32_t value = bit32_buffer_[i] >> 12;
int32_t value = bit32_buffer[i] >> 12;
dest[i] = (value > INT16_MAX) ? INT16_MAX : (value < -INT16_MAX) ? -INT16_MAX : (int16_t)value;
}
return samples;
}
void AudioDevice::OnInputData(std::function<void(const int16_t*, int)> callback) {
void AudioDevice::OnInputData(std::function<void(std::vector<int16_t>&& data)> callback) {
on_input_data_ = callback;
// 创建音频输入任务
if (audio_input_task_ == nullptr) {
xTaskCreate([](void* arg) {
auto audio_device = (AudioDevice*)arg;
audio_device->InputTask();
}, "audio_input", 4096 * 2, this, 3, &audio_input_task_);
}
}
void AudioDevice::OutputData(std::vector<int16_t>& data) {
@@ -191,12 +208,14 @@ void AudioDevice::OutputData(std::vector<int16_t>& data) {
void AudioDevice::InputTask() {
int duration = 30;
int input_frame_size = input_sample_rate_ / 1000 * duration;
int16_t input_buffer[input_frame_size];
int input_frame_size = input_sample_rate_ / 1000 * duration * input_channels_;
while (true) {
int samples = Read(input_buffer, input_frame_size);
std::vector<int16_t> input_data(input_frame_size);
int samples = Read(input_data.data(), input_data.size());
if (samples > 0) {
on_input_data_(input_buffer, samples);
if (on_input_data_) {
on_input_data_(std::move(input_data));
}
}
}
}
@@ -205,3 +224,19 @@ void AudioDevice::SetOutputVolume(int volume) {
output_volume_ = volume;
ESP_LOGI(TAG, "Set output volume to %d", output_volume_);
}
void AudioDevice::EnableInput(bool enable) {
if (enable == input_enabled_) {
return;
}
input_enabled_ = enable;
ESP_LOGI(TAG, "Set input enable to %s", enable ? "true" : "false");
}
void AudioDevice::EnableOutput(bool enable) {
if (enable == output_enabled_) {
return;
}
output_enabled_ = enable;
ESP_LOGI(TAG, "Set output enable to %s", enable ? "true" : "false");
}

View File

@@ -2,7 +2,6 @@
#define _AUDIO_DEVICE_H
#include <freertos/FreeRTOS.h>
#include <freertos/event_groups.h>
#include <driver/i2s_std.h>
#include <vector>
@@ -12,35 +11,46 @@
class AudioDevice {
public:
AudioDevice();
~AudioDevice();
virtual ~AudioDevice();
virtual void Initialize();
void Start(int input_sample_rate, int output_sample_rate);
void OnInputData(std::function<void(const int16_t*, int)> callback);
void OnInputData(std::function<void(std::vector<int16_t>&& data)> callback);
void OutputData(std::vector<int16_t>& data);
void SetOutputVolume(int volume);
virtual void SetOutputVolume(int volume);
virtual void EnableInput(bool enable);
virtual void EnableOutput(bool enable);
inline bool duplex() const { return duplex_; }
inline bool input_reference() const { return input_reference_; }
inline int input_sample_rate() const { return input_sample_rate_; }
inline int output_sample_rate() const { return output_sample_rate_; }
inline int input_channels() const { return input_channels_; }
inline int output_channels() const { return output_channels_; }
inline int output_volume() const { return output_volume_; }
int input_sample_rate() const { return input_sample_rate_; }
int output_sample_rate() const { return output_sample_rate_; }
bool duplex() const { return duplex_; }
int output_volume() const { return output_volume_; }
private:
TaskHandle_t audio_input_task_ = nullptr;
std::function<void(std::vector<int16_t>&& data)> on_input_data_;
void InputTask();
void CreateSimplexChannels();
protected:
bool duplex_ = false;
bool input_reference_ = false;
bool input_enabled_ = false;
bool output_enabled_ = false;
int input_sample_rate_ = 0;
int output_sample_rate_ = 0;
int output_volume_ = 80;
int input_channels_ = 1;
int output_channels_ = 1;
int output_volume_ = 70;
i2s_chan_handle_t tx_handle_ = nullptr;
i2s_chan_handle_t rx_handle_ = nullptr;
TaskHandle_t audio_input_task_ = nullptr;
EventGroupHandle_t event_group_;
std::function<void(const int16_t*, int)> on_input_data_;
void CreateDuplexChannels();
void CreateSimplexChannels();
void InputTask();
int Read(int16_t* dest, int samples);
void Write(const int16_t* data, int samples);
virtual void CreateDuplexChannels();
virtual int Read(int16_t* dest, int samples);
virtual int Write(const int16_t* data, int samples);
};
#endif // _AUDIO_DEVICE_H

View File

@@ -8,6 +8,12 @@ static const char* TAG = "AudioProcessor";
AudioProcessor::AudioProcessor()
: afe_communication_data_(nullptr) {
event_group_ = xEventGroupCreate();
}
void AudioProcessor::Initialize(int channels, bool reference) {
channels_ = channels;
reference_ = reference;
int ref_num = reference_ ? 1 : 0;
afe_config_t afe_config = {
.aec_init = false,
@@ -22,17 +28,17 @@ AudioProcessor::AudioProcessor()
.wakenet_model_name_2 = NULL,
.wakenet_mode = DET_MODE_90,
.afe_mode = SR_MODE_HIGH_PERF,
.afe_perferred_core = 0,
.afe_perferred_priority = 5,
.afe_perferred_core = 1,
.afe_perferred_priority = 1,
.afe_ringbuf_size = 50,
.memory_alloc_mode = AFE_MEMORY_ALLOC_MORE_PSRAM,
.afe_linear_gain = 1.0,
.agc_mode = AFE_MN_PEAK_AGC_MODE_2,
.pcm_config = {
.total_ch_num = 1,
.mic_num = 1,
.ref_num = 0,
.sample_rate = CONFIG_AUDIO_INPUT_SAMPLE_RATE,
.total_ch_num = channels_,
.mic_num = channels_ - ref_num,
.ref_num = ref_num,
.sample_rate = 16000,
},
.debug_init = false,
.debug_hook = {{ AFE_DEBUG_HOOK_MASE_TASK_IN, NULL }, { AFE_DEBUG_HOOK_FETCH_TASK_IN, NULL }},
@@ -47,7 +53,7 @@ AudioProcessor::AudioProcessor()
auto this_ = (AudioProcessor*)arg;
this_->AudioProcessorTask();
vTaskDelete(NULL);
}, "audio_communication", 4096 * 2, this, 5, NULL);
}, "audio_communication", 4096 * 2, this, 1, NULL);
}
AudioProcessor::~AudioProcessor() {
@@ -57,10 +63,10 @@ AudioProcessor::~AudioProcessor() {
vEventGroupDelete(event_group_);
}
void AudioProcessor::Input(const int16_t* data, int size) {
input_buffer_.insert(input_buffer_.end(), data, data + size);
void AudioProcessor::Input(std::vector<int16_t>& data) {
input_buffer_.insert(input_buffer_.end(), data.begin(), data.end());
auto chunk_size = esp_afe_vc_v1.get_feed_chunksize(afe_communication_data_);
auto chunk_size = esp_afe_vc_v1.get_feed_chunksize(afe_communication_data_) * channels_;
while (input_buffer_.size() >= chunk_size) {
auto chunk = input_buffer_.data();
esp_afe_vc_v1.feed(afe_communication_data_, chunk);
@@ -92,6 +98,9 @@ void AudioProcessor::AudioProcessorTask() {
xEventGroupWaitBits(event_group_, PROCESSOR_RUNNING, pdFALSE, pdTRUE, portMAX_DELAY);
auto res = esp_afe_vc_v1.fetch(afe_communication_data_);
if ((xEventGroupGetBits(event_group_) & PROCESSOR_RUNNING) == 0) {
continue;
}
if (res == nullptr || res->ret_value == ESP_FAIL) {
if (res != nullptr) {
ESP_LOGI(TAG, "Error code: %d", res->ret_value);

View File

@@ -15,7 +15,8 @@ public:
AudioProcessor();
~AudioProcessor();
void Input(const int16_t* data, int size);
void Initialize(int channels, bool reference);
void Input(std::vector<int16_t>& data);
void Start();
void Stop();
bool IsRunning();
@@ -26,6 +27,8 @@ private:
esp_afe_sr_data_t* afe_communication_data_ = nullptr;
std::vector<int16_t> input_buffer_;
std::function<void(std::vector<int16_t>&& data)> output_callback_;
int channels_;
bool reference_;
void AudioProcessorTask();
};

8
main/Board.cc Normal file
View File

@@ -0,0 +1,8 @@
#include "Board.h"
#include <esp_log.h>
// static const char *TAG = "Board";
bool Board::GetBatteryVoltage(int &voltage, bool& charging) {
return false;
}

45
main/Board.h Normal file
View File

@@ -0,0 +1,45 @@
#ifndef BOARD_H
#define BOARD_H
#include "config.h"
#include <Http.h>
#include <WebSocket.h>
#include <AudioDevice.h>
#include <string>
void* create_board();
class Board {
public:
static Board& GetInstance() {
static Board* instance = nullptr;
if (nullptr == instance) {
instance = static_cast<Board*>(create_board());
}
return *instance;
}
virtual void Initialize() = 0;
virtual void StartNetwork() = 0;
virtual ~Board() = default;
virtual AudioDevice* CreateAudioDevice() = 0;
virtual Http* CreateHttp() = 0;
virtual WebSocket* CreateWebSocket() = 0;
virtual bool GetNetworkState(std::string& network_name, int& signal_quality, std::string& signal_quality_text) = 0;
virtual bool GetBatteryVoltage(int &voltage, bool& charging);
virtual std::string GetJson() = 0;
protected:
Board() = default;
private:
Board(const Board&) = delete; // 禁用拷贝构造函数
Board& operator=(const Board&) = delete; // 禁用赋值操作
};
#define DECLARE_BOARD(BOARD_CLASS_NAME) \
void* create_board() { \
return new BOARD_CLASS_NAME(); \
}
#endif // BOARD_H

282
main/BoxAudioDevice.cc Normal file
View File

@@ -0,0 +1,282 @@
#include "BoxAudioDevice.h"
#include "Board.h"
#include <esp_log.h>
#include <cassert>
static const char* TAG = "BoxAudioDevice";
BoxAudioDevice::BoxAudioDevice() {
}
BoxAudioDevice::~BoxAudioDevice() {
ESP_ERROR_CHECK(esp_codec_dev_close(output_dev_));
esp_codec_dev_delete(output_dev_);
ESP_ERROR_CHECK(esp_codec_dev_close(input_dev_));
esp_codec_dev_delete(input_dev_);
audio_codec_delete_codec_if(in_codec_if_);
audio_codec_delete_ctrl_if(in_ctrl_if_);
audio_codec_delete_codec_if(out_codec_if_);
audio_codec_delete_ctrl_if(out_ctrl_if_);
audio_codec_delete_gpio_if(gpio_if_);
audio_codec_delete_data_if(data_if_);
ESP_ERROR_CHECK(i2c_del_master_bus(i2c_master_handle_));
}
void BoxAudioDevice::Initialize() {
duplex_ = true; // 是否双工
input_reference_ = AUDIO_INPUT_REFERENCE; // 是否使用参考输入,实现回声消除
input_channels_ = input_reference_ ? 2 : 1; // 输入通道数
// Initialize I2C peripheral
i2c_master_bus_config_t i2c_bus_cfg = {
.i2c_port = I2C_NUM_1,
.sda_io_num = (gpio_num_t)AUDIO_CODEC_I2C_SDA_PIN,
.scl_io_num = (gpio_num_t)AUDIO_CODEC_I2C_SCL_PIN,
.clk_source = I2C_CLK_SRC_DEFAULT,
.glitch_ignore_cnt = 7,
.intr_priority = 0,
.trans_queue_depth = 0,
.flags = {
.enable_internal_pullup = 1,
},
};
ESP_ERROR_CHECK(i2c_new_master_bus(&i2c_bus_cfg, &i2c_master_handle_));
CreateDuplexChannels();
#ifdef AUDIO_CODEC_USE_PCA9557
// Initialize PCA9557
i2c_device_config_t pca9557_cfg = {
.dev_addr_length = I2C_ADDR_BIT_LEN_7,
.device_address = 0x19,
.scl_speed_hz = 400000,
.scl_wait_us = 0,
.flags = {
.disable_ack_check = 0,
},
};
i2c_master_dev_handle_t pca9557_handle;
ESP_ERROR_CHECK(i2c_master_bus_add_device(i2c_master_handle_, &pca9557_cfg, &pca9557_handle));
assert(pca9557_handle != NULL);
auto pca9557_set_register = [](i2c_master_dev_handle_t pca9557_handle, uint8_t data_addr, uint8_t data) {
uint8_t data_[2] = {data_addr, data};
ESP_ERROR_CHECK(i2c_master_transmit(pca9557_handle, data_, 2, 50));
};
pca9557_set_register(pca9557_handle, 0x03, 0xfd);
pca9557_set_register(pca9557_handle, 0x01, 0x02);
#endif
// Do initialize of related interface: data_if, ctrl_if and gpio_if
audio_codec_i2s_cfg_t i2s_cfg = {
.port = I2S_NUM_0,
.rx_handle = rx_handle_,
.tx_handle = tx_handle_,
};
data_if_ = audio_codec_new_i2s_data(&i2s_cfg);
assert(data_if_ != NULL);
// Output
audio_codec_i2c_cfg_t i2c_cfg = {
.port = I2C_NUM_1,
.addr = AUDIO_CODEC_ES8311_ADDR,
.bus_handle = i2c_master_handle_,
};
out_ctrl_if_ = audio_codec_new_i2c_ctrl(&i2c_cfg);
assert(out_ctrl_if_ != NULL);
gpio_if_ = audio_codec_new_gpio();
assert(gpio_if_ != NULL);
es8311_codec_cfg_t es8311_cfg = {};
es8311_cfg.ctrl_if = out_ctrl_if_;
es8311_cfg.gpio_if = gpio_if_;
es8311_cfg.codec_mode = ESP_CODEC_DEV_WORK_MODE_DAC;
es8311_cfg.pa_pin = AUDIO_CODEC_PA_PIN;
es8311_cfg.use_mclk = true;
es8311_cfg.hw_gain.pa_voltage = 5.0;
es8311_cfg.hw_gain.codec_dac_voltage = 3.3;
out_codec_if_ = es8311_codec_new(&es8311_cfg);
assert(out_codec_if_ != NULL);
esp_codec_dev_cfg_t dev_cfg = {
.dev_type = ESP_CODEC_DEV_TYPE_OUT,
.codec_if = out_codec_if_,
.data_if = data_if_,
};
output_dev_ = esp_codec_dev_new(&dev_cfg);
assert(output_dev_ != NULL);
// Input
i2c_cfg.addr = AUDIO_CODEC_ES7210_ADDR;
in_ctrl_if_ = audio_codec_new_i2c_ctrl(&i2c_cfg);
assert(in_ctrl_if_ != NULL);
es7210_codec_cfg_t es7210_cfg = {};
es7210_cfg.ctrl_if = in_ctrl_if_;
es7210_cfg.mic_selected = ES7120_SEL_MIC1 | ES7120_SEL_MIC2 | ES7120_SEL_MIC3 | ES7120_SEL_MIC4;
in_codec_if_ = es7210_codec_new(&es7210_cfg);
assert(in_codec_if_ != NULL);
dev_cfg.dev_type = ESP_CODEC_DEV_TYPE_IN;
dev_cfg.codec_if = in_codec_if_;
input_dev_ = esp_codec_dev_new(&dev_cfg);
assert(input_dev_ != NULL);
ESP_LOGI(TAG, "BoxAudioDevice initialized");
}
void BoxAudioDevice::CreateDuplexChannels() {
assert(input_sample_rate_ == output_sample_rate_);
i2s_chan_config_t chan_cfg = {
.id = I2S_NUM_0,
.role = I2S_ROLE_MASTER,
.dma_desc_num = 6,
.dma_frame_num = 240,
.auto_clear_after_cb = true,
.auto_clear_before_cb = false,
.intr_priority = 0,
};
ESP_ERROR_CHECK(i2s_new_channel(&chan_cfg, &tx_handle_, &rx_handle_));
i2s_std_config_t std_cfg = {
.clk_cfg = {
.sample_rate_hz = (uint32_t)output_sample_rate_,
.clk_src = I2S_CLK_SRC_DEFAULT,
.ext_clk_freq_hz = 0,
.mclk_multiple = I2S_MCLK_MULTIPLE_256
},
.slot_cfg = {
.data_bit_width = I2S_DATA_BIT_WIDTH_16BIT,
.slot_bit_width = I2S_SLOT_BIT_WIDTH_AUTO,
.slot_mode = I2S_SLOT_MODE_STEREO,
.slot_mask = I2S_STD_SLOT_BOTH,
.ws_width = I2S_DATA_BIT_WIDTH_16BIT,
.ws_pol = false,
.bit_shift = true,
.left_align = true,
.big_endian = false,
.bit_order_lsb = false
},
.gpio_cfg = {
.mclk = (gpio_num_t)AUDIO_I2S_GPIO_MCLK,
.bclk = (gpio_num_t)AUDIO_I2S_GPIO_BCLK,
.ws = (gpio_num_t)AUDIO_I2S_GPIO_LRCK,
.dout = (gpio_num_t)AUDIO_I2S_GPIO_DOUT,
.din = I2S_GPIO_UNUSED,
.invert_flags = {
.mclk_inv = false,
.bclk_inv = false,
.ws_inv = false
}
}
};
i2s_tdm_config_t tdm_cfg = {
.clk_cfg = {
.sample_rate_hz = (uint32_t)input_sample_rate_,
.clk_src = I2S_CLK_SRC_DEFAULT,
.ext_clk_freq_hz = 0,
.mclk_multiple = I2S_MCLK_MULTIPLE_256,
.bclk_div = 8,
},
.slot_cfg = {
.data_bit_width = I2S_DATA_BIT_WIDTH_16BIT,
.slot_bit_width = I2S_SLOT_BIT_WIDTH_AUTO,
.slot_mode = I2S_SLOT_MODE_STEREO,
.slot_mask = i2s_tdm_slot_mask_t(I2S_TDM_SLOT0 | I2S_TDM_SLOT1 | I2S_TDM_SLOT2 | I2S_TDM_SLOT3),
.ws_width = I2S_TDM_AUTO_WS_WIDTH,
.ws_pol = false,
.bit_shift = true,
.left_align = false,
.big_endian = false,
.bit_order_lsb = false,
.skip_mask = false,
.total_slot = I2S_TDM_AUTO_SLOT_NUM
},
.gpio_cfg = {
.mclk = (gpio_num_t)AUDIO_I2S_GPIO_MCLK,
.bclk = (gpio_num_t)AUDIO_I2S_GPIO_BCLK,
.ws = (gpio_num_t)AUDIO_I2S_GPIO_LRCK,
.dout = I2S_GPIO_UNUSED,
.din = (gpio_num_t)AUDIO_I2S_GPIO_DIN,
.invert_flags = {
.mclk_inv = false,
.bclk_inv = false,
.ws_inv = false
}
}
};
ESP_ERROR_CHECK(i2s_channel_init_std_mode(tx_handle_, &std_cfg));
ESP_ERROR_CHECK(i2s_channel_init_tdm_mode(rx_handle_, &tdm_cfg));
ESP_ERROR_CHECK(i2s_channel_enable(tx_handle_));
ESP_ERROR_CHECK(i2s_channel_enable(rx_handle_));
ESP_LOGI(TAG, "Duplex channels created");
}
int BoxAudioDevice::Read(int16_t *buffer, int samples) {
if (input_enabled_) {
ESP_ERROR_CHECK_WITHOUT_ABORT(esp_codec_dev_read(input_dev_, (void*)buffer, samples * sizeof(int16_t)));
}
return samples;
}
int BoxAudioDevice::Write(const int16_t *buffer, int samples) {
if (output_enabled_) {
ESP_ERROR_CHECK_WITHOUT_ABORT(esp_codec_dev_write(output_dev_, (void*)buffer, samples * sizeof(int16_t)));
}
return samples;
}
void BoxAudioDevice::SetOutputVolume(int volume) {
ESP_ERROR_CHECK(esp_codec_dev_set_out_vol(output_dev_, volume));
AudioDevice::SetOutputVolume(volume);
}
void BoxAudioDevice::EnableInput(bool enable) {
if (enable == input_enabled_) {
return;
}
if (enable) {
esp_codec_dev_sample_info_t fs = {
.bits_per_sample = 16,
.channel = 4,
.channel_mask = ESP_CODEC_DEV_MAKE_CHANNEL_MASK(0),
.sample_rate = (uint32_t)output_sample_rate_,
.mclk_multiple = 0,
};
if (input_reference_) {
fs.channel_mask |= ESP_CODEC_DEV_MAKE_CHANNEL_MASK(1);
}
ESP_ERROR_CHECK(esp_codec_dev_open(input_dev_, &fs));
ESP_ERROR_CHECK(esp_codec_dev_set_in_channel_gain(input_dev_, ESP_CODEC_DEV_MAKE_CHANNEL_MASK(0), 30.0));
} else {
ESP_ERROR_CHECK(esp_codec_dev_close(input_dev_));
}
AudioDevice::EnableInput(enable);
}
void BoxAudioDevice::EnableOutput(bool enable) {
if (enable == output_enabled_) {
return;
}
if (enable) {
// Play 16bit 1 channel
esp_codec_dev_sample_info_t fs = {
.bits_per_sample = 16,
.channel = 1,
.channel_mask = 0,
.sample_rate = (uint32_t)output_sample_rate_,
.mclk_multiple = 0,
};
ESP_ERROR_CHECK(esp_codec_dev_open(output_dev_, &fs));
ESP_ERROR_CHECK(esp_codec_dev_set_out_vol(output_dev_, output_volume_));
} else {
ESP_ERROR_CHECK(esp_codec_dev_close(output_dev_));
}
AudioDevice::EnableOutput(enable);
}

38
main/BoxAudioDevice.h Normal file
View File

@@ -0,0 +1,38 @@
#ifndef _BOX_AUDIO_DEVICE_H
#define _BOX_AUDIO_DEVICE_H
#include "AudioDevice.h"
#include <driver/i2c_master.h>
#include <driver/i2s_tdm.h>
#include <esp_codec_dev.h>
#include <esp_codec_dev_defaults.h>
class BoxAudioDevice : public AudioDevice {
public:
BoxAudioDevice();
virtual ~BoxAudioDevice();
virtual void Initialize() override;
virtual void SetOutputVolume(int volume) override;
virtual void EnableInput(bool enable) override;
virtual void EnableOutput(bool enable) override;
private:
i2c_master_bus_handle_t i2c_master_handle_ = nullptr;
const audio_codec_data_if_t* data_if_ = nullptr;
const audio_codec_ctrl_if_t* out_ctrl_if_ = nullptr;
const audio_codec_if_t* out_codec_if_ = nullptr;
const audio_codec_ctrl_if_t* in_ctrl_if_ = nullptr;
const audio_codec_if_t* in_codec_if_ = nullptr;
const audio_codec_gpio_if_t* gpio_if_ = nullptr;
esp_codec_dev_handle_t output_dev_ = nullptr;
esp_codec_dev_handle_t input_dev_ = nullptr;
void CreateDuplexChannels() override;
int Read(int16_t* dest, int samples) override;
int Write(const int16_t* data, int samples) override;
};
#endif // _BOX_AUDIO_DEVICE_H

137
main/BuiltinLed.cc Normal file
View File

@@ -0,0 +1,137 @@
#include "BuiltinLed.h"
#include "Board.h"
#include <cstring>
#include <esp_log.h>
#define TAG "builtin_led"
BuiltinLed::BuiltinLed() {
mutex_ = xSemaphoreCreateMutex();
blink_event_group_ = xEventGroupCreate();
xEventGroupSetBits(blink_event_group_, BLINK_TASK_STOPPED_BIT);
if (BUILTIN_LED_GPIO == GPIO_NUM_NC) {
ESP_LOGI(TAG, "Builtin LED not connected");
return;
}
Initialize();
SetGrey();
}
BuiltinLed::~BuiltinLed() {
StopBlinkInternal();
if (led_strip_ != nullptr) {
led_strip_del(led_strip_);
}
if (mutex_ != nullptr) {
vSemaphoreDelete(mutex_);
}
if (blink_event_group_ != nullptr) {
vEventGroupDelete(blink_event_group_);
}
}
BuiltinLed& BuiltinLed::GetInstance() {
static BuiltinLed instance;
return instance;
}
void BuiltinLed::Initialize() {
led_strip_config_t strip_config = {};
strip_config.strip_gpio_num = BUILTIN_LED_GPIO;
strip_config.max_leds = 1;
strip_config.led_pixel_format = LED_PIXEL_FORMAT_GRB;
strip_config.led_model = LED_MODEL_WS2812;
led_strip_rmt_config_t rmt_config = {};
rmt_config.resolution_hz = 10 * 1000 * 1000; // 10MHz
ESP_ERROR_CHECK(led_strip_new_rmt_device(&strip_config, &rmt_config, &led_strip_));
led_strip_clear(led_strip_);
}
void BuiltinLed::SetColor(uint8_t r, uint8_t g, uint8_t b) {
r_ = r;
g_ = g;
b_ = b;
}
void BuiltinLed::TurnOn() {
if (led_strip_ == nullptr) {
return;
}
StopBlinkInternal();
xSemaphoreTake(mutex_, portMAX_DELAY);
led_strip_set_pixel(led_strip_, 0, r_, g_, b_);
led_strip_refresh(led_strip_);
xSemaphoreGive(mutex_);
}
void BuiltinLed::TurnOff() {
if (led_strip_ == nullptr) {
return;
}
StopBlinkInternal();
xSemaphoreTake(mutex_, portMAX_DELAY);
led_strip_clear(led_strip_);
xSemaphoreGive(mutex_);
}
void BuiltinLed::BlinkOnce() {
Blink(1, 100);
}
void BuiltinLed::Blink(int times, int interval_ms) {
StartBlinkTask(times, interval_ms);
}
void BuiltinLed::StartContinuousBlink(int interval_ms) {
StartBlinkTask(BLINK_INFINITE, interval_ms);
}
void BuiltinLed::StartBlinkTask(int times, int interval_ms) {
if (led_strip_ == nullptr) {
return;
}
StopBlinkInternal();
xSemaphoreTake(mutex_, portMAX_DELAY);
blink_times_ = times;
blink_interval_ms_ = interval_ms;
should_blink_ = true;
xEventGroupClearBits(blink_event_group_, BLINK_TASK_STOPPED_BIT);
xEventGroupSetBits(blink_event_group_, BLINK_TASK_RUNNING_BIT);
xTaskCreate([](void* obj) {
auto this_ = static_cast<BuiltinLed*>(obj);
int count = 0;
while (this_->should_blink_ && (this_->blink_times_ == BLINK_INFINITE || count < this_->blink_times_)) {
xSemaphoreTake(this_->mutex_, portMAX_DELAY);
led_strip_set_pixel(this_->led_strip_, 0, this_->r_, this_->g_, this_->b_);
led_strip_refresh(this_->led_strip_);
xSemaphoreGive(this_->mutex_);
vTaskDelay(this_->blink_interval_ms_ / portTICK_PERIOD_MS);
if (!this_->should_blink_) break;
xSemaphoreTake(this_->mutex_, portMAX_DELAY);
led_strip_clear(this_->led_strip_);
xSemaphoreGive(this_->mutex_);
vTaskDelay(this_->blink_interval_ms_ / portTICK_PERIOD_MS);
if (this_->blink_times_ != BLINK_INFINITE) count++;
}
this_->blink_task_ = nullptr;
xEventGroupClearBits(this_->blink_event_group_, BLINK_TASK_RUNNING_BIT);
xEventGroupSetBits(this_->blink_event_group_, BLINK_TASK_STOPPED_BIT);
vTaskDelete(NULL);
}, "blink", 2048, this, tskIDLE_PRIORITY, &blink_task_);
xSemaphoreGive(mutex_);
}
void BuiltinLed::StopBlinkInternal() {
should_blink_ = false;
xEventGroupWaitBits(blink_event_group_, BLINK_TASK_STOPPED_BIT, pdFALSE, pdTRUE, portMAX_DELAY);
}

52
main/BuiltinLed.h Normal file
View File

@@ -0,0 +1,52 @@
#ifndef _BUILTIN_LED_H_
#define _BUILTIN_LED_H_
#include <led_strip.h>
#include <freertos/semphr.h>
#include <freertos/task.h>
#include <freertos/event_groups.h>
#include <atomic>
#define BLINK_INFINITE -1
#define BLINK_TASK_STOPPED_BIT BIT0
#define BLINK_TASK_RUNNING_BIT BIT1
#define DEFAULT_BRIGHTNESS 16
class BuiltinLed {
public:
static BuiltinLed& GetInstance();
void BlinkOnce();
void Blink(int times, int interval_ms);
void StartContinuousBlink(int interval_ms);
void TurnOn();
void TurnOff();
void SetColor(uint8_t r, uint8_t g, uint8_t b);
void SetWhite(uint8_t brightness = DEFAULT_BRIGHTNESS) { SetColor(brightness, brightness, brightness); }
void SetGrey(uint8_t brightness = DEFAULT_BRIGHTNESS) { SetColor(brightness, brightness, brightness); }
void SetRed(uint8_t brightness = DEFAULT_BRIGHTNESS) { SetColor(brightness, 0, 0); }
void SetGreen(uint8_t brightness = DEFAULT_BRIGHTNESS) { SetColor(0, brightness, 0); }
void SetBlue(uint8_t brightness = DEFAULT_BRIGHTNESS) { SetColor(0, 0, brightness); }
private:
BuiltinLed();
~BuiltinLed();
BuiltinLed(const BuiltinLed&) = delete;
BuiltinLed& operator=(const BuiltinLed&) = delete;
SemaphoreHandle_t mutex_;
EventGroupHandle_t blink_event_group_;
TaskHandle_t blink_task_ = nullptr;
led_strip_handle_t led_strip_ = nullptr;
uint8_t r_ = 0, g_ = 0, b_ = 0;
int blink_times_ = 0;
int blink_interval_ms_ = 0;
std::atomic<bool> should_blink_{false};
void Initialize();
void StartBlinkTask(int times, int interval_ms);
void StopBlinkInternal();
};
#endif // _BUILTIN_LED_H_

View File

@@ -4,6 +4,9 @@
static const char* TAG = "Button";
Button::Button(gpio_num_t gpio_num) : gpio_num_(gpio_num) {
if (gpio_num == GPIO_NUM_NC) {
return;
}
button_config_t button_config = {
.type = BUTTON_TYPE_GPIO,
.long_press_time = 1000,
@@ -27,6 +30,9 @@ Button::~Button() {
}
void Button::OnPress(std::function<void()> callback) {
if (button_handle_ == nullptr) {
return;
}
on_press_ = callback;
iot_button_register_cb(button_handle_, BUTTON_PRESS_DOWN, [](void* handle, void* usr_data) {
Button* button = static_cast<Button*>(usr_data);
@@ -37,6 +43,9 @@ void Button::OnPress(std::function<void()> callback) {
}
void Button::OnLongPress(std::function<void()> callback) {
if (button_handle_ == nullptr) {
return;
}
on_long_press_ = callback;
iot_button_register_cb(button_handle_, BUTTON_LONG_PRESS_START, [](void* handle, void* usr_data) {
Button* button = static_cast<Button*>(usr_data);
@@ -47,6 +56,9 @@ void Button::OnLongPress(std::function<void()> callback) {
}
void Button::OnClick(std::function<void()> callback) {
if (button_handle_ == nullptr) {
return;
}
on_click_ = callback;
iot_button_register_cb(button_handle_, BUTTON_SINGLE_CLICK, [](void* handle, void* usr_data) {
Button* button = static_cast<Button*>(usr_data);
@@ -57,6 +69,9 @@ void Button::OnClick(std::function<void()> callback) {
}
void Button::OnDoubleClick(std::function<void()> callback) {
if (button_handle_ == nullptr) {
return;
}
on_double_click_ = callback;
iot_button_register_cb(button_handle_, BUTTON_DOUBLE_CLICK, [](void* handle, void* usr_data) {
Button* button = static_cast<Button*>(usr_data);

View File

@@ -3,15 +3,65 @@ set(SOURCES "AudioDevice.cc"
"SystemInfo.cc"
"SystemReset.cc"
"Application.cc"
"Display.cc"
"Button.cc"
"BuiltinLed.cc"
"Display.cc"
"Board.cc"
"main.cc"
)
set(INCLUDE_DIRS ".")
# 根据 BOARD_TYPE 配置添加对应的板级文件
if(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI)
# add all files from boards/bread-compact-wifi
set(BOARD_TYPE "bread-compact-wifi")
file(GLOB BOARD_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.cc)
list(APPEND SOURCES ${BOARD_SOURCES} "WifiBoard.cc")
list(APPEND INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE})
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ML307)
# add all files from boards/bread-compact-ml307
set(BOARD_TYPE "bread-compact-ml307")
file(GLOB BOARD_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.cc)
list(APPEND SOURCES ${BOARD_SOURCES} "Ml307Board.cc")
list(APPEND INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE})
elseif(CONFIG_BOARD_TYPE_ESP_BOX_3)
# add all files from boards/esp-box-3
set(BOARD_TYPE "esp-box-3")
file(GLOB BOARD_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.cc)
list(APPEND SOURCES ${BOARD_SOURCES} "WifiBoard.cc")
list(APPEND INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE})
list(APPEND SOURCES "BoxAudioDevice.cc")
elseif(CONFIG_BOARD_TYPE_KEVIN_BOX_0)
# add all files from boards/kevin-box-0
set(BOARD_TYPE "kevin-box-0")
file(GLOB BOARD_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.cc)
list(APPEND SOURCES ${BOARD_SOURCES} "Ml307Board.cc")
list(APPEND INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE})
list(APPEND SOURCES "BoxAudioDevice.cc")
elseif(CONFIG_BOARD_TYPE_KEVIN_BOX_1)
# add all files from boards/kevin-box-1
set(BOARD_TYPE "kevin-box-1")
file(GLOB BOARD_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.cc)
list(APPEND SOURCES ${BOARD_SOURCES} "Ml307Board.cc")
list(APPEND INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE})
list(APPEND SOURCES "BoxAudioDevice.cc")
elseif(CONFIG_BOARD_TYPE_LICHUANG_DEV)
# add all files from boards/lichuang-dev
set(BOARD_TYPE "lichuang-dev")
file(GLOB BOARD_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.cc)
list(APPEND SOURCES ${BOARD_SOURCES} "WifiBoard.cc")
list(APPEND INCLUDE_DIRS ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE})
list(APPEND SOURCES "BoxAudioDevice.cc")
endif()
if(CONFIG_USE_AFE_SR)
list(APPEND SOURCES "AudioProcessor.cc" "WakeWordDetect.cc")
endif()
idf_component_register(SRCS ${SOURCES}
INCLUDE_DIRS "."
EMBED_FILES "assets/err_reg.p3" "assets/err_pin.p3" "assets/err_wificonfig.p3"
INCLUDE_DIRS ${INCLUDE_DIRS}
)
# 使用 target_compile_definitions 来定义 BOARD_TYPE
target_compile_definitions(${COMPONENT_LIB} PRIVATE BOARD_TYPE=\"${BOARD_TYPE}\")

View File

@@ -1,6 +1,3 @@
#include "Display.h"
#include <esp_log.h>
#include <esp_err.h>
#include <esp_lcd_panel_ops.h>
@@ -9,12 +6,17 @@
#include <string>
#include <cstdlib>
#include "Display.h"
#include "Board.h"
#include "Application.h"
#define TAG "Display"
#ifdef CONFIG_USE_DISPLAY
Display::Display(int sda_pin, int scl_pin) : sda_pin_(sda_pin), scl_pin_(scl_pin) {
ESP_LOGI(TAG, "Display Pins: %d, %d", sda_pin_, scl_pin_);
if (sda_pin_ == GPIO_NUM_NC || scl_pin_ == GPIO_NUM_NC) {
ESP_LOGI(TAG, "Display not connected");
return;
}
i2c_master_bus_config_t bus_config = {
.i2c_port = I2C_NUM_0,
@@ -22,7 +24,7 @@ Display::Display(int sda_pin, int scl_pin) : sda_pin_(sda_pin), scl_pin_(scl_pin
.scl_io_num = (gpio_num_t)scl_pin_,
.clk_source = I2C_CLK_SRC_DEFAULT,
.glitch_ignore_cnt = 7,
.intr_priority = 1,
.intr_priority = 0,
.trans_queue_depth = 0,
.flags = {
.enable_internal_pullup = 1,
@@ -44,7 +46,7 @@ Display::Display(int sda_pin, int scl_pin) : sda_pin_(sda_pin), scl_pin_(scl_pin
.dc_low_on_data = 0,
.disable_control_phase = 0,
},
.scl_speed_hz = 400 * 1000,
.scl_speed_hz = 100 * 1000,
};
ESP_ERROR_CHECK(esp_lcd_new_panel_io_i2c_v2(i2c_bus_, &io_config, &panel_io_));
@@ -55,7 +57,7 @@ Display::Display(int sda_pin, int scl_pin) : sda_pin_(sda_pin), scl_pin_(scl_pin
panel_config.bits_per_pixel = 1;
esp_lcd_panel_ssd1306_config_t ssd1306_config = {
.height = CONFIG_DISPLAY_HEIGHT
.height = DISPLAY_HEIGHT
};
panel_config.vendor_config = &ssd1306_config;
@@ -73,45 +75,66 @@ Display::Display(int sda_pin, int scl_pin) : sda_pin_(sda_pin), scl_pin_(scl_pin
lvgl_port_cfg_t port_cfg = ESP_LVGL_PORT_INIT_CONFIG();
lvgl_port_init(&port_cfg);
const lvgl_port_display_cfg_t display_cfg = {
.io_handle = panel_io_,
.panel_handle = panel_,
.buffer_size = 128 * CONFIG_DISPLAY_HEIGHT,
.double_buffer = true,
.hres = 128,
.vres = CONFIG_DISPLAY_HEIGHT,
.monochrome = true,
.rotation = {
.swap_xy = 0,
.mirror_x = 0,
.mirror_y = 0,
},
.flags = {
.buff_dma = 0,
.buff_spiram = 0,
},
};
disp_ = lvgl_port_add_disp(&display_cfg);
lv_disp_set_rotation(disp_, LV_DISP_ROT_180);
// Set the display to on
ESP_LOGI(TAG, "Turning display on");
ESP_ERROR_CHECK(esp_lcd_panel_disp_on_off(panel_, true));
ESP_LOGI(TAG, "Adding LCD screen");
const lvgl_port_display_cfg_t display_cfg = {
.io_handle = panel_io_,
.panel_handle = panel_,
.control_handle = nullptr,
.buffer_size = DISPLAY_WIDTH * DISPLAY_HEIGHT,
.double_buffer = false,
.trans_size = 0,
.hres = DISPLAY_WIDTH,
.vres = DISPLAY_HEIGHT,
.monochrome = true,
.rotation = {
.swap_xy = false,
.mirror_x = DISPLAY_MIRROR_X,
.mirror_y = DISPLAY_MIRROR_Y,
},
.flags = {
.buff_dma = 1,
.buff_spiram = 0,
.sw_rotate = 0,
.full_refresh = 0,
.direct_mode = 0,
},
};
disp_ = lvgl_port_add_disp(&display_cfg);;
ESP_LOGI(TAG, "Display Loading...");
if (lvgl_port_lock(0)) {
label_ = lv_label_create(lv_disp_get_scr_act(disp_));
// lv_obj_set_style_text_font(label_, font_, 0);
lv_label_set_text(label_, "Initializing...");
lv_obj_set_width(label_, disp_->driver->hor_res);
lv_obj_set_height(label_, disp_->driver->ver_res);
notification_ = lv_label_create(lv_disp_get_scr_act(disp_));
// lv_obj_set_style_text_font(notification_, font_, 0);
lv_label_set_text(notification_, "Notification\nTest");
lv_obj_set_width(notification_, disp_->driver->hor_res);
lv_obj_set_height(notification_, disp_->driver->ver_res);
lv_obj_set_style_opa(notification_, LV_OPA_MIN, 0);
lvgl_port_unlock();
}
// Create a timer to update the display every 10 seconds
esp_timer_create_args_t update_display_timer_args = {
.callback = [](void *arg) {
Display* display = static_cast<Display*>(arg);
display->UpdateDisplay();
},
.arg = this,
.dispatch_method = ESP_TIMER_TASK,
.name = "UpdateDisplay",
.skip_unhandled_events = false,
};
ESP_ERROR_CHECK(esp_timer_create(&update_display_timer_args, &update_display_timer_));
ESP_ERROR_CHECK(esp_timer_start_periodic(update_display_timer_, 10 * 1000000));
}
Display::~Display() {
@@ -119,6 +142,10 @@ Display::~Display() {
esp_timer_stop(notification_timer_);
esp_timer_delete(notification_timer_);
}
if (update_display_timer_ != nullptr) {
esp_timer_stop(update_display_timer_);
esp_timer_delete(update_display_timer_);
}
lvgl_port_lock(0);
if (label_ != nullptr) {
@@ -127,6 +154,10 @@ Display::~Display() {
}
lvgl_port_unlock();
if (font_ != nullptr) {
lv_font_free(font_);
}
if (disp_ != nullptr) {
lvgl_port_deinit();
esp_lcd_panel_del(panel_);
@@ -176,4 +207,31 @@ void Display::ShowNotification(const std::string &text) {
}
}
#endif
void Display::UpdateDisplay() {
auto chat_state = Application::GetInstance().GetChatState();
if (chat_state == kChatStateIdle || chat_state == kChatStateConnecting || chat_state == kChatStateListening) {
std::string text;
auto& board = Board::GetInstance();
std::string network_name;
int signal_quality;
std::string signal_quality_text;
if (!board.GetNetworkState(network_name, signal_quality, signal_quality_text)) {
text = "No network";
} else {
text = network_name + "\n" + signal_quality_text;
if (std::abs(signal_quality) != 99) {
text += " (" + std::to_string(signal_quality) + ")";
}
}
int battery_voltage;
bool charging;
if (board.GetBatteryVoltage(battery_voltage, charging)) {
text += "\n" + std::to_string(battery_voltage) + "mV";
if (charging) {
text += " (Charging)";
}
}
SetText(text);
}
}

View File

@@ -17,6 +17,8 @@ public:
void SetText(const std::string &text);
void ShowNotification(const std::string &text);
void UpdateDisplay();
private:
int sda_pin_;
int scl_pin_;
@@ -26,9 +28,11 @@ private:
esp_lcd_panel_io_handle_t panel_io_ = nullptr;
esp_lcd_panel_handle_t panel_ = nullptr;
lv_disp_t *disp_ = nullptr;
lv_font_t *font_ = nullptr;
lv_obj_t *label_ = nullptr;
lv_obj_t *notification_ = nullptr;
esp_timer_handle_t notification_timer_ = nullptr;
esp_timer_handle_t update_display_timer_ = nullptr;
std::string text_;
};

View File

@@ -1,11 +1,14 @@
#include "FirmwareUpgrade.h"
#include "SystemInfo.h"
#include "Board.h"
#include <cJSON.h>
#include <esp_log.h>
#include <esp_partition.h>
#include <esp_http_client.h>
#include <esp_ota_ops.h>
#include <esp_app_format.h>
#include <esp_chip_info.h>
#include <vector>
#include <sstream>
@@ -14,7 +17,7 @@
#define TAG "FirmwareUpgrade"
FirmwareUpgrade::FirmwareUpgrade(Http& http) : http_(http) {
FirmwareUpgrade::FirmwareUpgrade() {
}
FirmwareUpgrade::~FirmwareUpgrade() {
@@ -24,10 +27,6 @@ void FirmwareUpgrade::SetCheckVersionUrl(std::string check_version_url) {
check_version_url_ = check_version_url;
}
void FirmwareUpgrade::SetPostData(const std::string& post_data) {
post_data_ = post_data;
}
void FirmwareUpgrade::SetHeader(const std::string& key, const std::string& value) {
headers_[key] = value;
}
@@ -41,20 +40,18 @@ void FirmwareUpgrade::CheckVersion() {
return;
}
auto http = Board::GetInstance().CreateHttp();
for (const auto& header : headers_) {
http_.SetHeader(header.first, header.second);
http->SetHeader(header.first, header.second);
}
if (post_data_.empty()) {
http_.Open("GET", check_version_url_);
} else {
http_.SetHeader("Content-Type", "application/json");
http_.SetContent(post_data_);
http_.Open("POST", check_version_url_);
}
http->SetHeader("Content-Type", "application/json");
http->SetContent(GetPostData());
http->Open("POST", check_version_url_);
auto response = http_.GetBody();
http_.Close();
auto response = http->GetBody();
http->Close();
delete http;
// Response: { "firmware": { "version": "1.0.0", "url": "http://" } }
// Parse the JSON response and check if the version is newer
@@ -130,15 +127,17 @@ void FirmwareUpgrade::Upgrade(const std::string& firmware_url) {
bool image_header_checked = false;
std::string image_header;
if (!http_.Open("GET", firmware_url)) {
auto http = Board::GetInstance().CreateHttp();
if (!http->Open("GET", firmware_url)) {
ESP_LOGE(TAG, "Failed to open HTTP connection");
delete http;
return;
}
size_t content_length = http_.GetBodyLength();
size_t content_length = http->GetBodyLength();
if (content_length == 0) {
ESP_LOGE(TAG, "Failed to get content length");
http_.Close();
delete http;
return;
}
@@ -146,10 +145,10 @@ void FirmwareUpgrade::Upgrade(const std::string& firmware_url) {
size_t total_read = 0, recent_read = 0;
auto last_calc_time = esp_timer_get_time();
while (true) {
int ret = http_.Read(buffer, sizeof(buffer));
int ret = http->Read(buffer, sizeof(buffer));
if (ret < 0) {
ESP_LOGE(TAG, "Failed to read HTTP data: %s", esp_err_to_name(ret));
http_.Close();
delete http;
return;
}
@@ -181,13 +180,13 @@ void FirmwareUpgrade::Upgrade(const std::string& firmware_url) {
auto current_version = esp_app_get_description()->version;
if (memcmp(new_app_info.version, current_version, sizeof(new_app_info.version)) == 0) {
ESP_LOGE(TAG, "Firmware version is the same, skipping upgrade");
http_.Close();
delete http;
return;
}
if (esp_ota_begin(update_partition, OTA_WITH_SEQUENTIAL_WRITES, &update_handle)) {
esp_ota_abort(update_handle);
http_.Close();
delete http;
ESP_LOGE(TAG, "Failed to begin OTA");
return;
}
@@ -199,11 +198,11 @@ void FirmwareUpgrade::Upgrade(const std::string& firmware_url) {
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to write OTA data: %s", esp_err_to_name(err));
esp_ota_abort(update_handle);
http_.Close();
delete http;
return;
}
}
http_.Close();
delete http;
esp_err_t err = esp_ota_end(update_handle);
if (err != ESP_OK) {
@@ -257,3 +256,99 @@ bool FirmwareUpgrade::IsNewVersionAvailable(const std::string& currentVersion, c
return newer.size() > current.size();
}
void FirmwareUpgrade::SetBoardJson(const std::string& board_json) {
board_json_ = board_json;
}
std::string FirmwareUpgrade::GetPostData() {
/*
{
"flash_size": 4194304,
"psram_size": 0,
"minimum_free_heap_size": 123456,
"mac_address": "00:00:00:00:00:00",
"chip_model_name": "esp32s3",
"chip_info": {
"model": 1,
"cores": 2,
"revision": 0,
"features": 0
},
"application": {
"name": "my-app",
"version": "1.0.0",
"compile_time": "2021-01-01T00:00:00Z"
"idf_version": "4.2-dev"
"elf_sha256": ""
},
"partition_table": [
"app": {
"label": "app",
"type": 1,
"subtype": 2,
"address": 0x10000,
"size": 0x100000
}
],
"ota": {
"label": "ota_0"
}
}
*/
std::string json = "{";
json += "\"flash_size\":" + std::to_string(SystemInfo::GetFlashSize()) + ",";
json += "\"minimum_free_heap_size\":" + std::to_string(SystemInfo::GetMinimumFreeHeapSize()) + ",";
json += "\"mac_address\":\"" + SystemInfo::GetMacAddress() + "\",";
json += "\"chip_model_name\":\"" + SystemInfo::GetChipModelName() + "\",";
json += "\"chip_info\":{";
esp_chip_info_t chip_info;
esp_chip_info(&chip_info);
json += "\"model\":" + std::to_string(chip_info.model) + ",";
json += "\"cores\":" + std::to_string(chip_info.cores) + ",";
json += "\"revision\":" + std::to_string(chip_info.revision) + ",";
json += "\"features\":" + std::to_string(chip_info.features);
json += "},";
json += "\"application\":{";
auto app_desc = esp_app_get_description();
json += "\"name\":\"" + std::string(app_desc->project_name) + "\",";
json += "\"version\":\"" + std::string(app_desc->version) + "\",";
json += "\"compile_time\":\"" + std::string(app_desc->date) + "T" + std::string(app_desc->time) + "Z\",";
json += "\"idf_version\":\"" + std::string(app_desc->idf_ver) + "\",";
char sha256_str[65];
for (int i = 0; i < 32; i++) {
snprintf(sha256_str + i * 2, sizeof(sha256_str) - i * 2, "%02x", app_desc->app_elf_sha256[i]);
}
json += "\"elf_sha256\":\"" + std::string(sha256_str) + "\"";
json += "},";
json += "\"partition_table\": [";
esp_partition_iterator_t it = esp_partition_find(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY, NULL);
while (it) {
const esp_partition_t *partition = esp_partition_get(it);
json += "{";
json += "\"label\":\"" + std::string(partition->label) + "\",";
json += "\"type\":" + std::to_string(partition->type) + ",";
json += "\"subtype\":" + std::to_string(partition->subtype) + ",";
json += "\"address\":" + std::to_string(partition->address) + ",";
json += "\"size\":" + std::to_string(partition->size);
json += "},";
it = esp_partition_next(it);
}
json.pop_back(); // Remove the last comma
json += "],";
json += "\"ota\":{";
auto ota_partition = esp_ota_get_running_partition();
json += "\"label\":\"" + std::string(ota_partition->label) + "\"";
json += "},";
json += "\"board\":" + board_json_;
// Close the JSON object
json += "}";
return json;
}

View File

@@ -5,15 +5,13 @@
#include <string>
#include <map>
#include <Http.h>
class FirmwareUpgrade {
public:
FirmwareUpgrade(Http& http);
FirmwareUpgrade();
~FirmwareUpgrade();
void SetBoardJson(const std::string& board_json);
void SetCheckVersionUrl(std::string check_version_url);
void SetPostData(const std::string& post_data);
void SetHeader(const std::string& key, const std::string& value);
void CheckVersion();
bool HasNewVersion() { return has_new_version_; }
@@ -21,18 +19,18 @@ public:
void MarkCurrentVersionValid();
private:
Http& http_;
std::string check_version_url_;
bool has_new_version_ = false;
std::string firmware_version_;
std::string firmware_url_;
std::string post_data_;
std::string board_json_;
std::map<std::string, std::string> headers_;
void Upgrade(const std::string& firmware_url);
std::function<void(int progress, size_t speed)> upgrade_callback_;
std::vector<int> ParseVersion(const std::string& version);
bool IsNewVersionAvailable(const std::string& currentVersion, const std::string& newVersion);
std::string GetPostData();
};
#endif // _FIRMWARE_UPGRADE_H

View File

@@ -18,73 +18,24 @@ config WEBSOCKET_ACCESS_TOKEN
help
Access token for websocket communication.
config AUDIO_INPUT_SAMPLE_RATE
int "Audio Input Sample Rate"
default 16000
choice BOARD_TYPE
prompt "Board Type"
default BOARD_TYPE_BREAD_COMPACT_WIFI
help
Audio input sample rate.
config AUDIO_OUTPUT_SAMPLE_RATE
int "Audio Output Sample Rate"
default 24000
help
Audio output sample rate.
config AUDIO_DEVICE_I2S_MIC_GPIO_WS
int "I2S GPIO WS"
default 4
help
GPIO number of the I2S WS.
config AUDIO_DEVICE_I2S_MIC_GPIO_BCLK
int "I2S GPIO BCLK"
default 5
help
GPIO number of the I2S BCLK.
config AUDIO_DEVICE_I2S_MIC_GPIO_DIN
int "I2S GPIO DIN"
default 6
help
GPIO number of the I2S DIN.
config AUDIO_DEVICE_I2S_SPK_GPIO_DOUT
int "I2S GPIO DOUT"
default 7
help
GPIO number of the I2S DOUT.
config AUDIO_DEVICE_I2S_SIMPLEX
bool "I2S Simplex"
default y
help
Enable I2S Simplex mode.
config AUDIO_DEVICE_I2S_SPK_GPIO_BCLK
int "I2S SPK GPIO BCLK"
default 15
depends on AUDIO_DEVICE_I2S_SIMPLEX
help
GPIO number of the I2S MIC BCLK.
config AUDIO_DEVICE_I2S_SPK_GPIO_WS
int "I2S SPK GPIO WS"
default 16
depends on AUDIO_DEVICE_I2S_SIMPLEX
help
GPIO number of the I2S MIC WS.
config BOOT_BUTTON_GPIO
int "Boot Button GPIO"
default 0
help
GPIO number of the boot button.
config VOLUME_UP_BUTTON_GPIO
int "Volume Up Button GPIO"
default 40
help
GPIO number of the volume up button.
Board type. 开发板类型
config BOARD_TYPE_BREAD_COMPACT_WIFI
bool "面包板新版接线WiFi"
config BOARD_TYPE_BREAD_COMPACT_ML307
bool "面包板新版接线ML307 AT"
config BOARD_TYPE_ESP_BOX_3
bool "ESP BOX 3"
config BOARD_TYPE_KEVIN_BOX_0
bool "Kevin Box 0"
config BOARD_TYPE_KEVIN_BOX_1
bool "Kevin Box 1"
config BOARD_TYPE_LICHUANG_DEV
bool "立创开发板"
endchoice
config USE_AFE_SR
bool "Use Espressif AFE SR"
@@ -92,51 +43,4 @@ config USE_AFE_SR
help
Use AFE SR for wake word detection.
config USE_ML307
bool "Use ML307"
default n
help
Use ML307 as the modem.
config ML307_RX_PIN
int "ML307 RX Pin"
default 11
depends on USE_ML307
help
GPIO number of the ML307 RX.
config ML307_TX_PIN
int "ML307 TX Pin"
default 12
depends on USE_ML307
help
GPIO number of the ML307 TX.
config USE_DISPLAY
bool "Use Display"
default n
help
Use Display.
config DISPLAY_HEIGHT
int "Display Height"
default 32
depends on USE_DISPLAY
help
Display height in pixels.
config DISPLAY_SDA_PIN
int "Display SDA Pin"
default 41
depends on USE_DISPLAY
help
GPIO number of the Display SDA.
config DISPLAY_SCL_PIN
int "Display SCL Pin"
default 42
depends on USE_DISPLAY
help
GPIO number of the Display SCL.
endmenu

112
main/Ml307Board.cc Normal file
View File

@@ -0,0 +1,112 @@
#include "Ml307Board.h"
#include "Application.h"
#include <esp_log.h>
#include <Ml307Http.h>
#include <Ml307SslTransport.h>
#include <WebSocket.h>
#include <esp_timer.h>
static const char *TAG = "Ml307Board";
static std::string csq_to_string(int csq) {
if (csq == -1) {
return "No network";
} else if (csq >= 0 && csq <= 9) {
return "Very bad";
} else if (csq >= 10 && csq <= 14) {
return "Bad";
} else if (csq >= 15 && csq <= 19) {
return "Fair";
} else if (csq >= 20 && csq <= 24) {
return "Good";
} else if (csq >= 25 && csq <= 31) {
return "Very good";
}
return "Invalid";
}
Ml307Board::Ml307Board() : modem_(ML307_TX_PIN, ML307_RX_PIN, 4096) {
}
void Ml307Board::StartNetwork() {
auto& application = Application::GetInstance();
auto& display = application.GetDisplay();
display.SetText(std::string("Wait for network\n"));
int result = modem_.WaitForNetworkReady();
if (result == -1) {
application.Alert("Error", "PIN is not ready");
} else if (result == -2) {
application.Alert("Error", "Registration denied");
}
// Print the ML307 modem information
std::string module_name = modem_.GetModuleName();
std::string imei = modem_.GetImei();
std::string iccid = modem_.GetIccid();
ESP_LOGI(TAG, "ML307 Module: %s", module_name.c_str());
ESP_LOGI(TAG, "ML307 IMEI: %s", imei.c_str());
ESP_LOGI(TAG, "ML307 ICCID: %s", iccid.c_str());
}
void Ml307Board::StartModem() {
auto& display = Application::GetInstance().GetDisplay();
display.SetText(std::string("Starting modem"));
modem_.SetDebug(false);
modem_.SetBaudRate(921600);
auto& application = Application::GetInstance();
// If low power, the material ready event will be triggered by the modem because of a reset
modem_.OnMaterialReady([this, &application]() {
ESP_LOGI(TAG, "ML307 material ready");
application.Schedule([this, &application]() {
application.SetChatState(kChatStateIdle);
StartNetwork();
});
});
}
void Ml307Board::Initialize() {
ESP_LOGI(TAG, "Initializing Ml307Board");
StartModem();
}
AudioDevice* Ml307Board::CreateAudioDevice() {
return new AudioDevice();
}
Http* Ml307Board::CreateHttp() {
return new Ml307Http(modem_);
}
WebSocket* Ml307Board::CreateWebSocket() {
return new WebSocket(new Ml307SslTransport(modem_, 0));
}
bool Ml307Board::GetNetworkState(std::string& network_name, int& signal_quality, std::string& signal_quality_text) {
if (!modem_.network_ready()) {
return false;
}
network_name = modem_.GetCarrierName();
signal_quality = modem_.GetCsq();
signal_quality_text = csq_to_string(signal_quality);
return signal_quality != -1;
}
std::string Ml307Board::GetJson() {
// Set the board type for OTA
std::string board_type = BOARD_TYPE;
std::string module_name = modem_.GetModuleName();
std::string carrier_name = modem_.GetCarrierName();
std::string imei = modem_.GetImei();
std::string iccid = modem_.GetIccid();
int csq = modem_.GetCsq();
std::string board_json = std::string("{\"type\":\"" + board_type + "\",");
board_json += "\"revision\":\"" + module_name + "\",";
board_json += "\"carrier\":\"" + carrier_name + "\",";
board_json += "\"csq\":\"" + std::to_string(csq) + "\",";
board_json += "\"imei\":\"" + imei + "\",";
board_json += "\"iccid\":\"" + iccid + "\"}";
return board_json;
}

24
main/Ml307Board.h Normal file
View File

@@ -0,0 +1,24 @@
#ifndef ML307_BOARD_H
#define ML307_BOARD_H
#include "Board.h"
#include <Ml307AtModem.h>
class Ml307Board : public Board {
protected:
Ml307AtModem modem_;
void StartModem();
public:
Ml307Board();
virtual void Initialize() override;
virtual void StartNetwork() override;
virtual AudioDevice* CreateAudioDevice() override;
virtual Http* CreateHttp() override;
virtual WebSocket* CreateWebSocket() override;
virtual bool GetNetworkState(std::string& network_name, int& signal_quality, std::string& signal_quality_text) override;
virtual std::string GetJson() override;
};
#endif // ML307_BOARD_H

View File

@@ -3,7 +3,6 @@
#include <esp_log.h>
#include <esp_flash.h>
#include <esp_mac.h>
#include <esp_chip_info.h>
#include <esp_system.h>
#include <esp_partition.h>
#include <esp_app_desc.h>
@@ -41,96 +40,6 @@ std::string SystemInfo::GetChipModelName() {
return std::string(CONFIG_IDF_TARGET);
}
std::string SystemInfo::GetJsonString() {
/*
{
"flash_size": 4194304,
"psram_size": 0,
"minimum_free_heap_size": 123456,
"mac_address": "00:00:00:00:00:00",
"chip_model_name": "esp32s3",
"chip_info": {
"model": 1,
"cores": 2,
"revision": 0,
"features": 0
},
"application": {
"name": "my-app",
"version": "1.0.0",
"compile_time": "2021-01-01T00:00:00Z"
"idf_version": "4.2-dev"
"elf_sha256": ""
},
"partition_table": [
"app": {
"label": "app",
"type": 1,
"subtype": 2,
"address": 0x10000,
"size": 0x100000
}
],
"ota": {
"label": "ota_0"
}
}
*/
std::string json = "{";
json += "\"flash_size\":" + std::to_string(GetFlashSize()) + ",";
json += "\"minimum_free_heap_size\":" + std::to_string(GetMinimumFreeHeapSize()) + ",";
json += "\"mac_address\":\"" + GetMacAddress() + "\",";
json += "\"chip_model_name\":\"" + GetChipModelName() + "\",";
json += "\"chip_info\":{";
esp_chip_info_t chip_info;
esp_chip_info(&chip_info);
json += "\"model\":" + std::to_string(chip_info.model) + ",";
json += "\"cores\":" + std::to_string(chip_info.cores) + ",";
json += "\"revision\":" + std::to_string(chip_info.revision) + ",";
json += "\"features\":" + std::to_string(chip_info.features);
json += "},";
json += "\"application\":{";
auto app_desc = esp_app_get_description();
json += "\"name\":\"" + std::string(app_desc->project_name) + "\",";
json += "\"version\":\"" + std::string(app_desc->version) + "\",";
json += "\"compile_time\":\"" + std::string(app_desc->date) + "T" + std::string(app_desc->time) + "Z\",";
json += "\"idf_version\":\"" + std::string(app_desc->idf_ver) + "\",";
char sha256_str[65];
for (int i = 0; i < 32; i++) {
snprintf(sha256_str + i * 2, sizeof(sha256_str) - i * 2, "%02x", app_desc->app_elf_sha256[i]);
}
json += "\"elf_sha256\":\"" + std::string(sha256_str) + "\"";
json += "},";
json += "\"partition_table\": [";
esp_partition_iterator_t it = esp_partition_find(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY, NULL);
while (it) {
const esp_partition_t *partition = esp_partition_get(it);
json += "{";
json += "\"label\":\"" + std::string(partition->label) + "\",";
json += "\"type\":" + std::to_string(partition->type) + ",";
json += "\"subtype\":" + std::to_string(partition->subtype) + ",";
json += "\"address\":" + std::to_string(partition->address) + ",";
json += "\"size\":" + std::to_string(partition->size);
json += "},";
it = esp_partition_next(it);
}
json.pop_back(); // Remove the last comma
json += "],";
json += "\"ota\":{";
auto ota_partition = esp_ota_get_running_partition();
json += "\"label\":\"" + std::string(ota_partition->label) + "\"";
json += "}";
// Close the JSON object
json += "}";
return json;
}
esp_err_t SystemInfo::PrintRealTimeStats(TickType_t xTicksToWait) {
#define ARRAY_SIZE_OFFSET 5
TaskStatus_t *start_array = NULL, *end_array = NULL;

View File

@@ -13,7 +13,6 @@ public:
static size_t GetFreeHeapSize();
static std::string GetMacAddress();
static std::string GetChipModelName();
static std::string GetJsonString();
static esp_err_t PrintRealTimeStats(TickType_t xTicksToWait);
};

View File

@@ -41,6 +41,10 @@ void SystemReset::ResetNvsFlash() {
if (ret != ESP_OK) {
ESP_LOGE(TAG, "Failed to erase NVS flash");
}
ret = nvs_flash_init();
if (ret != ESP_OK) {
ESP_LOGE(TAG, "Failed to initialize NVS flash");
}
}
void SystemReset::ResetToFactory() {

View File

@@ -3,11 +3,18 @@
class SystemReset {
public:
SystemReset();
static SystemReset& GetInstance() {
static SystemReset instance;
return instance;
}
void CheckButtons();
private:
SystemReset(); // 构造函数私有化
SystemReset(const SystemReset&) = delete; // 禁用拷贝构造
SystemReset& operator=(const SystemReset&) = delete; // 禁用赋值操作
void ResetNvsFlash();
void ResetToFactory();
void RestartInSeconds(int seconds);

View File

@@ -1,5 +1,6 @@
#include <esp_log.h>
#include <model_path.h>
#include <arpa/inet.h>
#include "WakeWordDetect.h"
#include "Application.h"
@@ -15,6 +16,24 @@ WakeWordDetect::WakeWordDetect()
wake_word_opus_() {
event_group_ = xEventGroupCreate();
}
WakeWordDetect::~WakeWordDetect() {
if (afe_detection_data_ != nullptr) {
esp_afe_sr_v1.destroy(afe_detection_data_);
}
if (wake_word_encode_task_stack_ != nullptr) {
free(wake_word_encode_task_stack_);
}
vEventGroupDelete(event_group_);
}
void WakeWordDetect::Initialize(int channels, bool reference) {
channels_ = channels;
reference_ = reference;
int ref_num = reference_ ? 1 : 0;
srmodel_list_t *models = esp_srmodel_init("model");
for (int i = 0; i < models->num; i++) {
@@ -25,7 +44,7 @@ WakeWordDetect::WakeWordDetect()
}
afe_config_t afe_config = {
.aec_init = false,
.aec_init = reference_,
.se_init = true,
.vad_init = true,
.wakenet_init = true,
@@ -37,17 +56,17 @@ WakeWordDetect::WakeWordDetect()
.wakenet_model_name_2 = NULL,
.wakenet_mode = DET_MODE_90,
.afe_mode = SR_MODE_HIGH_PERF,
.afe_perferred_core = 0,
.afe_perferred_priority = 5,
.afe_perferred_core = 1,
.afe_perferred_priority = 1,
.afe_ringbuf_size = 50,
.memory_alloc_mode = AFE_MEMORY_ALLOC_MORE_PSRAM,
.afe_linear_gain = 1.0,
.agc_mode = AFE_MN_PEAK_AGC_MODE_2,
.pcm_config = {
.total_ch_num = 1,
.mic_num = 1,
.ref_num = 0,
.sample_rate = CONFIG_AUDIO_INPUT_SAMPLE_RATE
.total_ch_num = channels_,
.mic_num = channels_ - ref_num,
.ref_num = ref_num,
.sample_rate = 16000
},
.debug_init = false,
.debug_hook = {{ AFE_DEBUG_HOOK_MASE_TASK_IN, NULL }, { AFE_DEBUG_HOOK_FETCH_TASK_IN, NULL }},
@@ -62,19 +81,7 @@ WakeWordDetect::WakeWordDetect()
auto this_ = (WakeWordDetect*)arg;
this_->AudioDetectionTask();
vTaskDelete(NULL);
}, "audio_detection", 4096 * 2, this, 5, NULL);
}
WakeWordDetect::~WakeWordDetect() {
if (afe_detection_data_ != nullptr) {
esp_afe_sr_v1.destroy(afe_detection_data_);
}
if (wake_word_encode_task_stack_ != nullptr) {
free(wake_word_encode_task_stack_);
}
vEventGroupDelete(event_group_);
}, "audio_detection", 4096 * 2, this, 1, NULL);
}
void WakeWordDetect::OnWakeWordDetected(std::function<void()> callback) {
@@ -97,10 +104,10 @@ bool WakeWordDetect::IsDetectionRunning() {
return xEventGroupGetBits(event_group_) & DETECTION_RUNNING_EVENT;
}
void WakeWordDetect::Feed(const int16_t* data, int size) {
input_buffer_.insert(input_buffer_.end(), data, data + size);
void WakeWordDetect::Feed(std::vector<int16_t>& data) {
input_buffer_.insert(input_buffer_.end(), data.begin(), data.end());
auto chunk_size = esp_afe_sr_v1.get_feed_chunksize(afe_detection_data_);
auto chunk_size = esp_afe_sr_v1.get_feed_chunksize(afe_detection_data_) * channels_;
while (input_buffer_.size() >= chunk_size) {
esp_afe_sr_v1.feed(afe_detection_data_, input_buffer_.data());
input_buffer_.erase(input_buffer_.begin(), input_buffer_.begin() + chunk_size);
@@ -166,21 +173,19 @@ void WakeWordDetect::EncodeWakeWordData() {
auto start_time = esp_timer_get_time();
// encode detect packets
OpusEncoder* encoder = new OpusEncoder();
encoder->Configure(CONFIG_AUDIO_INPUT_SAMPLE_RATE, 1, 60);
encoder->Configure(16000, 1, 60);
encoder->SetComplexity(0);
this_->wake_word_opus_.resize(4096 * 4);
size_t offset = 0;
for (auto& pcm: this_->wake_word_pcm_) {
encoder->Encode(pcm, [this_, &offset](const uint8_t* opus, size_t opus_size) {
size_t protocol_size = sizeof(BinaryProtocol) + opus_size;
size_t protocol_size = sizeof(BinaryProtocol3) + opus_size;
if (offset + protocol_size < this_->wake_word_opus_.size()) {
auto protocol = (BinaryProtocol*)(&this_->wake_word_opus_[offset]);
protocol->version = htons(PROTOCOL_VERSION);
protocol->type = htons(0);
auto protocol = (BinaryProtocol3*)(&this_->wake_word_opus_[offset]);
protocol->type = 0;
protocol->reserved = 0;
protocol->timestamp = 0;
protocol->payload_size = htonl(opus_size);
protocol->payload_size = htons(opus_size);
memcpy(protocol->payload, opus, opus_size);
offset += protocol_size;
}

View File

@@ -19,7 +19,8 @@ public:
WakeWordDetect();
~WakeWordDetect();
void Feed(const int16_t* data, int size);
void Initialize(int channels, bool reference);
void Feed(std::vector<int16_t>& data);
void OnWakeWordDetected(std::function<void()> callback);
void OnVadStateChange(std::function<void(bool speaking)> callback);
void StartDetection();
@@ -36,6 +37,8 @@ private:
std::function<void()> wake_word_detected_callback_;
std::function<void(bool speaking)> vad_state_change_callback_;
bool is_speaking_ = false;
int channels_;
bool reference_;
TaskHandle_t wake_word_encode_task_ = nullptr;
StaticTask_t wake_word_encode_task_buffer_;

103
main/WifiBoard.cc Normal file
View File

@@ -0,0 +1,103 @@
#include "WifiBoard.h"
#include "Application.h"
#include "WifiStation.h"
#include "WifiConfigurationAp.h"
#include "SystemInfo.h"
#include "BuiltinLed.h"
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#include <EspHttp.h>
#include <TcpTransport.h>
#include <TlsTransport.h>
#include <WebSocket.h>
#include <esp_log.h>
static const char *TAG = "WifiBoard";
static std::string rssi_to_string(int rssi) {
if (rssi >= -55) {
return "Very good";
} else if (rssi >= -65) {
return "Good";
} else if (rssi >= -75) {
return "Fair";
} else if (rssi >= -85) {
return "Poor";
} else {
return "No network";
}
}
void WifiBoard::StartNetwork() {
auto& application = Application::GetInstance();
auto& display = application.GetDisplay();
auto& builtin_led = BuiltinLed::GetInstance();
// Try to connect to WiFi, if failed, launch the WiFi configuration AP
auto& wifi_station = WifiStation::GetInstance();
display.SetText(std::string("Connect to WiFi\n") + wifi_station.GetSsid());
wifi_station.Start();
if (!wifi_station.IsConnected()) {
application.Alert("Info", "Configuring WiFi");
builtin_led.SetBlue();
builtin_led.Blink(1000, 500);
auto& wifi_ap = WifiConfigurationAp::GetInstance();
wifi_ap.SetSsidPrefix("Xiaozhi");
wifi_ap.Start();
// Wait forever until reset after configuration
while (true) {
vTaskDelay(pdMS_TO_TICKS(1000));
}
}
}
void WifiBoard::Initialize() {
ESP_LOGI(TAG, "Initializing WifiBoard");
}
Http* WifiBoard::CreateHttp() {
return new EspHttp();
}
WebSocket* WifiBoard::CreateWebSocket() {
std::string url = CONFIG_WEBSOCKET_URL;
if (url.find("wss://") == 0) {
return new WebSocket(new TlsTransport());
} else {
return new WebSocket(new TcpTransport());
}
}
bool WifiBoard::GetNetworkState(std::string& network_name, int& signal_quality, std::string& signal_quality_text) {
if (wifi_config_mode_) {
auto& wifi_ap = WifiConfigurationAp::GetInstance();
network_name = wifi_ap.GetSsid();
signal_quality = -99;
signal_quality_text = wifi_ap.GetWebServerUrl();
return true;
}
auto& wifi_station = WifiStation::GetInstance();
if (!wifi_station.IsConnected()) {
return false;
}
network_name = wifi_station.GetSsid();
signal_quality = wifi_station.GetRssi();
signal_quality_text = rssi_to_string(signal_quality);
return signal_quality != -1;
}
std::string WifiBoard::GetJson() {
// Set the board type for OTA
auto& wifi_station = WifiStation::GetInstance();
std::string board_type = BOARD_TYPE;
std::string board_json = std::string("{\"type\":\"" + board_type + "\",");
if (!wifi_config_mode_) {
board_json += "\"ssid\":\"" + wifi_station.GetSsid() + "\",";
board_json += "\"rssi\":" + std::to_string(wifi_station.GetRssi()) + ",";
board_json += "\"channel\":" + std::to_string(wifi_station.GetChannel()) + ",";
board_json += "\"ip\":\"" + wifi_station.GetIpAddress() + "\",";
}
board_json += "\"mac\":\"" + SystemInfo::GetMacAddress() + "\"}";
return board_json;
}

19
main/WifiBoard.h Normal file
View File

@@ -0,0 +1,19 @@
#ifndef WIFI_BOARD_H
#define WIFI_BOARD_H
#include "Board.h"
class WifiBoard : public Board {
protected:
bool wifi_config_mode_ = false;
public:
virtual void Initialize() override;
virtual void StartNetwork() override;
virtual Http* CreateHttp() override;
virtual WebSocket* CreateWebSocket() override;
virtual bool GetNetworkState(std::string& network_name, int& signal_quality, std::string& signal_quality_text) override;
virtual std::string GetJson() override;
};
#endif // WIFI_BOARD_H

BIN
main/assets/err_pin.p3 Normal file

Binary file not shown.

BIN
main/assets/err_reg.p3 Normal file

Binary file not shown.

Binary file not shown.

View File

@@ -0,0 +1,22 @@
#include "Ml307Board.h"
#include "SystemReset.h"
#include <esp_log.h>
#define TAG "CompactMl307Board"
class CompactMl307Board : public Ml307Board {
public:
virtual void Initialize() override {
ESP_LOGI(TAG, "Initializing CompactMl307Board");
// Check if the reset button is pressed
SystemReset::GetInstance().CheckButtons();
Ml307Board::Initialize();
}
virtual AudioDevice* CreateAudioDevice() override {
return new AudioDevice();
}
};
DECLARE_BOARD(CompactMl307Board);

View File

@@ -0,0 +1,46 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
#define AUDIO_INPUT_SAMPLE_RATE 16000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_I2S_METHOD_SIMPLEX
#ifdef AUDIO_I2S_METHOD_SIMPLEX
#define AUDIO_I2S_MIC_GPIO_WS GPIO_NUM_4
#define AUDIO_I2S_MIC_GPIO_SCK GPIO_NUM_5
#define AUDIO_I2S_MIC_GPIO_DIN GPIO_NUM_6
#define AUDIO_I2S_SPK_GPIO_DOUT GPIO_NUM_7
#define AUDIO_I2S_SPK_GPIO_BCLK GPIO_NUM_15
#define AUDIO_I2S_SPK_GPIO_LRCK GPIO_NUM_16
#else
#define AUDIO_I2S_GPIO_LRCK GPIO_NUM_4
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_5
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_6
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_7
#endif
#define BUILTIN_LED_GPIO GPIO_NUM_48
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define VOLUME_UP_BUTTON_GPIO GPIO_NUM_40
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_39
#define DISPLAY_SDA_PIN GPIO_NUM_41
#define DISPLAY_SCL_PIN GPIO_NUM_42
#define DISPLAY_WIDTH 128
#define DISPLAY_HEIGHT 32
#define DISPLAY_MIRROR_X true
#define DISPLAY_MIRROR_Y true
#define ML307_RX_PIN GPIO_NUM_11
#define ML307_TX_PIN GPIO_NUM_12
#endif // _BOARD_CONFIG_H_

View File

@@ -0,0 +1,22 @@
#include "WifiBoard.h"
#include "SystemReset.h"
#include <esp_log.h>
#define TAG "CompactWifiBoard"
class CompactWifiBoard : public WifiBoard {
public:
virtual void Initialize() override {
ESP_LOGI(TAG, "Initializing CompactWifiBoard");
// Check if the reset button is pressed
SystemReset::GetInstance().CheckButtons();
WifiBoard::Initialize();
}
virtual AudioDevice* CreateAudioDevice() override {
return new AudioDevice();
}
};
DECLARE_BOARD(CompactWifiBoard);

View File

@@ -0,0 +1,42 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
#define AUDIO_INPUT_SAMPLE_RATE 16000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_I2S_METHOD_SIMPLEX
#ifdef AUDIO_I2S_METHOD_SIMPLEX
#define AUDIO_I2S_MIC_GPIO_WS GPIO_NUM_4
#define AUDIO_I2S_MIC_GPIO_SCK GPIO_NUM_5
#define AUDIO_I2S_MIC_GPIO_DIN GPIO_NUM_6
#define AUDIO_I2S_SPK_GPIO_DOUT GPIO_NUM_7
#define AUDIO_I2S_SPK_GPIO_BCLK GPIO_NUM_15
#define AUDIO_I2S_SPK_GPIO_LRCK GPIO_NUM_16
#else
#define AUDIO_I2S_GPIO_LRCK GPIO_NUM_4
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_5
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_6
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_7
#endif
#define BUILTIN_LED_GPIO GPIO_NUM_48
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define VOLUME_UP_BUTTON_GPIO GPIO_NUM_40
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_39
#define DISPLAY_SDA_PIN GPIO_NUM_41
#define DISPLAY_SCL_PIN GPIO_NUM_42
#define DISPLAY_WIDTH 128
#define DISPLAY_HEIGHT 32
#define DISPLAY_MIRROR_X true
#define DISPLAY_MIRROR_Y true
#endif // _BOARD_CONFIG_H_

View File

@@ -0,0 +1,20 @@
#include "WifiBoard.h"
#include "BoxAudioDevice.h"
#include <esp_log.h>
#define TAG "EspBox3Board"
class EspBox3Board : public WifiBoard {
public:
virtual void Initialize() override {
ESP_LOGI(TAG, "Initializing EspBox3Board");
WifiBoard::Initialize();
}
virtual AudioDevice* CreateAudioDevice() override {
return new BoxAudioDevice();
}
};
DECLARE_BOARD(EspBox3Board);

View File

@@ -0,0 +1,36 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
#define AUDIO_INPUT_SAMPLE_RATE 16000
#define AUDIO_OUTPUT_SAMPLE_RATE 16000
#define AUDIO_INPUT_REFERENCE true
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_2
#define AUDIO_I2S_GPIO_LRCK GPIO_NUM_45
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_17
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_16
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_15
#define AUDIO_CODEC_PA_PIN GPIO_NUM_46
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_8
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_18
#define AUDIO_CODEC_ES8311_ADDR ES8311_CODEC_DEFAULT_ADDR
#define AUDIO_CODEC_ES7210_ADDR ES7210_CODEC_DEFAULT_ADDR
#define BUILTIN_LED_GPIO GPIO_NUM_NC
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define VOLUME_UP_BUTTON_GPIO GPIO_NUM_NC
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_NC
#define DISPLAY_SDA_PIN GPIO_NUM_NC
#define DISPLAY_SCL_PIN GPIO_NUM_NC
#define DISPLAY_WIDTH 128
#define DISPLAY_HEIGHT 64
#define DISPLAY_MIRROR_X true
#define DISPLAY_MIRROR_Y true
#endif // _BOARD_CONFIG_H_

View File

@@ -0,0 +1,84 @@
#include "Ml307Board.h"
#include "BoxAudioDevice.h"
#include <esp_log.h>
#include <esp_spiffs.h>
#include <driver/gpio.h>
#include <esp_adc/adc_oneshot.h>
#include <esp_adc/adc_cali.h>
#include <esp_adc/adc_cali_scheme.h>
static const char *TAG = "KevinBoxBoard";
class KevinBoxBoard : public Ml307Board {
private:
adc_oneshot_unit_handle_t adc1_handle_;
adc_cali_handle_t adc1_cali_handle_;
void MountStorage() {
// Mount the storage partition
esp_vfs_spiffs_conf_t conf = {
.base_path = "/storage",
.partition_label = "storage",
.max_files = 5,
.format_if_mount_failed = true,
};
esp_vfs_spiffs_register(&conf);
}
void Enable4GModule() {
// Make GPIO15 HIGH to enable the 4G module
gpio_config_t ml307_enable_config = {
.pin_bit_mask = (1ULL << 15),
.mode = GPIO_MODE_OUTPUT,
.pull_up_en = GPIO_PULLUP_DISABLE,
.pull_down_en = GPIO_PULLDOWN_DISABLE,
.intr_type = GPIO_INTR_DISABLE,
};
gpio_config(&ml307_enable_config);
gpio_set_level(GPIO_NUM_15, 1);
}
virtual void InitializeADC() {
adc_oneshot_unit_init_cfg_t init_config1 = {};
init_config1.unit_id = ADC_UNIT_1;
ESP_ERROR_CHECK(adc_oneshot_new_unit(&init_config1, &adc1_handle_));
//-------------ADC1 Config---------------//
adc_oneshot_chan_cfg_t config = {
.atten = ADC_ATTEN_DB_12,
.bitwidth = ADC_BITWIDTH_DEFAULT,
};
ESP_ERROR_CHECK(adc_oneshot_config_channel(adc1_handle_, ADC_CHANNEL_0, &config));
adc_cali_curve_fitting_config_t cali_config = {
.unit_id = ADC_UNIT_1,
.chan = ADC_CHANNEL_0,
.atten = ADC_ATTEN_DB_12,
.bitwidth = ADC_BITWIDTH_DEFAULT,
};
ESP_ERROR_CHECK(adc_cali_create_scheme_curve_fitting(&cali_config, &adc1_cali_handle_));
}
public:
virtual void Initialize() override {
ESP_LOGI(TAG, "Initializing KevinBoxBoard");
InitializeADC();
MountStorage();
Enable4GModule();
Ml307Board::Initialize();
}
virtual AudioDevice* CreateAudioDevice() override {
return new BoxAudioDevice();
}
virtual bool GetBatteryVoltage(int &voltage, bool& charging) override {
int adc_reading;
ESP_ERROR_CHECK(adc_oneshot_read(adc1_handle_, ADC_CHANNEL_0, &adc_reading));
ESP_ERROR_CHECK(adc_cali_raw_to_voltage(adc1_cali_handle_, adc_reading, &voltage));
charging = false;
return true;
}
};
DECLARE_BOARD(KevinBoxBoard);

View File

@@ -0,0 +1,39 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
#define AUDIO_INPUT_SAMPLE_RATE 24000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_INPUT_REFERENCE true
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_0
#define AUDIO_I2S_GPIO_LRCK GPIO_NUM_47
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_48
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_45
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_21
#define AUDIO_CODEC_PA_PIN GPIO_NUM_40
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_39
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_38
#define AUDIO_CODEC_ES8311_ADDR ES8311_CODEC_DEFAULT_ADDR
#define AUDIO_CODEC_ES7210_ADDR ES7210_CODEC_DEFAULT_ADDR
#define BUILTIN_LED_GPIO GPIO_NUM_8
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define VOLUME_UP_BUTTON_GPIO GPIO_NUM_6
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_7
#define DISPLAY_SDA_PIN GPIO_NUM_4
#define DISPLAY_SCL_PIN GPIO_NUM_5
#define DISPLAY_WIDTH 128
#define DISPLAY_HEIGHT 64
#define DISPLAY_MIRROR_X true
#define DISPLAY_MIRROR_Y true
#define ML307_RX_PIN GPIO_NUM_17
#define ML307_TX_PIN GPIO_NUM_16
#endif // _BOARD_CONFIG_H_

View File

@@ -0,0 +1,93 @@
#include "Ml307Board.h"
#include "BoxAudioDevice.h"
#include <esp_log.h>
#include <esp_spiffs.h>
#include <driver/gpio.h>
#include <esp_adc/adc_oneshot.h>
#include <esp_adc/adc_cali.h>
#include <esp_adc/adc_cali_scheme.h>
static const char *TAG = "KevinBoxBoard";
class KevinBoxBoard : public Ml307Board {
private:
adc_oneshot_unit_handle_t adc1_handle_;
adc_cali_handle_t adc1_cali_handle_;
void MountStorage() {
// Mount the storage partition
esp_vfs_spiffs_conf_t conf = {
.base_path = "/storage",
.partition_label = "storage",
.max_files = 5,
.format_if_mount_failed = true,
};
esp_vfs_spiffs_register(&conf);
}
void Enable4GModule() {
// Make GPIO15 HIGH to enable the 4G module
gpio_config_t ml307_enable_config = {
.pin_bit_mask = (1ULL << 15),
.mode = GPIO_MODE_OUTPUT,
.pull_up_en = GPIO_PULLUP_DISABLE,
.pull_down_en = GPIO_PULLDOWN_DISABLE,
.intr_type = GPIO_INTR_DISABLE,
};
gpio_config(&ml307_enable_config);
gpio_set_level(GPIO_NUM_15, 1);
}
virtual void InitializeADC() {
adc_oneshot_unit_init_cfg_t init_config1 = {};
init_config1.unit_id = ADC_UNIT_1;
ESP_ERROR_CHECK(adc_oneshot_new_unit(&init_config1, &adc1_handle_));
//-------------ADC1 Config---------------//
adc_oneshot_chan_cfg_t config = {
.atten = ADC_ATTEN_DB_12,
.bitwidth = ADC_BITWIDTH_DEFAULT,
};
ESP_ERROR_CHECK(adc_oneshot_config_channel(adc1_handle_, ADC_CHANNEL_0, &config));
adc_cali_curve_fitting_config_t cali_config = {
.unit_id = ADC_UNIT_1,
.chan = ADC_CHANNEL_0,
.atten = ADC_ATTEN_DB_12,
.bitwidth = ADC_BITWIDTH_DEFAULT,
};
ESP_ERROR_CHECK(adc_cali_create_scheme_curve_fitting(&cali_config, &adc1_cali_handle_));
}
public:
virtual void Initialize() override {
ESP_LOGI(TAG, "Initializing KevinBoxBoard");
InitializeADC();
MountStorage();
Enable4GModule();
gpio_config_t charging_io = {
.pin_bit_mask = (1ULL << 2),
.mode = GPIO_MODE_INPUT,
.pull_up_en = GPIO_PULLUP_ENABLE,
.pull_down_en = GPIO_PULLDOWN_DISABLE,
.intr_type = GPIO_INTR_DISABLE,
};
gpio_config(&charging_io);
Ml307Board::Initialize();
}
virtual AudioDevice* CreateAudioDevice() override {
return new BoxAudioDevice();
}
virtual bool GetBatteryVoltage(int &voltage, bool& charging) override {
ESP_ERROR_CHECK(adc_oneshot_get_calibrated_result(adc1_handle_, adc1_cali_handle_, ADC_CHANNEL_0, &voltage));
charging = gpio_get_level(GPIO_NUM_2) == 0;
ESP_LOGI(TAG, "Battery voltage: %d, Charging: %d", voltage, charging);
return true;
}
};
DECLARE_BOARD(KevinBoxBoard);

View File

@@ -0,0 +1,39 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
#define AUDIO_INPUT_SAMPLE_RATE 24000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_INPUT_REFERENCE true
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_42
#define AUDIO_I2S_GPIO_LRCK GPIO_NUM_47
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_48
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_45
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_21
#define AUDIO_CODEC_PA_PIN GPIO_NUM_17
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_39
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_38
#define AUDIO_CODEC_ES8311_ADDR ES8311_CODEC_DEFAULT_ADDR
#define AUDIO_CODEC_ES7210_ADDR ES7210_CODEC_DEFAULT_ADDR
#define BUILTIN_LED_GPIO GPIO_NUM_8
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define VOLUME_UP_BUTTON_GPIO GPIO_NUM_6
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_7
#define DISPLAY_SDA_PIN GPIO_NUM_4
#define DISPLAY_SCL_PIN GPIO_NUM_5
#define DISPLAY_WIDTH 128
#define DISPLAY_HEIGHT 64
#define DISPLAY_MIRROR_X false
#define DISPLAY_MIRROR_Y false
#define ML307_RX_PIN GPIO_NUM_20
#define ML307_TX_PIN GPIO_NUM_19
#endif // _BOARD_CONFIG_H_

View File

@@ -0,0 +1,20 @@
#include "WifiBoard.h"
#include "BoxAudioDevice.h"
#include <esp_log.h>
#define TAG "LiChuangDevBoard"
class LiChuangDevBoard : public WifiBoard {
public:
virtual void Initialize() override {
ESP_LOGI(TAG, "Initializing LiChuangDevBoard");
WifiBoard::Initialize();
}
virtual AudioDevice* CreateAudioDevice() override {
return new BoxAudioDevice();
}
};
DECLARE_BOARD(LiChuangDevBoard);

View File

@@ -0,0 +1,37 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
#define AUDIO_INPUT_SAMPLE_RATE 24000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_INPUT_REFERENCE true
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_38
#define AUDIO_I2S_GPIO_LRCK GPIO_NUM_13
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_14
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_12
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_45
#define AUDIO_CODEC_USE_PCA9557
#define AUDIO_CODEC_PA_PIN GPIO_NUM_40
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_1
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_2
#define AUDIO_CODEC_ES8311_ADDR ES8311_CODEC_DEFAULT_ADDR
#define AUDIO_CODEC_ES7210_ADDR 0x82
#define BUILTIN_LED_GPIO GPIO_NUM_48
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define VOLUME_UP_BUTTON_GPIO GPIO_NUM_NC
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_NC
#define DISPLAY_SDA_PIN GPIO_NUM_NC
#define DISPLAY_SCL_PIN GPIO_NUM_NC
#define DISPLAY_WIDTH 128
#define DISPLAY_HEIGHT 64
#define DISPLAY_MIRROR_X false
#define DISPLAY_MIRROR_Y false
#endif // _BOARD_CONFIG_H_

View File

@@ -1,13 +1,14 @@
## IDF Component Manager Manifest File
dependencies:
78/esp-builtin-led: "^1.0.2"
78/esp-wifi-connect: "^1.1.0"
78/esp-opus-encoder: "^1.0.2"
78/esp-ml307: "^1.2.1"
78/esp-wifi-connect: "~1.2.0"
78/esp-opus-encoder: "~1.0.2"
78/esp-ml307: "~1.3.0"
espressif/led_strip: "^2.4.1"
espressif/esp_codec_dev: "^1.3.1"
espressif/esp-sr: "^1.9.0"
espressif/button: "^3.3.1"
lvgl/lvgl: "^8.4.0"
esp_lvgl_port: "^1.4.0"
lvgl/lvgl: "~8.4.0"
esp_lvgl_port: "~2.4.1"
## Required IDF version
idf:
version: ">=5.3"

View File

@@ -7,16 +7,11 @@
#include "Application.h"
#include "SystemInfo.h"
#include "SystemReset.h"
#define TAG "main"
extern "C" void app_main(void)
{
// Check if the reset button is pressed
SystemReset system_reset;
system_reset.CheckButtons();
// Initialize the default event loop
ESP_ERROR_CHECK(esp_event_loop_create_default());

68
pack.py
View File

@@ -1,68 +0,0 @@
#! /usr/bin/env python3
import csv
import os
# 例如1000, 0x1000, 1M
def read_value(text):
text = text.strip()
if text.endswith('K'):
return int(text[:-1]) * 1024
elif text.endswith('M'):
return int(text[:-1]) * 1024 * 1024
else:
if text.startswith('0x'):
return int(text, 16)
else:
return int(text)
def write_bin(image_data, offset, file_path, max_size=None):
# Read file_path and write to image_data
with open(file_path, 'rb') as f:
data = f.read()
if max_size is not None:
assert len(data) <= max_size, f"Data from {file_path} is too large"
image_data[offset:offset+len(data)] = data
print(f"Write {os.path.basename(file_path)} to 0x{offset:08X} with size 0x{len(data):08X}")
'''
根据 partitions.csv 文件,把 bin 文件打包成一个 4MB 的 image 文件,方便烧录
'''
def pack_firmware_image():
# Create a 4MB image filled with 0xFF
image_size = 4 * 1024 * 1024
image_data = bytearray([0xFF] * image_size)
build_dir = os.path.join(os.path.dirname(__file__), 'build')
write_bin(image_data, 0, os.path.join(build_dir, 'bootloader', 'bootloader.bin'))
write_bin(image_data, 0x8000, os.path.join(build_dir, 'partition_table', 'partition-table.bin'))
# 读取 partitions.csv 文件
with open('partitions.csv', 'r') as f:
reader = csv.reader(f)
for row in reader:
if row[0] == 'model':
file_path = os.path.join(build_dir, 'srmodels', 'srmodels.bin')
elif row[0] == 'factory':
file_path = os.path.join(build_dir, 'xiaozhi.bin')
else:
continue
offset = read_value(row[3])
size = read_value(row[4])
write_bin(image_data, offset, file_path, size)
# 写入 image 文件
output_path = os.path.join(build_dir, 'xiaozhi.img')
with open(output_path, 'wb') as f:
f.write(image_data)
print(f"Image file {output_path} created with size 0x{len(image_data):08X}")
# Compress image with zip without directory
os.system(f"zip -j {output_path}.zip {output_path}")
if __name__ == '__main__':
pack_firmware_image()

View File

@@ -4,6 +4,7 @@ nvs, data, nvs, 0x9000, 0x4000,
otadata, data, ota, 0xd000, 0x2000,
phy_init, data, phy, 0xf000, 0x1000,
model, data, spiffs, 0x10000, 0xF0000,
storage, data, spiffs, 0x100000, 1M,
factory, app, factory, 0x200000, 4M,
ota_0, app, ota_0, 0x600000, 4M,
ota_1, app, ota_1, 0xA00000, 4M,
1 # ESP-IDF Partition Table
4 otadata, data, ota, 0xd000, 0x2000,
5 phy_init, data, phy, 0xf000, 0x1000,
6 model, data, spiffs, 0x10000, 0xF0000,
7 storage, data, spiffs, 0x100000, 1M,
8 factory, app, factory, 0x200000, 4M,
9 ota_0, app, ota_0, 0x600000, 4M,
10 ota_1, app, ota_1, 0xA00000, 4M,

View File

@@ -1,51 +0,0 @@
#! /usr/bin/env python3
from dotenv import load_dotenv
load_dotenv()
import os
import oss2
import json
def get_version():
with open('CMakeLists.txt', 'r') as f:
for line in f:
if line.startswith('set(PROJECT_VER'):
return line.split('"')[1]
return '0.0.0'
def upload_bin_to_oss(bin_path, oss_key):
auth = oss2.Auth(os.environ['OSS_ACCESS_KEY_ID'], os.environ['OSS_ACCESS_KEY_SECRET'])
bucket = oss2.Bucket(auth, os.environ['OSS_ENDPOINT'], os.environ['OSS_BUCKET_NAME'])
bucket.put_object(oss_key, open(bin_path, 'rb'))
if __name__ == '__main__':
# 获取版本号
version = get_version()
print(f'version: {version}')
# 上传 bin 文件到 OSS
upload_bin_to_oss('build/xiaozhi.bin', f'firmwares/xiaozhi-{version}.bin')
# File URL
file_url = os.path.join(os.environ['OSS_BUCKET_URL'], f'firmwares/xiaozhi-{version}.bin')
print(f'Uploaded bin to OSS: {file_url}')
firmware_json = {
"version": version,
"url": file_url
}
with open(f"build/firmware.json", "w") as f:
json.dump(firmware_json, f, indent=4)
# copy firmware.json to server
firmware_config_path = os.environ['FIRMWARE_CONFIG_PATH']
ret = os.system(f'scp build/firmware.json {firmware_config_path}')
if ret != 0:
print(f'Failed to copy firmware.json to server')
exit(1)
print(f'Copied firmware.json to server: {firmware_config_path}')

158
versions.py Normal file
View File

@@ -0,0 +1,158 @@
#! /usr/bin/env python3
from dotenv import load_dotenv
load_dotenv()
import os
import struct
import zipfile
import oss2
import json
def get_chip_id_string(chip_id):
return {
0x0000: "esp32",
0x0002: "esp32s2",
0x0005: "esp32c3",
0x0009: "esp32s3",
0x000C: "esp32c2",
0x000D: "esp32c6",
0x0010: "esp32h2",
0x0011: "esp32c5",
0x0012: "esp32p4",
0x0017: "esp32c5",
}[chip_id]
def get_flash_size(flash_size):
MB = 1024 * 1024
return {
0x00: 1 * MB,
0x01: 2 * MB,
0x02: 4 * MB,
0x03: 8 * MB,
0x04: 16 * MB,
0x05: 32 * MB,
0x06: 64 * MB,
0x07: 128 * MB,
}[flash_size]
def get_app_desc(data):
magic = struct.unpack("<I", data[0x00:0x04])[0]
if magic != 0xabcd5432:
raise Exception("Invalid app desc magic")
version = data[0x10:0x30].decode("utf-8").strip('\0')
project_name = data[0x30:0x50].decode("utf-8").strip('\0')
time = data[0x50:0x60].decode("utf-8").strip('\0')
date = data[0x60:0x70].decode("utf-8").strip('\0')
idf_ver = data[0x70:0x90].decode("utf-8").strip('\0')
elf_sha256 = data[0x90:0xb0].hex()
return {
"name": project_name,
"version": version,
"compile_time": date + "T" + time,
"idf_version": idf_ver,
"elf_sha256": elf_sha256,
}
def get_board_name(folder):
basename = os.path.basename(folder)
if basename.startswith("v0.2"):
return "bread-simple"
if basename.startswith("v0.3") or basename.startswith("v0.4") or basename.startswith("v0.5") or basename.startswith("v0.6"):
if "ML307" in basename:
return "bread-compact-ml307"
else:
return "bread-compact-wifi"
raise Exception(f"Unknown board name: {basename}")
def read_binary(dir_path):
merged_bin_path = os.path.join(dir_path, "merged-binary.bin")
data = open(merged_bin_path, "rb").read()[0x200000:]
if data[0] != 0xE9:
print(dir_path, "is not a valid image")
return
# get flash size
flash_size = get_flash_size(data[0x3] >> 4)
chip_id = get_chip_id_string(data[0xC])
# get segments
segment_count = data[0x1]
segments = []
offset = 0x18
for i in range(segment_count):
segment_size = struct.unpack("<I", data[offset + 4:offset + 8])[0]
offset += 8
segment_data = data[offset:offset + segment_size]
offset += segment_size
segments.append(segment_data)
assert offset < len(data), "offset is out of bounds"
# extract bin file
bin_path = os.path.join(dir_path, "xiaozhi.bin")
if not os.path.exists(bin_path):
print("extract bin file to", bin_path)
open(bin_path, "wb").write(data)
# The app desc is in the first segment
desc = get_app_desc(segments[0])
return {
"chip_id": chip_id,
"flash_size": flash_size,
"board": get_board_name(dir_path),
"application": desc,
}
def extract_zip(zip_path, extract_path):
if not os.path.exists(extract_path):
os.makedirs(extract_path)
print(f"Extracting {zip_path} to {extract_path}")
with zipfile.ZipFile(zip_path, 'r') as zip_ref:
zip_ref.extractall(extract_path)
def upload_dir_to_oss(source_dir, target_dir):
auth = oss2.Auth(os.environ['OSS_ACCESS_KEY_ID'], os.environ['OSS_ACCESS_KEY_SECRET'])
bucket = oss2.Bucket(auth, os.environ['OSS_ENDPOINT'], os.environ['OSS_BUCKET_NAME'])
for filename in os.listdir(source_dir):
oss_key = os.path.join(target_dir, filename)
print('uploading', oss_key)
bucket.put_object(oss_key, open(os.path.join(source_dir, filename), 'rb'))
def main():
release_dir = "releases"
versions = []
# look for zip files startswith "v"
for name in os.listdir(release_dir):
if name.startswith("v") and name.endswith(".zip"):
tag = name[:-4]
folder = os.path.join(release_dir, tag)
if not os.path.exists(folder):
os.makedirs(folder)
extract_zip(os.path.join(release_dir, name), folder)
info = read_binary(folder)
target_dir = os.path.join("firmwares", tag)
info["tag"] = tag
info["url"] = os.path.join(os.environ['OSS_BUCKET_URL'], target_dir, "xiaozhi.bin")
open(os.path.join(folder, "info.json"), "w").write(json.dumps(info, indent=4))
# upload all file to oss
upload_dir_to_oss(folder, target_dir)
# read info.json
info = json.load(open(os.path.join(folder, "info.json")))
versions.append(info)
# sort versions by version
versions.sort(key=lambda x: x["tag"], reverse=True)
# write versions to file
versions_path = os.path.join(release_dir, "versions.json")
open(versions_path, "w").write(json.dumps(versions, indent=4))
print(f"Versions written to {versions_path}")
# copy versions.json to server
versions_config_path = os.environ['VERSIONS_CONFIG_PATH']
ret = os.system(f'scp {versions_path} {versions_config_path}')
if ret != 0:
print(f'Failed to copy versions.json to server')
exit(1)
print(f'Copied versions.json to server: {versions_config_path}')
if __name__ == "__main__":
main()

View File

@@ -1,160 +0,0 @@
# AI 语音交互通信协议文档
## 1. 连接建立与鉴权
客户端通过 WebSocket 连接到服务器时,需要在 HTTP 头中包含以下信息:
- `Authorization`: Bearer token格式为 "Bearer <access_token>"
- `Device-Id`: 设备 MAC 地址
- `Protocol-Version`: 协议版本号,当前为 2
WebSocket URL: `wss://api.tenclass.net/xiaozhi/v1`
## 2. 二进制数据
客户端发送的二进制数据使用固定头格式的协议,如下:
```cpp
struct BinaryProtocol {
uint16_t version; // 二进制协议版本,当前为 2
uint16_t type; // 消息类型0音频流数据1JSON
uint32_t reserved; // 保留字段
uint32_t timestamp; // 时间戳保留用作回声消除也可以用于UDP不可靠传输中的排序
uint32_t payload_size; // 负载大小
uint8_t payload[]; // 可以是音频数据Opus 编码或协商的音频格式),也可以封装 JSON
} __attribute__((packed));
```
注意:所有多字节整数字段使用网络字节序(大端序)。
目前二进制数据跟 JSON 都是走同一个 WebSocket 连接,未来实时对话模式下,二进制音频数据可能走 UDP可以扩展 hello 消息进行协商。
## 3. 音频数据传输
- 客户端到服务器: 使用二进制协议发送 Opus 编码的音频数据
- 服务器到客户端: 使用二进制协议发送 Opus 编码的音频数据,格式与客户端发送的相同
出现 payload_size 为 0 的音频数据包可以用做句子边界标记,可以忽略,但不要报错。
## 4. 握手消息
连接建立后,客户端发送一个 JSON 格式的 "hello" 消息,初始化服务器端的音频解码器。
不需要等待服务器响应,随后即可发送音频数据。
```json
{
"type": "hello",
"response_mode": "auto",
"audio_params": {
"format": "opus",
"sample_rate": 16000,
"channels": 1
}
}
```
应答模式 `response_mode` 可以为 `auto``manual`
`auto`:自动应答模式,服务器实时计算音频 VAD 并自动决定何时开始应答。
`manual`:手动应答模式,客户端状态从 `listening` 变为 `idle` 时,服务器可以应答。
## 5. 状态更新
客户端在状态变化时发送 JSON 消息:
```json
{
"type": "state",
"state": "<新状态>"
}
```
可能发送的状态值包括: `idle`, `wake_word_detected`, `listening`, `speaking`
示例:
1、按住说话`response_mode``manual`
- 当按住说话按钮时,如果未连接服务器,则连接服务器,并编码、缓存当前音频数据,连接成功后,客户端设置状态为 `listening`,并在 hello 消息之后发送缓存的音频数据。
- 当按住说话按钮时,如果已连接服务器,则客户端设置状态为 `listening`,并发送音频数据。
- 当释放说话按钮时,状态变为 `idle`,此时服务器开始识别。
- 服务器开始应答时,推送 `stt``tts` 消息。
- 客户端开始播放音频时,状态设为 `speaking`
- 客户端结束播放音频时,状态设为 `idle`
-`speaking` 状态下,按住说话按钮,会立即停止当前音频播放,状态变为 `listening`
2、语音唤醒轮流对话`response_mode``auto`
- 连接服务器,发送 hello 消息,发送唤醒词音频数据,然后发送状态 `wake_word_detected`,服务器开始应答。
- 客户端开始播放音频时,状态设为 `speaking`,此时客户端不会发送音频数据。
- 客户端结束播放音频时,状态设为 `listening`,此时客户端发送音频数据。
- 服务器计算音频 VAD 自动选择时机开始应答时,推送 `stt``tts` 消息。
- 客户端收到 `tts`.`start` 时,开始播放音频,状态设为 `speaking`
- 客户端收到 `tts`.`stop` 时,停止播放音频,状态设为 `listening`
3、语音唤醒实时对话`response_mode``real_time`
- 连接服务器,发送 hello 消息,发送唤醒词音频数据,然后发送状态 `wake_word_detected`,服务器开始应答。
- 客户端开始播放音频时,状态设为 `speaking`
- 客户端结束播放音频时,状态设为 `listening`
-`speaking``listening` 状态下,客户端都会发送音频数据。
- 服务器计算音频 VAD 自动选择时机开始应答时,推送 `stt``tts` 消息。
- 客户端收到 `stt` 时,状态设为 `listening`。如果当前有音频正在播放,则在当前 sentence 结束后停止播放音频。
- 客户端收到 `tts`.`start` 时,开始播放音频,状态设为 `speaking`
- 客户端收到 `tts`.`stop` 时,停止播放音频,状态设为 `listening`
## 6. 服务器到客户端的消息
### 6.1 语音识别结果 (STT)
```json
{
"type": "stt",
"text": "<识别出的文本>"
}
```
### 6.2 文本转语音 (TTS)
TTS开始:
```json
{
"type": "tts",
"state": "start",
"sample_rate": 24000
}
```
句子开始:
```json
{
"type": "tts",
"state": "sentence_start",
"text": "你在干什么呀?"
}
```
句子结束:
```json
{
"type": "tts",
"state": "sentence_end"
}
```
TTS结束:
```json
{
"type": "tts",
"state": "stop"
}
```
## 7. 连接管理
- 客户端检测到 WebSocket 断开连接时,应该停止音频播放并重置为空闲状态
- 在断开连接后,客户端按需重新发起连接(比如按钮按下或语音唤醒)
这个文档概括了 WebSocket 通信协议的主要方面。