Compare commits

...

275 Commits

Author SHA1 Message Date
Terrence
add271f0c1 set gfx core to cpu1 2025-09-16 01:41:21 +08:00
Terrence
38157aa180 Add device-side AEC to EchoEar 2025-09-16 00:14:01 +08:00
Terrence
1bacf40cd4 Bump to 1.9.2 2025-09-15 23:55:04 +08:00
Terrence
99aa15822b 开机启动显示开发板信息,提前启动event loop 2025-09-15 23:16:08 +08:00
Terrence
1ffc5190b6 fix: esp_emote_gfx compiling errors 2025-09-08 08:01:20 +08:00
Terrence
73dbeb4b9a update surfer-c3-1.14tft font size 2025-09-05 11:57:46 +08:00
Terrence
1e94e884b8 fix: c3 stack protection error, remove lvgl jpg library 2025-09-05 11:37:31 +08:00
Terrence
b35bf0c344 fix compiling errors 2025-09-04 13:47:02 +08:00
Xiaoxia
5d3f597137 Bump to v1.9.0 (#1157)
* update v2 partition table readme

* feat: Add user only tool

* Add image cache

* smaller cache and buffer, more heap

* use MAIN_EVENT_CLOCK_TICK to avoid audio glitches

* fix: esp_psram_get_size not found in c3

* Bump to 1.9.0
2025-09-04 12:30:26 +08:00
ggc121238
3e37551923 add waveshare-s3-audio-board (#1139)
* add waveshare-s3-audio-board

* Modify the product link in the readme of the waveshare esp32s3-audio-board

* Modify reset time
2025-09-01 10:50:39 +08:00
Xiaoxia
d09537ed5c Add V2 parition tables (#1137)
* v1.9.0: update font icons, add mqtt reconnect

* Add v2 parition tables
2025-08-29 09:04:23 +08:00
Create123
86921f4862 Add M5Stack AtomEchoS3R Board. (#1123)
Signed-off-by: hlym123 <lwylwt@qq.com>
2025-08-26 14:05:10 +08:00
Terrence
7af366b7b2 fix: ES7120_SEL_MIC1 => ES7210_SEL_MIC1 2025-08-23 16:05:49 +08:00
Xiaoxia
ddbb24942d v1.8.9: Upgrade component versions (#1118) 2025-08-23 07:12:14 +08:00
Ben
610a4a0703 Update README.md (#1115)
delete '的'
2025-08-22 18:49:26 +08:00
香草味的纳西妲喵
7cd37427b2 feat: 添加批量转换OGG音频的相关脚本,移动声波配网HTML文件到scripts文件夹下 (#1107)
* feat: 添加批量转换OGG音频的相关脚本,移动声波配网HTML文件到scripts文件夹下

* Rename

* moved README.md
2025-08-22 00:53:18 +08:00
laride
2d772dad68 fix: resolve some audio issues on esp-hi (#1027)
* fix: resolve crash when closing codec dev on esp-hi

* fix: fix incorrect status display in non-zh-CN languages

* fix: reduce noise when not in Speaking state
2025-08-19 11:50:00 +08:00
Terrence
156eb15f58 fix: dual mic without afe 2025-08-16 03:08:00 +08:00
Xiaoxia
c59c515706 v1.8.8: release with esp-sr==2.1.4 and without font placeholder (#1086) 2025-08-15 04:50:33 +08:00
Terrence
44b8d5e4c1 fix: c3 wakeword not working with esp-sr 2.1.5 2025-08-15 01:07:00 +08:00
Xiaoxia
cc07ef447e Revert "camera 优化:在原有的RGB565处理下,容易超时改为JPEG格式 (#1029)" (#1085)
This reverts commit d6b1414967.
2025-08-14 22:23:29 +08:00
Ky1eYang
cf4afde88e add: 添加声音检测的可视化以及声波demod的准确度 (#1077)
Co-authored-by: yangkaiyue <yangkaiyue1@tenclass.com>
2025-08-14 22:11:56 +08:00
Dong Ning
d6b1414967 camera 优化:在原有的RGB565处理下,容易超时改为JPEG格式 (#1029)
* camera 优化

feat(camera): 修改摄像头配置为JPEG格式并优化图像处理逻辑

将摄像头输出格式从RGB565改为JPEG以提高传输效率,同时调整JPEG质量为10
重构预览图像处理逻辑,支持直接处理JPEG格式并自动转换为RGB565
优化Explain方法中的JPEG队列处理,减少内存分配和拷贝操作

* 修复代码缩进

调整代码缩进格式以提升可读性

* fix(esp32_camera): 修复RGB565格式预览图像的字节序问题

添加字节交换处理,将大端序转换为小端序,确保预览图像显示正确

* 使用旧的处理方式

* refactor(esp32_camera): 移除preview_buffer_直接使用preview_image_.data
2025-08-14 22:11:15 +08:00
Terrence
00dd89079b Bump to 1.8.7 2025-08-13 11:50:27 +08:00
vritser
cfb635d870 feat: 适配 ESP-S3-EV-Board-2 开发板 (#1043)
* feat: 适配 ESP-S3-EV-Board-2 开发板

* fix: config name

* fix: ev_board type choice

* fix: remove version config

---------

Co-authored-by: vritser <vritser@gmail.com>
2025-08-12 20:21:18 +08:00
Xiaoxia
9ae34d3b45 feat: Add locales with OGG sounds (#1057)
* fix jiuchuan build problem

* feat: Add locales with OGG sounds

* fix building echoear

* Support ogg files frame duration <= 60
2025-08-12 18:41:00 +08:00
Terrence
08b8b04c6c remove data mutex from read/write 2025-08-12 17:11:57 +08:00
Terrence
f890acfc7c fix i2s race condition error 2025-08-12 14:54:16 +08:00
Terrence
a4fe4d8d99 fix firmware image size 2025-08-12 14:53:17 +08:00
Xiaoxia
593b495139 xmin-c3 share MCP tool and sleep mode can be disabled (#1054) 2025-08-09 03:08:42 +08:00
Xiaoxia
2a02dd65be fix ReadAudioData frame duration (#1051) 2025-08-08 21:50:28 +08:00
virgil
845b760db3 fix: Delete the esp_jpeg_simd component to resolve function conflicts (#1049) 2025-08-08 21:00:34 +08:00
Xiaoxia
f86637cf1c fix: codec unexpectedly closed by timer after open (#1046)
Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
2025-08-08 15:39:12 +08:00
Xiaoxia
363073658a fix: OTA buffer overflow caused by slow SetChatMessage (#1031) 2025-08-06 01:16:52 +08:00
laride
da228f2582 fix: ci idf.py not found (#1030) 2025-08-05 22:18:28 +08:00
laride
36476f05cd feat: add build CI (#1028) 2025-08-05 20:03:37 +08:00
flying1425
90602d3802 waveshare-c6-lcd-1.69 新增电池电量显示功能及电池供电状态下的PWR按键开关机功能。 (#1020)
* 新增电池电量显示功能及电池供电状态下的PWR按键开关机功能。

* 代码风格优化

---------

Co-authored-by: flyingtjy <flyingtjy@gmail.com>
2025-08-04 22:38:13 +08:00
Zxczzzzzzz
b2e1c5bb5c 添加了一个声波配网的HTML界面 (#996)
* Add HTML page for sonic-based Wi-Fi provisioning

* Add HTML page for sonic-based Wi-Fi provisioning

* Add soundwave provisioning UI and fix bugs

* Add soundwave provisioning UI and fix bugs

* Move sonic_wifi_config.html to the docs folder

---------

Co-authored-by: luyuhan <xiaolumylove@icloud.com>
2025-08-02 14:13:35 +08:00
Terrence
d58d55cfe7 Bump to 1.8.5 2025-08-02 01:27:14 +08:00
espressif2022
cd23e0f155 feat: add emote_gfx UI for EchoEar (#1022)
* feat: add emote_gfx UI for EchoEar

* feat: delete local assets
2025-08-01 18:07:13 +08:00
Xiaoxia
26d9ff283f Fix custom wakeword for dual mic (#1018) 2025-08-01 13:30:17 +08:00
Xiaoxia
fb85019c3c change bread-compact-wifi-s3cam test gpio num (#1017) 2025-08-01 05:28:27 +08:00
HonestQiao
4859d57fea 修复esp32-p4配网客户端无法连接、连接获取不到ip或者无法打开配置页面的问题 (#1012) 2025-07-31 05:14:17 +08:00
Xiaoxia
03394fe38d update the sleep time of xmini-c3 with wake word to 300s (#1007) 2025-07-30 15:25:39 +08:00
jake12355
e0e12450c5 jiuchuan-s3修改按键定义取消不对话自动关机修复屏幕显示不全 (#997)
* jiuchuan-s3修改按键定义取消不对话自动关机修复屏幕显示不全

* jiuchuan-s3修改按键定义取消不对话自动关机修复屏幕显示不全
2025-07-29 17:28:40 +08:00
Xiaoxia
e5ac40aac8 fix audio pm (#1004) 2025-07-29 15:25:40 +08:00
Xiaoxia
345c8be467 Add custom wakeword threshold option (#1003) 2025-07-29 10:56:52 +08:00
香草味的纳西妲喵
def6301292 增加其他分辨率表情转换选项 (#987)
* Create requirements.txt

* Update README.md

* add: 增加其他尺寸表情选项
2025-07-26 02:31:09 +08:00
Xiaoxia
df7cbdfcb6 fix esp-hi crashing in esp_codec_dev_close() (#984) 2025-07-25 10:36:56 +08:00
Terrence
d38763d5ef Reduce SRAM usage of audio tasks 2025-07-25 08:27:12 +08:00
LILYGO_L
e90e540933 增强T-CameraPlus-S3麦克风接收音量 (#958)
* Adapt for LilyGO-T-Circle-S3 device

* Adapt for LilyGO-T-Circle-S3 device

* Remove comments and modify the size of the lilygo-t-circle-s3 image

* Modify the code style and format to Google C++

* Modify the code style and format to Google C++

* Fixed bugs in the LILYGO T-Circle-S3 board and added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Merge branch 'main' of https://github.com/Llgok/xiaozhi-esp32

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Fix the color display issue for T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Update T-CameraPlus-S3_V1.2 Version Xiaozhi Example

* Resolve the issue where the camera on the T-CameraPlus-S3_V1.2 board cannot be used normally.

* Enhance microphone reception volume

* fix the issue where voice wake-up is not working

* fix the issue where voice wake-up is not working
2025-07-23 23:02:45 +08:00
Ky1eYang
656bf3c7fa FIX: 修复双声道声波配网失效, 添加屏幕打印SSID/密码 (#971)
* debug: 添加声波配网的log打印点display

* fix: 修复双声道下声波配网失效的问题

* fix: codec可能为nullptr的问题(需要从单例board获取)

* Update afsk_demod.cc

fix coding style

---------

Co-authored-by: yangkaiyue <yangkaiyue1@tenclass.com>
Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-07-23 22:59:07 +08:00
Y1hsiaochunnn
ca35b0761b Update README.md (#968)
The README text description is incorrect. It needs to be corrected.
2025-07-23 22:47:19 +08:00
Terrence
b031a829c0 Bump to 1.8.2 2025-07-22 22:43:41 +08:00
Xiaoxia
3c11cceb43 增加自定义唤醒词启动失败的提升 (#965) 2025-07-22 18:57:25 +08:00
Y1hsiaochunnn
15f233e773 Add compatibility for Waveshare ESP32-S3-Touch-AMOLED-2.06 (#960)
* Add Waveshare ESP32-S3-Touch-AMOLED-2.06

* Update some configuration settings

* Add configuration to the configuration file

* Fix the abnormal areas
2025-07-22 18:02:31 +08:00
Forairaaaaa
721b58f8c7 Fix atoms3 backlight control (#959) 2025-07-22 17:55:33 +08:00
Xiaoxia
eb0bba2c89 Update LICENSE 2025-07-22 10:06:11 +08:00
Surfer
0d45c636a8 Surfer-C3-1.14tft 支持默认离线语音唤醒配置等 (#951)
* 添加 Surfer-C3-1.14TFT开发板支持

添加 Surfer-C3-1.14TFT开发板,基于C3的1.14寸LCD开发板,支持GPIO2  ADC 电池电量检测,应用于wifi Modem-Sleep 和低电量提醒。

* 修改surfer-c3-1.14tff电池电量适配值

修改surfer-c3-1.14tff电池电量适配值,电池容量400ma

* 修改支持默认离线语音唤醒配置等

1. 增加 config.json 文件默认支持离线语音唤醒配置和USB_SERIAL_JTAG
2. 微调电池电量值显示范围参数
2025-07-21 22:13:56 +08:00
Wiking_Xu
3e2bc9ee74 新增正点原子BOX2板子,有WIFI版本和4G版本 (#954) 2025-07-21 22:13:12 +08:00
Terrence
6a7a403117 修复C3唤醒提示音 2025-07-21 14:40:35 +08:00
Terrence
d1c047d060 Update to esp-ml307@3.1.1 2025-07-21 05:46:05 +08:00
Terrence
a35a344f42 Fix frame samples for server AEC 2025-07-20 07:53:52 +08:00
Terrence
efc6f238e7 Fix Server AEC 2025-07-20 03:57:36 +08:00
Xiaoxia
3c71558a5f v1.8.0: Audio 代码重构与低功耗优化 (#943)
* Reconstruct Audio Code

* Remove old IoT implementation

* Add MQTT-UDP documentation

* OTA升级失败时,可以继续使用
2025-07-19 22:45:22 +08:00
laride
0621578f55 fix: fix image player compile error (#933) 2025-07-18 14:08:04 +08:00
laride
5c8707075f feat: add state change events and callbacks (#798) 2025-07-18 01:35:31 +08:00
jake12355
c68c959e9b [优化]-jiuchuan-s3-优化电源管理架构 (#875)
* 添加 jiuchuang-s3 开发板支持

* 增加编译指南

* 优开发板支持包文件目录,更新README.md

删除了多余板载文件
[* ]README.md -> 更新了编译指南和烧录指南

* 修改板级支持包名

* 使用乐鑫提供的电源监测

* 修复部分代码格式问题

* 解决合并冲突

* 解决部分合并内容

* 完善合并内容

* 修复电量映射表错误

* 优化电源管理架构-使用rtc-io电源保持

---------

Co-authored-by: unknown <jake12355>
Co-authored-by: Hangon66 <2630612613@qq.com>
2025-07-18 01:29:30 +08:00
ZhouKe
1aea59a472 multinet wakeword (#927)
Co-authored-by: zk <982145@qq.com>
2025-07-18 01:18:40 +08:00
laride
be46cf1731 feat: upgrade component versions for esp-hi (#931) 2025-07-18 01:09:40 +08:00
Ky1eYang
0c971f76e3 feat: 添加自定义消息接收并打印在display或串口上 (#921)
* feat: 添加声波配网, 需调整application的ReadAudio公有, 需添加条件编译, 位于'afsk_demod.h'内定义参数

* mod: afsk的重构,旨在提高代码可读性并遵循Google C++代码风格指南

* mod: 更新依赖esp-wifi-connect需求版号

* feat: 添加声波配网, 需调整application的ReadAudio公有, 需添加条件编译, 位于'afsk_demod.h'内定义参数

* mod: afsk的重构,旨在提高代码可读性并遵循Google C++代码风格指南

* mod: 更新依赖esp-wifi-connect需求版号

* mod: 添加判断只有在WiFi配置模式下才会调用ReadAudio, 否则delay(联网成功重启后该任务不会被启动)

* add: 添加USE_ACOUSTIC_WIFI_PROVISIONING进MENU开关声波配网功能

* feat: 添加mqtt自定义消息接收, 修复menu的声波配网config

* 简化接收custom消息的处理, 只打印或处理payload字段

* fix: 修复payload在执行时指向被释放的线程不安全问题

---------

Co-authored-by: yangkaiyue <yangkaiyue1@tenclass.com>
2025-07-18 01:07:57 +08:00
Xiaoxia
6bb95073a4 v1.7.7: 支持 EC801E 模组,增加 xmini-c3-v3, xmini-c3-4g (#930) 2025-07-17 20:18:21 +08:00
diskman88
ea8769e1fc add labplus-Ledong-v2 and labplus-mpython-v3 board: (#897)
* add labplus-Ledong-v2 and labplus-mpython-v3 board:
	modified:   main/CMakeLists.txt
	modified:   main/Kconfig.projbuild
	new file:   main/boards/labplus-Ledong-v2/README.md
	new file:   main/boards/labplus-Ledong-v2/config.h
	new file:   main/boards/labplus-Ledong-v2/config.json
	new file:   main/boards/labplus-Ledong-v2/labplus_Ledong_v2.cc
	new file:   main/boards/labplus-mpython-v3/README.md
	new file:   main/boards/labplus-mpython-v3/config.h
	new file:   main/boards/labplus-mpython-v3/config.json
	new file:   main/boards/labplus-mpython-v3/mpython_pro.cc

* Changes to be committed:
	new file:   .clangd

* 修改板名:labplus-Ledong-v2 -> labplus-ledong-v2
	modified:   main/CMakeLists.txt
	modified:   main/Kconfig.projbuild
	renamed:    main/boards/labplus-Ledong-v2/README.md -> main/boards/labplus-ledong-v2/README.md
	renamed:    main/boards/labplus-Ledong-v2/config.h -> main/boards/labplus-ledong-v2/config.h
	renamed:    main/boards/labplus-Ledong-v2/config.json -> main/boards/labplus-ledong-v2/config.json
	renamed:    main/boards/labplus-Ledong-v2/labplus_Ledong_v2.cc -> main/boards/labplus-ledong-v2/labplus_ledong_v2.cc
2025-07-17 00:12:53 +08:00
wurongmin
0b0e1391f0 添加微雪电子 esp32-c6-touch-amoled-1.43 和 esp32-c6-touch-amoled-1.43-b的支持 (#907)
* 添加微雪电子 esp32-c6-touch-amoled-1.43 和 esp32-c6-touch-amoled-1.43-b的支持

* Fix: 根据PR评论优化代码格式
2025-07-15 00:27:30 +08:00
Surfer
0b364a83b3 添加 Surfer-C3-1.14TFT开发板支持 (#914)
* 添加 Surfer-C3-1.14TFT开发板支持

添加 Surfer-C3-1.14TFT开发板,基于C3的1.14寸LCD开发板,支持GPIO2  ADC 电池电量检测,应用于wifi Modem-Sleep 和低电量提醒。

* 修改surfer-c3-1.14tff电池电量适配值

修改surfer-c3-1.14tff电池电量适配值,电池容量400ma
2025-07-15 00:26:20 +08:00
DeepStart
5da837541b 增加新版太极小派适配 (#902)
* Add Guition Boards

* Add Product Links

* 适配新版太极派

* 适配新版太极派
2025-07-11 05:46:49 +08:00
Terrence
cec0b5b273 防止误刷xmini-c3固件导致永久性损害 2025-07-10 15:53:09 +08:00
virgil
8e0f76c0ef fix: fix wakenet model index (#893)
* fix: fix wakenet model index

* fix:Fix warning

* feat: Use jarvis and sophia as wakeup words
2025-07-09 03:19:08 +08:00
小林同志
f556e72604 规范函数名,修复一个重写错误 (#894)
Co-authored-by: smalllin0 <aslinqf@163.com>
2025-07-09 03:14:20 +08:00
ning yuxiang
2cdec25162 Add support for EchoEar V1.2 version PCB. (#891)
Co-authored-by: NingYuxiang <ningyuxiang@espressif.com>
2025-07-09 03:10:43 +08:00
flying1425
f002a1185b 添加对微雪电子 ESP32-S3-Touch-LCD-3.5B 开发板的支持 (#849)
* 添加对微雪电子 ESP32-S3-Touch-LCD-3.5B 开发板的支持

* 更改config.json的错误

* 修改板子的id以waveshare开头

* 更改config.json

* 修改config.json的 name

---------

Co-authored-by: flyingtjy <flyingtjy@gmail.com>
2025-07-05 18:09:23 +08:00
小林同志
1314ccfc0f 使用原始字符串字面量代替转义,提高可读性 (#861)
* 使用原始字符串字面量代替转义,提高可读性

* 使用原始字符串字面量代替转义,提高可读性

* 增加一个使用ESP-IDF Monitor作为输出显示内容的类

* 修改代码风格
2025-07-05 15:11:02 +08:00
laride
b7c1989a34 fix: resolve potential crash on touch isr (#885) 2025-07-05 14:56:11 +08:00
Ky1eYang
10016a3ea5 feat: 添加声波配网, 需调整application的ReadAudio公有, 需添加条件编译, 位于'afsk_demod.h'内定义参数 (#852)
* feat: 添加声波配网, 需调整application的ReadAudio公有, 需添加条件编译, 位于'afsk_demod.h'内定义参数

* mod: afsk的重构,旨在提高代码可读性并遵循Google C++代码风格指南

* mod: 更新依赖esp-wifi-connect需求版号

* feat: 添加声波配网, 需调整application的ReadAudio公有, 需添加条件编译, 位于'afsk_demod.h'内定义参数

* mod: afsk的重构,旨在提高代码可读性并遵循Google C++代码风格指南

* mod: 更新依赖esp-wifi-connect需求版号

* mod: 添加判断只有在WiFi配置模式下才会调用ReadAudio, 否则delay(联网成功重启后该任务不会被启动)

* add: 添加USE_ACOUSTIC_WIFI_PROVISIONING进MENU开关声波配网功能

---------

Co-authored-by: yangkaiyue <yangkaiyue1@tenclass.com>
2025-07-05 14:45:48 +08:00
0x1abin
f744c9bf3e 添加编译优化配置以减小生成的固件大小 (#864)
先合并进来看看大家的反馈。
2025-07-02 06:05:58 +08:00
laride
ad2c187433 添加 EchoEar 喵伴 (#869)
* add EchoEar

Co-authored-by: Qin Feng <qingfeng@espressif.com>

* feat: use new display backlight API

* feat: use MCP protocol

* remove yuying lcd

* add echoear README.md

---------

Co-authored-by: NingYuxiang <ningyuxiang@espressif.com>
Co-authored-by: Qin Feng <qingfeng@espressif.com>
2025-06-30 23:01:17 +08:00
virgil
8a0ec8b657 fix: fixed the issue of slow startup for devices without cameras (#866) 2025-06-30 22:41:42 +08:00
Terrence
d1c586239c fix CPU usage of esp32c3 with ml307 2025-06-29 06:01:41 +08:00
Hangon66
f2f54ba1fd 添加 jiuchuan-s3 开发板支持 (#775)
* 添加 jiuchuang-s3 开发板支持

* 增加编译指南

* 优开发板支持包文件目录,更新README.md

删除了多余板载文件
[* ]README.md -> 更新了编译指南和烧录指南

* 修改板级支持包名

* 使用乐鑫提供的电源监测

* 修复部分代码格式问题

* 解决合并冲突

* 解决部分合并内容

* 完善合并内容

* 修复电量映射表错误

---------

Co-authored-by: unknown <jake12355>
2025-06-29 05:41:42 +08:00
virgil
f1277934d1 feat: Support choosing different config.json (#848) 2025-06-26 02:55:45 +08:00
Terrence
c17bd15baa Bump to 1.7.6 2025-06-24 10:19:06 +08:00
Terrence
b3ab3d0920 在S3芯片上使用更多的PSRAM,解决立创开发板拍照时可能出现内存不足的问题 2025-06-24 04:59:00 +08:00
Terrence
2b0362a812 Fix: misconfigured target of atommatrix-echo-base 2025-06-19 14:52:53 +08:00
Terrence
d3367d6b92 Update esp-ml307 to 2.2.1 2025-06-18 05:24:51 +08:00
Terrence
116234a147 Bump to 1.7.5 2025-06-16 20:55:57 +08:00
Xiaoxia
f29c1a11d9 ml307: Add sleep mode (#826) 2025-06-16 19:05:31 +08:00
laride
f98ffdbb5c fix: optimize MCP commands for ESP-Hi (#825) 2025-06-16 16:03:21 +08:00
Xiaoxia
89f10365b1 xmini-c3: 休眠时关闭es8311可减少20mA电流 (#822)
* Correct class member name

* xmini-c3: 休眠时关闭es8311可减少20mA电流
2025-06-16 12:51:27 +08:00
ZhouKe
e9f23ea231 -增加带面包板的摄像头功能 (#815)
* -增加带面包板的摄像头功能

* rename board

---------

Co-authored-by: zk <982145@qq.com>
2025-06-13 21:03:13 +08:00
小鹏
7435c98609 1.增加robot舵机初始位置校准 2.fix(mcp_sever) 超出范围异常捕获类型 bug (#817)
* otto v1.4.0 MCP

1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay

* otto v1.4.1 gif as components

gif as components

* electronBot v1.1.0 mcp

1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类

* 规范代码

1.规范代码
2.修复切换主题死机bug

* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug

* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型  bug
2025-06-13 21:02:03 +08:00
Terrence
bf125446b3 feat: Use BOOT button to enter audio testing state when Wi-Fi configuring 2025-06-13 19:57:07 +08:00
flying1425
dfad6a5b2c 添加微雪电子esp32-c6-lcd-1.69、esp32-c6-Touch-lcd-1.69的支持 (#816)
* 添加了摄像头和触控支持

* 添加微雪电子esp32-c6-lcd-1.69、esp32-c6-Touch-lcd-1.69的支持

---------

Co-authored-by: flyingtjy <flyingtjy@gmail.com>
2025-06-13 18:20:22 +08:00
YeezB
d460af8426 feat: Add df-k10 MCP control on board RGB LED (#810)
* feat: Add dfk10 MCP blink

* fix: delete iot protocol related part
2025-06-13 18:08:08 +08:00
小鹏
7bb17f7539 fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug (#812)
* otto v1.4.0 MCP

1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay

* otto v1.4.1 gif as components

gif as components

* electronBot v1.1.0 mcp

1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类

* 规范代码

1.规范代码
2.修复切换主题死机bug

* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug
2025-06-13 13:46:14 +08:00
Terrence
f8c9126442 Bump to 1.7.4 2025-06-12 21:53:26 +08:00
Lucinhu
a118e8f786 feat: es8311添加功放使能引脚的反向控制 (#808) 2025-06-12 20:37:15 +08:00
Terrence
85f3f1ba9f 修复model分区未初始化导致重启 2025-06-12 15:06:31 +08:00
Terrence
e2777cc16b esp-hi special app partition 2025-06-12 15:06:31 +08:00
wuxingzhong
5bb7c6deb8 fix: 修复修改vad模型为:vadnet1 medium时, 不生效问题. (#802) 2025-06-12 15:02:48 +08:00
netseye
895a3cfa72 fix: 修复tab5 esp-hosted 升级带来的crash问题 (#795)
Co-authored-by: Jeakin <Jeakin@botu.cc>
2025-06-11 12:14:22 +08:00
laride
c9dec29d73 fix: delay WebServer startup to mitigate stack overflow (#797) 2025-06-11 12:12:39 +08:00
Terrence
968ed1fae3 v1.7.3: 参考ESP-HI,为所有C3板子增加10多KB可用SRAM 2025-06-10 01:33:20 +08:00
Terrence
f8cd0d30cd fix idf_component.yml 2025-06-09 12:11:51 +08:00
laride
01215d77ed fix: 修复 ESP-Hi 在联网时 crash 的问题 (#790)
* fix: resolve crash issue during network connection on ESP-Hi

* fix: adjust dependency rules for some components
2025-06-09 11:48:20 +08:00
wdmomoxx
3df2f3970a 添加esp32支持唤醒词 (#782)
* Update README.md

* Update config.h

增加MCP控制方式

* Update esp32_cgc_board.cc

增加MCP控制方式

* Update CMakeLists.txt

增加ESP32 CGC 144开发板

* Update Kconfig.projbuild

增加ESP32 CGC 144开发板

* Create README.md

增加ESP32 CGC 144开发板

* Add files via upload

* Update config.h

修改注释

* Update Kconfig.projbuild

增加ESP32语言唤醒支持(目前需要开启PSRAM)

* Add files via upload

Add wake word to esp32

* Update sdkconfig.defaults.esp32

增加看门狗超时
2025-06-09 04:43:46 +08:00
Terrence
386e58a224 Bump to 1.7.2 2025-06-08 12:01:24 +08:00
Terrence
504e694caa 先在lichuang-dev上测试NSNet降噪 2025-06-08 02:52:23 +08:00
Terrence
d58037eecd fix p4 compiling errors 2025-06-08 02:39:53 +08:00
Terrence
218d0318fe fix audio debugger compiling errors 2025-06-06 21:39:22 +08:00
Terrence
7bc5f7bb0c Add audio debugger 2025-06-06 20:24:00 +08:00
Terrence
24ede22197 wechat ui支持摄像头预览 2025-06-06 16:54:53 +08:00
flying1425
dc4e82abaf 添加了摄像头和触控支持 (#772)
Co-authored-by: flyingtjy <flyingtjy@gmail.com>
2025-06-06 14:27:26 +08:00
virgil
5c51fbc2f8 Feat sscma camera (#770)
* feat: add sscma camera

* fix: 修复sscma camera 无法使用等问题

* feat: update README.md

* style: optimize styles

* style: fix styles

* Update sscma_camera.h

---------

Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-06-06 14:22:39 +08:00
Xiaoxia
8729fad9a8 v1.7.1: fix send audio (#771)
* screen -> display

* Add AUDIO_CODEC_DEFAULT_MIC_GAIN

* WebRTC -> NSNet

* fix: send audio priority
2025-06-06 14:03:46 +08:00
yusuhua
b18dda3f2f 增加LilyGo T-CameraPlus-S3的摄像头红外过滤功能 (#766)
* set camera HMirror to true

* change key1 to camera button

* change key1 to camera button

* change key1 to camera button

* add ir filter controller

* add ir filter controller

* add ir filter controller
2025-06-06 11:18:02 +08:00
小鹏
7bb12f31f0 增加ottoRobot和electronBot的支持 (#757)
* otto v1.4.0 MCP

1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay

* otto v1.4.1 gif as components

gif as components

* electronBot v1.1.0 mcp

1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类

* 规范代码

1.规范代码
2.修复切换主题死机bug
2025-06-05 23:51:49 +08:00
Terrence
6d02b7724a Update related projects 2025-06-05 20:42:45 +08:00
Terrence
5621fa5a65 fix: 长按切换到点按后,无法唤醒的问题 2025-06-05 18:41:23 +08:00
LILYGO_L
2f1229f09f 修复T-CameraPlus-S3_V1.2板子摄像头不能正常使用问题 (#758)
* Resolve the issue where the camera on the T-CameraPlus-S3_V1.2 board cannot be used normally.
2025-06-05 17:24:43 +08:00
Li Junru
54ff46788b fix(esp-hi): fix can not wake up (#754) 2025-06-05 15:52:11 +08:00
Terrence
f5d6cd2d36 Bump to 1.7.0 2025-06-04 20:32:41 +08:00
Spotpear
d4ae55a4e3 修复屏幕条纹 (#748) 2025-06-04 14:44:33 +08:00
Terrence
abba768e90 Add tools/call stack size for HTTPS upload 2025-06-04 14:17:43 +08:00
Terrence
f41210577b v1.6.8: Move partition CSV files into a folder 2025-06-04 12:11:55 +08:00
Xiaoxia
16ec95f6d5 Update documentation (An MCP-based Chatbot) 2025-06-04 04:17:49 +08:00
Li Junru
336eff728d feat(esp-hi): update adc mic version (#741) 2025-06-04 01:32:57 +08:00
laride
bfc695b1ee feat: ESP-Hi 改用 MCP 协议 (#740)
* feat(esp-hi): use MCP protocol

* docs(esp-hi): Add README

* feat(esp-hi): remove redundant checks and add option to disable WebUI
2025-06-03 16:45:03 +08:00
laride
9122664f0f feat: 添加 ESP-Hi WebUI (#739)
* feat(esp-hi): Skip download if file already exists locally

* feat(esp-hi): add WebUI
2025-06-03 15:24:11 +08:00
Terrence
e52703df24 feat: Add Serial-Number to OTA header 2025-06-03 14:15:13 +08:00
wdmomoxx
02abd71588 更新ESP32 CGC MCP控制方式,添加一个ESP32 CGC 144的开发板 (#736)
* Update README.md

* Update config.h

增加MCP控制方式

* Update esp32_cgc_board.cc

增加MCP控制方式

* Update CMakeLists.txt

增加ESP32 CGC 144开发板

* Update Kconfig.projbuild

增加ESP32 CGC 144开发板

* Create README.md

增加ESP32 CGC 144开发板

* Add files via upload

* Update config.h

修改注释
2025-06-03 12:20:57 +08:00
zhouchunyuan
32c5b3e6d4 正点原子DNESP32S3开发板新增相机(ov2640) (#733)
* 正点原子DNESP32S3开发板新增相机(ov2640)

* 增加了相机预初始化,如无相机则camera_ = nullptr;并修正了语法分号

利用了 esp32_camera.h 的 esp_camera_init()/esp_camera_deinit() 函数
2025-06-02 13:51:54 +08:00
Terrence
67c44403a3 Update ESP-IDF version to 5.4 2025-06-01 21:44:12 +08:00
Terrence
e4ced2a62e sdkconfig: fix default value 2025-06-01 16:02:22 +08:00
Xiaoxia
ae57131c15 Add wake word to xmini-c3 (#730)
* esp-hi: MCP protocol is not ready yet

* Add wake word to xmini-c3
2025-05-31 22:21:03 +08:00
yusuhua
6cb025859f LilyGo T-CameraPlus-S3低版本发布及增加拍照键 (#720)
* set camera HMirror to true

* change key1 to camera button

* change key1 to camera button

* change key1 to camera button
2025-05-30 10:13:18 +08:00
liyuaxue
47bdfa2c77 hide low_battery (#721) 2025-05-30 01:10:28 +08:00
Terrence
1a695a203b fix compiling errors 2025-05-30 01:09:51 +08:00
Terrence
1858da4973 Avoid tool call blocking main thread (dropping packets) 2025-05-30 00:53:11 +08:00
espressif2022
f5c1c30c5e feat: add new board esp-hi (#666)
* feat: add new board esp-hi

* feat(esp-hi): update servo_dog_ctrl

---------

Co-authored-by: Li Junru <lijunru@espressif.com>
Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-05-29 23:29:33 +08:00
Liam
565c54e7d4 MCP协议文档 (#717)
* MCP协议文档

* Update mcp_protocol.md

---------

Co-authored-by: loadingxv <loadingvx@gmail.com>
Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-05-29 20:12:21 +08:00
Spotpear
34a77f75df 增加sp-esp32-s3-1.28-box开发板 (#714) 2025-05-29 20:04:03 +08:00
laride
e74fd34bf1 feat(Sparkbot): add camera flip control via MCP (#716) 2025-05-29 20:01:08 +08:00
Terrence
89fbf89d0f fix exception 2025-05-29 13:46:19 +08:00
yusuhua
0e6375e347 set camera HMirror to true (#707) 2025-05-29 09:39:25 +08:00
wcc-zf
0902ced1c4 修复param.required()==false时可能触发未定义的问题 (#705) 2025-05-29 09:37:54 +08:00
yusuhua
aef527f7af LilyGo T-CameraPlus-S3 add camera function (#704)
* add camera function

* add camera function
2025-05-28 23:35:42 +08:00
Y1hsiaochunnn
245be0ea1b Three new Waveshare ESP32-P4 development boards have been added (#691)
* Added third-party hardware support

Waveshare-ESP32-P4-WIFI6-Touch-LCD-3.4C,
Waveshare-ESP32-P4-WIFI6-Touch-LCD-4C,
Waveshare-ESP32-P4-WIFI6-Touch-LCD-4B

* Remove the README

* Merge ESP32-P4-WIFI6-Touch-LCD-3.4C and ESP32-P4-WIFI6-Touch-LCD-4C into one board

* Add readme files
2025-05-28 23:20:14 +08:00
HonestQiao
7f7f24e5ad 调整摄像头拍照方向 (#693) 2025-05-28 06:47:30 +08:00
YeezB
030a6b9da2 feature: Add dfrobot-k10 vision recognition (#688) 2025-05-27 21:44:03 +08:00
HonestQiao
12cf213351 增加DFRobot ESP32-S3 AI智能摄像头模块,适配视觉识别功能 (#692)
* 增加 DFRobot ESP32-S3 AI智能摄像头模块

* 适配视觉识别功能

* 修改dfrobot-esp32-s3-ai-cam为df-s3-ai-cam
2025-05-27 21:43:01 +08:00
LILYGO_L
da62523765 Added support for the T-CameraPlus-S3_V1.2 board (#680)
* Adapt for LilyGO-T-Circle-S3 device

* Adapt for LilyGO-T-Circle-S3 device

* Remove comments and modify the size of the lilygo-t-circle-s3 image

* Modify the code style and format to Google C++

* Modify the code style and format to Google C++

* Fixed bugs in the LILYGO T-Circle-S3 board and added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Merge branch 'main' of https://github.com/Llgok/xiaozhi-esp32

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Fix the color display issue for T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Update T-CameraPlus-S3_V1.2 Version Xiaozhi Example
2025-05-27 21:39:48 +08:00
Xiaoxia
d80f94387a v1.6.6: Set MCP as default IoT Protocol (#690) 2025-05-27 14:58:49 +08:00
Create123
0c83263762 Atoms3r cam m12 echo base (#689)
* main/boards: Add camera support for M5Stack CoreS3.

* main/boards: Add camera support for M5Stack AtomS3R-CAM/M12 + EchoBase.
2025-05-27 12:45:09 +08:00
laride
2b553ce6ff feat: add camera functions (ESP-SparkBot) (#687)
Co-authored-by: lvhaiyu <lvhaiyu@espressif.com>
2025-05-27 11:48:31 +08:00
Create123
391aa74abe main/boards: Add camera support for M5Stack CoreS3. (#686) 2025-05-27 09:55:29 +08:00
Terrence
3a2e6875b6 Add audio_send_queue to synchronize timestamp 2025-05-27 05:44:46 +08:00
Terrence
d17bfccb35 fix: compile errors 2025-05-27 04:50:21 +08:00
Kevincoooool
0e42bd2b18 Add camera support board(sp v3/v4 korvo) (#682)
* Add camera support board

* Update esp32_camera.cc

---------

Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-05-26 18:28:51 +08:00
hanhanlili
8555b89692 fix power manager error (#681) 2025-05-26 18:24:33 +08:00
Terrence
0c57df1cd8 v1.6.5: Improve performance and memory usage 2025-05-26 14:30:44 +08:00
Terrence
277f87ae5f revert gpio level (#627) 2025-05-26 14:17:54 +08:00
Terrence
9f90c7dd6a remove preview_thread_, cost more time but less memory usage 2025-05-26 07:04:25 +08:00
Terrence
f890220ce3 remove unused ota class members 2025-05-26 06:11:47 +08:00
Terrence
ce1211c86d update esp32 camera code 2025-05-26 06:10:53 +08:00
Terrence
249d12ac25 feat: Add camera functions (lichuang-dev) 2025-05-25 17:07:07 +08:00
Terrence
ecfebc4a29 Add preview image to lcd display 2025-05-25 07:02:44 +08:00
Terrence
0204b8800b update status bar immediately after network started 2025-05-24 07:25:34 +08:00
Terrence
6d5331dddd [MCP] extend tools/list payload size to 8000 2025-05-24 03:06:01 +08:00
Terrence
0b98392286 Remove update display timer 2025-05-24 03:03:33 +08:00
Terrence
c39f6cfb02 Bump to 1.6.3 2025-05-23 04:59:29 +08:00
Terrence
55a1032779 do not allocate memory if old IoT protocol is not used 2025-05-23 04:26:47 +08:00
Terrence
5da66773d5 Add MCP server 2025-05-22 19:19:48 +08:00
Terrence
f142c5469c Adjust mqtt variables 2025-05-22 19:19:48 +08:00
Spotpear
8eecdd1ffc '增加sp-esp32-s3-1.54-muma开发板' (#655) 2025-05-22 14:30:44 +08:00
hanhanlili
319d3332be 增加敏思开发板 (#649)
* 增加敏思开发板

* 去除不需要的代码

---------

Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-05-22 14:29:38 +08:00
MakerM0
9874a1b009 增加4G扩展 (#653)
在不影响wifi原版功能的基础上,方便想增加4G功能的朋友通过内置的fpc座子连接4G模块;
默认开机是wifi模式,在联网成功前,双击主按钮,可以进行WIFI和4G网络切换;
若在wifi模式,在成功连接前,可单击主按钮可以进行重新配网;
2025-05-21 22:19:25 +08:00
liyuaxue
0183830915 add zhengchen_ml307 board(征辰科技) (#641)
* add zhengchen_ml307 board

* add boards_http
2025-05-21 22:09:12 +08:00
Y1hsiaochunnn
61bf5754db Add some register configuration for 1.75 amoled screen to adapt to the new screen (#646) 2025-05-21 15:56:07 +08:00
WDF-king
180ca89bd3 初次提交亘具科技s3 1.54小智代码 (#576)
* 初次提交亘具科技s3 1.54小智代码

* 第一次修改命名

* 修改名字,移除GennJu文件夹

* 提交修改了名字的genju文件夹

---------

Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-05-20 21:17:02 +08:00
Y1hsiaochunnn
b49ad1755d Add board support for Waveshare ESP32-S3-Touch-AMOLED-1.75 (#643)
* Add board support for Waveshare ESP32-S3-Touch-AMOLED-1.75

* Fix component excess strings

* Fix the ui

* Update Kconfig.projbuild

Correction board for USB_DEVICE_AEC requirements
2025-05-20 20:59:09 +08:00
Terrence
d2f42527f4 fix: iot screen invalid iot descriptor 2025-05-20 09:23:14 +08:00
liyuaxue
b0b471e27e add zhengchen_boards_wifi(征辰科技) (#635)
* add zhengchen_boards_wifi

* add zhengchen_boards_wifi1
2025-05-19 15:37:06 +08:00
Y1hsiaochunnn
538e07aa70 Add a description of supported hardware chips (#637)
LGTM!
2025-05-19 14:59:42 +08:00
Terrence
c3e488e744 Iot: Use English description 2025-05-19 10:06:05 +08:00
Terrence
425f46df4b Fix: esp_lcd_ili9881c only for p4 2025-05-19 10:04:34 +08:00
Create123
f7ea248845 main/boards: Add support for M5Stack Tab5. (#632) 2025-05-19 01:43:49 +08:00
Terrence
7925f77739 Fix: 切换状态后一段时间内无法录音问题 2025-05-19 01:29:39 +08:00
HonestQiao
beffb362dc 将gpio_led的FadeCallback放到IRAM (#629)
* 添加四博智联AI陪伴盒子适配,典型pdm麦克风设备

* doit_s3_aibox LED切换到GPIO LED

* doit_s3_aibox 添加说明文件

* doit_s3_aibox 音量最大值设置到100

* 将gpio_led的FadeCallback放到IRAM
2025-05-16 22:26:17 +08:00
Ky1eYang
27d85356e4 修改send packet时直接获取时间戳为队列获取时间戳 (#628)
* 修改实时aec传输时间戳改队列方式

* 删/注释化一下log代码, pr

---------

Co-authored-by: yangkaiyue <yangkaiyue1@tenclass.com>
2025-05-16 18:46:20 +08:00
Y1hsiaochunnn
7ceeffc157 Adjust dependency components for ESP32-P4 (#623)
* Adjust dependency components for ESP32-P4

* Add backlight control and touch control to Waveshare ESP32-P4-NANO

* Increase the amount of saved information for Waveshare ESP32-P4-NANO
2025-05-16 18:36:43 +08:00
Y1hsiaochunnn
1c63934756 Adjust ESP32-P4 chip compatibility Add Waveshare ESP32-P4-NANO board support (#589)
* Added ESP32-P4 chip support

* Add Waveshare ESP32-P4-NANO board support

* Add Waveshare ESP32-P4-NANO board support

* Add prefix "Waveshare" to waveshare products

* Adjust the ESP32-P4-NANO README and fix the index link.

* Update code style

* Optimize code styles

* Adjust to naming requirements

* Adjust to naming requirements
2025-05-16 02:41:35 +08:00
Terrence
45c62f5a53 Add session_id to websocket message 2025-05-13 17:16:24 +08:00
Zhihui Zheng
8e9f59efae chore: 更新 .gitignore 文件 (#591)
- 添加 .cache 目录到忽略列表
- 保持 .DS_Store 在忽略列表中
2025-05-12 14:43:10 +08:00
Terrence
fcc35d132c fix unused variable warning 2025-05-12 04:43:45 +08:00
Terrence
3047ac4556 Fix Initialize() 2025-05-11 21:14:09 +08:00
yusuhua
7abcd30def Lilygo T-CameraPlus-S3 关闭延时关机以及削平电池百分比 (#598)
* lilygo-t-cameraplus-s3 add pmic

* set lilygo-t-cameraplus-s3 seconds_to_shutdown to -1
2025-05-10 19:18:37 +08:00
Xiaoxia
be18c1aa04 Add server AEC option 2025-05-09 14:00:26 +08:00
yusuhua
7ea67146d5 lilygo-t-cameraplus-s3 add pmic (#586) 2025-05-09 04:24:58 +08:00
Terrence
724f440fab fixbug: button driver cannot be in stack memory 2025-05-09 03:56:51 +08:00
Terrence
a0ad3e6dc3 Default realtime chat for esp-box-3, lichuang-dev 2025-05-08 02:32:24 +08:00
Terrence
b6489a3456 Double click BOOT to switch 4G/Wi-Fi 2025-05-07 17:11:26 +08:00
Terrence
ee0df3470c Default press time to 0 2025-05-07 16:42:23 +08:00
Terrence
c0acf30b25 Bump to 1.6.2 2025-05-07 15:27:51 +08:00
Terrence
5cb31a5e91 Add CEREG to board JSON for 4G LBS to fix weather location 2025-05-07 15:26:53 +08:00
Terrence
696786ad4c Upgrade component versions 2025-05-07 15:26:53 +08:00
Terrence
7c6b9d8fa7 Use English output log 2025-05-07 15:26:53 +08:00
flying1425
0beae83143 解决部分esp32-s3-touch-lcd-3.5开机黑屏不显示的问题 (#581)
* 解决部分开机黑屏的问题

* Update esp32-s3-touch-lcd-3.5.cc

---------

Co-authored-by: flyingtjy <flyingtjy@gmail.com>
Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-05-07 15:26:08 +08:00
Terrence
fd6235750d Upgrade components 2025-05-07 04:55:51 +08:00
Terrence
b00c5b50c3 Adjust LCD display 2025-05-07 04:49:21 +08:00
Terrence
4be2464fbc Remove warnings 2025-05-07 04:40:17 +08:00
Haibo Gu
d505b3b1ae 【需求】双网络类型通过长按boot切换网络 (#520)
* 【需求】双网络类型通过长按boot切换网络

* Update Kconfig.projbuild

---------

Co-authored-by: guhaibo <guhaibo@boe.com.cn>
Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-05-07 02:33:38 +08:00
Liam
8e638e3884 OTA checkversion增加了一些log, 便于三方开发调试自己的OTA接口 (#568)
* feat(ota): add logging for missing sections in Ota::CheckVersion

Add detailed logging to inform when specific sections (activation, mqtt, websocket, server_time, firmware) are not found during the OTA version check. This improves debugging and visibility into the OTA process.

* 恢复提示词

---------

Co-authored-by: loadingxv <loadingvx@gmail.com>
2025-05-04 19:25:50 +09:00
ourkix
f590e49f2d 添加乐鑫开发板esp32-s3-lcd-ev-board支持 (#543)
* 添加乐鑫开发版esp32-s3-lcd-ev-board支持

* 新增乐鑫开发版esp32-s3-lcd-ev-board支持

* 新增乐鑫开发板esp32-s3-lcd-ev-board支持,修复led和背光bug

* Delete main/boards/ESP32-S3-LCD-EV-Board directory

* 新增乐鑫开发板esp32-s3-lcd-ev-board支持,修复led和背光bug

* 去除冗余代码

* 添加esp-s3-lcd-ev-board的硬件版本选择配置

* Delete main/boards/esp-s3-lcd-ev-board-1.4 directory

* 添加esp-s3-lcd-ev-board的硬件版本选择配置

* 新增不同版本的名称标识,使ota更新时能识别
2025-04-30 18:21:35 +08:00
LILYGO_L
b5595c0ab4 Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY. (#537)
* Adapt for LilyGO-T-Circle-S3 device

* Adapt for LilyGO-T-Circle-S3 device

* Remove comments and modify the size of the lilygo-t-circle-s3 image

* Modify the code style and format to Google C++

* Modify the code style and format to Google C++

* Fixed bugs in the LILYGO T-Circle-S3 board and added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Merge branch 'main' of https://github.com/Llgok/xiaozhi-esp32

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.

* Fix the color display issue for T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
2025-04-29 20:42:50 +08:00
Terrence
4caffd31ed Remove No serial message 2025-04-29 20:42:31 +08:00
Terrence
c94e82e865 Add timestamp to MQTT UDP packet 2025-04-29 20:04:17 +08:00
Terrence
f6334246ca Add dummy audio processor 2025-04-29 18:17:08 +08:00
Terrence
c201e5955f Revert gc9a01 version 2025-04-29 18:15:45 +08:00
Xiaoxia
b804343d30 Audio stream packet with timestamp 2025-04-28 23:10:24 +08:00
Xiaoxia
7fdf78408e Add v2.0 to versions.py 2025-04-28 17:05:56 +08:00
Terrence
5f8389c135 Add binary protocol v2 & v3 to websocket 2025-04-28 16:29:33 +08:00
Terrence
1f03d770cc update ml307 version 2025-04-28 16:28:53 +08:00
Terrence
6fbf272613 Allow 10 audio packets in recv queue 2025-04-28 11:19:28 +08:00
matu666
a8ba7e75fe add lcd touch for esp32-s3-touch-amoled-1.8 (#550) 2025-04-26 20:45:02 +08:00
Wiking_Xu
6fabcaabba 新增正点原子三款开发板 (#538) 2025-04-25 07:19:58 +08:00
ZhouKe
7d5237a604 add lcd touch for lichuang_dev (#531)
* add lcd touch for lichuang_dev

* Update lichuang_dev_board.cc

---------

Co-authored-by: zk <982145@qq.com>
Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-04-23 15:00:17 +08:00
Terrence
738e170d54 Update esp-wifi-connect to 2.4.2 2025-04-23 07:24:55 +08:00
JackHuang021
abb594bddd atk_dnesp32s3: fix set xl9555 port1 output state (#525)
Signed-off-by: Jack Huang <jackhuang021@gmail.com>
2025-04-22 10:28:26 +08:00
Terrence
3404180a77 更新Wi-Fi组件版本,从OTA接口读取Websocket服务器 2025-04-21 15:13:11 +08:00
Terrence
c380617cca Support MQTT endpoint port 2025-04-21 15:13:11 +08:00
大汉子民
a6619dcdb9 更新支持 元控·青春板卡 (#486)
* 更新支持 元控·青春板卡

* 更新README.md,修改多灯库使用问题
2025-04-17 08:36:25 +08:00
Terrence
dff8f9cb5b Bump to 1.6.0 2025-04-14 23:17:06 +08:00
Terrence
68a3f7d2f7 Allow force OTA upgrade to a previous version 2025-04-14 22:05:29 +08:00
Terrence
6e73a11ac9 Add two types, system and alert. 2025-04-14 16:02:41 +08:00
Terrence
d5d8b34b2b Add activation version 2 2025-04-14 15:44:06 +08:00
Xiaoxia
f76f31aa12 Prevent too many opus packets in queue 2025-04-13 23:12:44 +08:00
Terrence
eac5830439 remove duplicated header files 2025-04-13 15:35:36 +08:00
Terrence
78d7b40914 将项目版本更新至1.5.9,优化内存使用,增加主任务栈大小至8192,优化固件升级流程,重构主循环为MainEventLoop,添加新版本检查功能,更新音频编解码器的DMA配置常量。 2025-04-13 11:47:15 +08:00
HonestQiao
4ef6ef3552 增加四博智联AI陪伴盒子 (#309)
* 添加四博智联AI陪伴盒子适配,典型pdm麦克风设备

* doit_s3_aibox LED切换到GPIO LED

* doit_s3_aibox 添加说明文件

* doit_s3_aibox 音量最大值设置到100
2025-04-13 07:04:46 +08:00
tangyumei3535
f141a6bf32 Add board: ESP-Spot (#460)
* feat: add ESP-Spot S3

Co-authored-by: Tang Yumei <tangyumei@espressif.com>

* feat: add button and power control

long press KEY to shut down.

* feat: change for s3 adc

* fix: adc cali fail

* add ESP-Spot S3 Readme

* fix ES8311 config

---------

Co-authored-by: Liu Ruichao <liuruichao@espressif.com>
2025-04-13 06:45:05 +08:00
Almost-LoverY
4787eb41ab The atk-dnesp32s3-box is compatible with ES8311 devices and non-ES8311 devices (#477)
* The atk-dnesp32s3-box is compatible with ES8311 devices and non-ES8311 devices

* Update atk_dnesp32s3_box.cc

format code

---------

Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-04-12 17:17:27 +08:00
mtdxc
a3a833b19d add ci build.yml (#475) 2025-04-12 09:34:33 +08:00
virgil
076d907abf sensecap watcher manufacture (#469)
* feat: add shutdown and battery cmd.

* fix: fixed the issue that the LCD does not light up when some devices are turned on.

* fix:  fix task sys_evt  stack overflow.

* feat: Optimize UI display for circles; add Added factory reset function.

* feat:  "low_battery_label_" obj  configurable

* feat: add read_mac cmd

* fix: fix "low_battery_label_" obj  redefine

* style: modify Google C++ Style.

* Update sensecap_watcher.cc

Remove extra spaces

---------

Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-04-12 09:33:07 +08:00
Xiaoxia
04c0da059f Fix missing display 2025-04-11 03:11:32 +08:00
Terrence
a5dfe67504 move ota headers to ota.cc 2025-04-10 18:56:17 +08:00
mtdxc
f1ad29be3b Boxlite aec (#465)
* 允许espbox-lite打开aec
微信模式使用小字体

* 软件回声消除

* 支持双麦采集

---------

Co-authored-by: cqm <cqm@97kid.com>
2025-04-10 17:06:15 +08:00
Terrence
18e44ec775 Fix i2c header files 2025-04-10 01:57:18 +08:00
Xiaoxia
81aa8a43ce Bump to 1.5.8 2025-04-09 23:58:32 +08:00
Terrence
dc28f0d9c2 use smaller emoji in wechat message style 2025-04-09 23:30:41 +08:00
Terrence
a6ce728386 Fix ML307 FIFO Overflow 2025-04-09 10:41:12 +08:00
Terrence
652e5cbcdd avoid double error alerts 2025-04-09 09:13:18 +08:00
Terrence
69540c6551 fix nullptr error 2025-04-09 08:24:02 +08:00
MOV
9e0de3b302 fix:Cuican Running Error for v1.5.6 (#456)
* fix:Modify the README and add Moji images

* fix: Moji LCD initialization configuration.

* fix: DISPLAY_MIRROR_X false >> true

* fix: Moji GPIO conflict

* fix: Moji GPIO conflict

* fix: Cuican Build config
2025-04-09 00:48:46 +08:00
繁花君1号
27a14c249a 修复 bug (#454)
Co-authored-by: geng_MAX <geng_MAX@hotmail.com>
2025-04-08 23:13:26 +08:00
Xiaoxia
c0e702ff47 Bump to 1.5.6 2025-04-05 06:39:59 +08:00
Xiaoxia
91f98a49d9 magiclick 2p4/2p5: add click to reconfigure wifi 2025-04-05 03:27:53 +08:00
Xiaoxia
976ad2eac6 atk box build with wechat message style 2025-04-04 15:58:23 +08:00
Xiaoxia
fcd886c217 移除长按对话,改为单击多轮对话 2025-04-04 15:53:50 +08:00
MakerM0
eae0ca9315 增加电量显示和睡眠 (#444) 2025-04-04 15:37:40 +08:00
virgil
464633e7a1 Fix lcd bug (#443)
* fix: fixed the issue that the LCD does not light up when some devices are turned on.

* fix:  fix task sys_evt  stack overflow.
2025-04-04 15:33:17 +08:00
Xiaoxia
70883058e8 fix startup sound 2025-04-04 15:29:44 +08:00
HonestQiao
0ac2f09bb0 GPIO LED更新,可以设置LEDC定时器和控制通道 (#439)
* 添加普通LED支持,支持不同状态亮度设置和呼吸灯效果

* GPIO LED更新,可以设置LEDC定时器和控制通道
2025-04-03 02:25:03 +08:00
Xiaoxia
c9dfcb7c93 更新开发者文档(自定义开发板、IoT设备) 2025-04-02 09:11:41 +08:00
香草味的纳西妲
7c6b39e65d add: LVGL Image Converter Script 增加LVGL图片转换工具脚本,可以自定义小智的表情文件。 (#423)
* add: LVGL Image Converter

* add: README

* Rename README.MD to README.md

* remove readme image

---------

Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-04-01 06:51:11 +08:00
Kevincoooool
31c1f07481 Korvo开发板支持AEC (#431) 2025-03-31 00:07:12 +08:00
959 changed files with 50400 additions and 4274 deletions

2
.clangd Normal file
View File

@@ -0,0 +1,2 @@
CompileFlags:
Remove: [-f*, -m*]

106
.github/workflows/build.yml vendored Normal file
View File

@@ -0,0 +1,106 @@
name: Build Boards
on:
push:
branches:
- main
- ci/* # for ci test
pull_request:
branches:
- main
permissions:
contents: read
jobs:
prepare:
name: Determine boards to build
runs-on: ubuntu-latest
outputs:
boards: ${{ steps.select.outputs.boards }}
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install jq
run: sudo apt-get update && sudo apt-get install -y jq
- id: list
name: Get all board list
run: |
echo "all_boards=$(python scripts/release.py --list-boards --json)" >> $GITHUB_OUTPUT
- id: select
name: Select boards based on changes
env:
ALL_BOARDS: ${{ steps.list.outputs.all_boards }}
run: |
EVENT_NAME="${{ github.event_name }}"
# For push to main branch, build all boards
if [[ "$EVENT_NAME" == "push" ]]; then
echo "boards=$ALL_BOARDS" >> $GITHUB_OUTPUT
exit 0
fi
# For pull_request
BASE_SHA="${{ github.event.pull_request.base.sha }}"
HEAD_SHA="${{ github.event.pull_request.head.sha }}"
echo "Base: $BASE_SHA, Head: $HEAD_SHA"
CHANGED=$(git diff --name-only $BASE_SHA $HEAD_SHA || true)
echo "Changed files:\n$CHANGED"
NEED_ALL=0
declare -A AFFECTED
while IFS= read -r file; do
if [[ "$file" == main/* && "$file" != main/boards/* ]]; then
NEED_ALL=1
fi
if [[ "$file" == main/boards/* ]]; then
board=$(echo "$file" | cut -d '/' -f3)
AFFECTED[$board]=1
fi
done <<< "$CHANGED"
if [[ "$NEED_ALL" -eq 1 ]]; then
echo "boards=$ALL_BOARDS" >> $GITHUB_OUTPUT
else
if [[ ${#AFFECTED[@]} -eq 0 ]]; then
echo "boards=[]" >> $GITHUB_OUTPUT
else
JSON=$(printf '%s\n' "${!AFFECTED[@]}" | sort -u | jq -R -s -c 'split("\n")[:-1]')
echo "boards=$JSON" >> $GITHUB_OUTPUT
fi
fi
build:
name: Build ${{ matrix.board }}
needs: prepare
if: ${{ needs.prepare.outputs.boards != '[]' }}
strategy:
fail-fast: false # 单个 board 失败不影响其它 board
matrix:
board: ${{ fromJson(needs.prepare.outputs.boards) }}
runs-on: ubuntu-latest
container:
image: espressif/idf:release-v5.4
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build current board
shell: bash
run: |
source $IDF_PATH/export.sh
python scripts/release.py ${{ matrix.board }}
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: xiaozhi_${{ matrix.board }}_${{ github.sha }}.bin
path: build/merged-binary.bin
if-no-files-found: error

8
.gitignore vendored
View File

@@ -10,4 +10,10 @@ dependencies.lock
.env
releases/
main/assets/lang_config.h
.DS_Store
main/mmap_generate_emoji.h
.DS_Store
.cache
main/mmap_generate_emoji.h
*.pyc
*.bin
mmap_generate_*.h

View File

@@ -4,7 +4,7 @@
# CMakeLists in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.16)
set(PROJECT_VER "1.5.5")
set(PROJECT_VER "1.9.2")
# Add this line to disable the specific warning
add_compile_options(-Wno-missing-field-initializers)

View File

@@ -1,6 +1,7 @@
MIT License
Copyright (c) 2024 Xiaoxia
Copyright (c) 2025 Shenzhen Xinzhi Future Technology Co., Ltd.
Copyright (c) 2025 Project Contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal

119
README.md
View File

@@ -1,42 +1,43 @@
# 小智 AI 聊天机器人 XiaoZhi AI Chatbot
# An MCP-based Chatbot | 一个基于 MCP 的聊天机器人
(中文 | [English](README_en.md) | [日本語](README_ja.md)
这是虾哥的第一个硬件作品。
## 视频
👉 [ESP32+SenseVoice+Qwen72B打造你的AI聊天伴侣【bilibili】](https://www.bilibili.com/video/BV11msTenEH3/)
👉 [给小智装上 DeepSeek 的聪明大脑【bilibili】](https://www.bilibili.com/video/BV1GQP6eNEFG/)
👉 [人类:给 AI 装摄像头 vs AI当场发现主人三天没洗头【bilibili】](https://www.bilibili.com/video/BV1bpjgzKEhd/)
👉 [手工打造你的 AI 女友新手入门教程【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
## 项目目的
## 介绍
本项目是一个开源项目,以 MIT 许可证发布,允许任何人免费使用,并可以用于商业用途。
这是一个由虾哥开源的 ESP32 项目,以 MIT 许可证发布,允许任何人免费使用,用于商业用途。
我们希望通过这个项目,能够帮助更多人入门 AI 硬件开发,了解如何将当下飞速发展的大语言模型应用到实际的硬件设备中。无论你是对 AI 感兴趣的学生,还是想要探索新技术的开发者,都可以通过这个项目获得宝贵的学习经验。
我们希望通过这个项目,能够帮助大家了解 AI 硬件开发,将当下飞速发展的大语言模型应用到实际的硬件设备中。
欢迎所有人参与到项目的开发和改进中来。如果你有任何想法或建议,请随时提出 Issue 或加入群聊。
如果你有任何想法或建议,请随时提出 Issues 或加入 QQ 群1011329060
学习交流 QQ 群376893254
### 基于 MCP 控制万物
## 已实现功能
小智 AI 聊天机器人作为一个语音交互入口,利用 Qwen / DeepSeek 等大模型的 AI 能力,通过 MCP 协议实现多端控制。
![通过MCP控制万物](docs/mcp-based-graph.jpg)
### 已实现功能
- Wi-Fi / ML307 Cat.1 4G
- BOOT 键唤醒和打断,支持点击和长按两种触发方式
- 离线语音唤醒 [ESP-SR](https://github.com/espressif/esp-sr)
- 流式语音对话WebSocket 或 UDP 协议
- 支持国语、粤语、英语、日语、韩语 5 种语言识别 [SenseVoice](https://github.com/FunAudioLLM/SenseVoice)
- 声纹识别,识别是谁在喊 AI 的名字 [3D Speaker](https://github.com/modelscope/3D-Speaker)
- 大模型 TTS火山引擎 或 CosyVoice
- 大模型 LLMQwen, DeepSeek, Doubao
- 可配置的提示词和音色(自定义角色)
- 短期记忆,每轮对话后自我总结
- OLED / LCD 显示屏,显示信号强弱或对话内容
- 支持 LCD 显示图片表情
- 支持多语言(中文、英文
- 支持两种通信协议([Websocket](docs/websocket.md) 或 MQTT+UDP
- 采用 OPUS 音频编解码
- 基于流式 ASR + LLM + TTS 架构的语音交互
- 声纹识别,识别当前说话人的身份 [3D Speaker](https://github.com/modelscope/3D-Speaker)
- OLED / LCD 显示屏,支持表情显示
- 电量显示与电源管理
- 支持多语言(中文、英文、日文)
- 支持 ESP32-C3、ESP32-S3、ESP32-P4 芯片平台
- 通过设备端 MCP 实现设备控制音量、灯光、电机、GPIO 等)
- 通过云端 MCP 扩展大模型能力智能家居控制、PC桌面操作、知识搜索、邮件收发等
## 硬件部分
## 硬件
### 面包板手工制作实践
@@ -46,24 +47,23 @@
面包板效果图如下:
![面包板效果图](docs/wiring2.jpg)
![面包板效果图](docs/v1/wiring2.jpg)
### 支持的开源硬件
### 支持 70 多个开源硬件(仅展示部分)
- <a href="https://oshwhub.com/li-chuang-kai-fa-ban/li-chuang-shi-zhan-pai-esp32-s3-kai-fa-ban" target="_blank" title="立创·实战派 ESP32-S3 开发板">立创·实战派 ESP32-S3 开发板</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="乐鑫 ESP32-S3-BOX3">乐鑫 ESP32-S3-BOX3</a>
- <a href="https://docs.m5stack.com/zh_CN/core/CoreS3" target="_blank" title="M5Stack CoreS3">M5Stack CoreS3</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">AtomS3R + Echo Base</a>
- <a href="https://docs.m5stack.com/en/core/ATOM%20Matrix" target="_blank" title="AtomMatrix + Echo Base">AtomMatrix + Echo Base</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">M5Stack AtomS3R + Echo Base</a>
- <a href="https://gf.bilibili.com/item/detail/1108782064" target="_blank" title="神奇按钮 2.4">神奇按钮 2.4</a>
- <a href="https://www.waveshare.net/shop/ESP32-S3-Touch-AMOLED-1.8.htm" target="_blank" title="微雪电子 ESP32-S3-Touch-AMOLED-1.8">微雪电子 ESP32-S3-Touch-AMOLED-1.8</a>
- <a href="https://github.com/Xinyuan-LilyGO/T-Circle-S3" target="_blank" title="LILYGO T-Circle-S3">LILYGO T-Circle-S3</a>
- <a href="https://oshwhub.com/tenclass01/xmini_c3" target="_blank" title="虾哥 Mini C3">虾哥 Mini C3</a>
- <a href="https://oshwhub.com/movecall/moji-xiaozhi-ai-derivative-editi" target="_blank" title="Movecall Moji ESP32S3">Moji 小智AI衍生版</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">璀璨·AI吊坠</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="无名科技Nologo-星智-1.54">无名科技Nologo-星智-1.54TFT</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="无名科技Nologo-星智-0.96">无名科技Nologo-星智-0.96TFT</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">璀璨·AI 吊坠</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="无名科技Nologo-星智-1.54">无名科技 Nologo-星智-1.54TFT</a>
- <a href="https://www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html" target="_blank" title="SenseCAP Watcher">SenseCAP Watcher</a>
- <a href="https://www.bilibili.com/video/BV1BHJtz6E2S/" target="_blank" title="ESP-HI 超低成本机器狗">ESP-HI 超低成本机器狗</a>
<div style="display: flex; justify-content: space-between;">
<a href="docs/v1/lichuang-s3.jpg" target="_blank" title="立创·实战派 ESP32-S3 开发板">
<img src="docs/v1/lichuang-s3.jpg" width="240" />
@@ -83,14 +83,11 @@
<a href="docs/v1/waveshare.jpg" target="_blank" title="微雪电子 ESP32-S3-Touch-AMOLED-1.8">
<img src="docs/v1/waveshare.jpg" width="240" />
</a>
<a href="docs/lilygo-t-circle-s3.jpg" target="_blank" title="LILYGO T-Circle-S3">
<img src="docs/lilygo-t-circle-s3.jpg" width="240" />
<a href="docs/v1/lilygo-t-circle-s3.jpg" target="_blank" title="LILYGO T-Circle-S3">
<img src="docs/v1/lilygo-t-circle-s3.jpg" width="240" />
</a>
<a href="docs/xmini-c3.jpg" target="_blank" title="虾哥 Mini C3">
<img src="docs/xmini-c3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-moji-esp32s3.jpg" target="_blank" title="Movecall Moji 小智AI衍生版">
<img src="docs/v1/movecall-moji-esp32s3.jpg" width="240" />
<a href="docs/v1/xmini-c3.jpg" target="_blank" title="虾哥 Mini C3">
<img src="docs/v1/xmini-c3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-cuican-esp32s3.jpg" target="_blank" title="CuiCan">
<img src="docs/v1/movecall-cuican-esp32s3.jpg" width="240" />
@@ -98,44 +95,60 @@
<a href="docs/v1/wmnologo_xingzhi_1.54.jpg" target="_blank" title="无名科技Nologo-星智-1.54">
<img src="docs/v1/wmnologo_xingzhi_1.54.jpg" width="240" />
</a>
<a href="docs/v1/wmnologo_xingzhi_0.96.jpg" target="_blank" title="无名科技Nologo-星智-0.96">
<img src="docs/v1/wmnologo_xingzhi_0.96.jpg" width="240" />
</a>
<a href="docs/v1/sensecap_watcher.jpg" target="_blank" title="SenseCAP Watcher">
<img src="docs/v1/sensecap_watcher.jpg" width="240" />
</a>
<a href="docs/v1/esp-hi.jpg" target="_blank" title="ESP-HI 超低成本机器狗">
<img src="docs/v1/esp-hi.jpg" width="240" />
</a>
</div>
## 固件部分
## 软件
### 免开发环境烧录
### 固件烧录
新手第一次操作建议先不要搭建开发环境,直接使用免开发环境烧录的固件。
固件默认接入 [xiaozhi.me](https://xiaozhi.me) 官方服务器,目前个人用户注册账号可以免费使用 Qwen 实时模型。
👉 [Flash烧录固件无IDF开发环境](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)
固件默认接入 [xiaozhi.me](https://xiaozhi.me) 官方服务器,个人用户注册账号可以免费使用 Qwen 实时模型。
👉 [新手烧录固件教程](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)
### 开发环境
- Cursor 或 VSCode
- 安装 ESP-IDF 插件,选择 SDK 版本 5.3 或以上
- 安装 ESP-IDF 插件,选择 SDK 版本 5.4 或以上
- Linux 比 Windows 更好,编译速度快,也免去驱动问题的困扰
- 使用 Google C++ 代码风格,提交代码时请确保符合规范
- 本项目使用 Google C++ 代码风格,提交代码时请确保符合规范
### 开发者文档
## 智能体配置
- [自定义开发板指南](main/boards/README.md) - 学习如何为小智 AI 创建自定义开发板
- [MCP 协议物联网控制用法说明](docs/mcp-usage.md) - 了解如何通过 MCP 协议控制物联网设备
- [MCP 协议交互流程](docs/mcp-protocol.md) - 设备端 MCP 协议的实现方式
- [MQTT + UDP 混合通信协议文档](docs/mqtt-udp.md)
- [一份详细的 WebSocket 通信协议文档](docs/websocket.md)
如果你已经拥有一个小智 AI 聊天机器人设备,可以登录 [xiaozhi.me](https://xiaozhi.me) 控制台进行配置
## 大模型配置
如果你已经拥有一个小智 AI 聊天机器人设备,并且已接入官方服务器,可以登录 [xiaozhi.me](https://xiaozhi.me) 控制台进行配置。
👉 [后台操作视频教程(旧版界面)](https://www.bilibili.com/video/BV1jUCUY2EKM/)
## 技术原理与私有化部署
## 相关开源项目
👉 [一份详细的 WebSocket 通信协议文档](docs/websocket.md)
在个人电脑上部署服务器,可以参考以下第三方开源的项目:
在个人电脑上部署服务器,可以参考另一位作者同样以 MIT 许可证开源的项目 [xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server)
- [xinnan-tech/xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) Python 服务器
- [joey-zhou/xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) Java 服务器
- [AnimeAIChat/xiaozhi-server-go](https://github.com/AnimeAIChat/xiaozhi-server-go) Golang 服务器
使用小智通信协议的第三方客户端项目:
- [huangjunsen0406/py-xiaozhi](https://github.com/huangjunsen0406/py-xiaozhi) Python 客户端
- [TOM88812/xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) Android 客户端
- [100askTeam/xiaozhi-linux](http://github.com/100askTeam/xiaozhi-linux) 百问科技提供的 Linux 客户端
- [78/xiaozhi-sf32](https://github.com/78/xiaozhi-sf32) 思澈科技的蓝牙芯片固件
- [QuecPython/solution-xiaozhiAI](https://github.com/QuecPython/solution-xiaozhiAI) 移远提供的 QuecPython 固件
## Star History

View File

@@ -1,67 +1,68 @@
# XiaoZhi AI Chatbot
# An MCP-based Chatbot
([中文](README.md) | English | [日本語](README_ja.md))
(English | [中文](README.md) | [日本語](README_ja.md))
This is Terrence's first hardware project.
## Video
👉 [Build your AI chat companion with ESP32+SenseVoice+Qwen72B!【bilibili】](https://www.bilibili.com/video/BV11msTenEH3/)
👉 [Human: Give AI a camera vs AI: Instantly finds out the owner hasn't washed hair for three days【bilibili】](https://www.bilibili.com/video/BV1bpjgzKEhd/)
👉 [Equipping XiaoZhi with DeepSeek's smart brain【bilibili】](https://www.bilibili.com/video/BV1GQP6eNEFG/)
👉 [Handcraft your AI girlfriend, beginner's guide【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
👉 [Build your own AI companion, a beginner's guide【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
## Introduction
## Project Purpose
This is an open-source ESP32 project, released under the MIT license, allowing anyone to use it for free, including for commercial purposes.
This is an open-source project released under the MIT license, allowing anyone to use it freely, including for commercial purposes.
We hope this project helps everyone understand AI hardware development and apply rapidly evolving large language models to real hardware devices.
Through this project, we aim to help more people get started with AI hardware development and understand how to implement rapidly evolving large language models in actual hardware devices. Whether you're a student interested in AI or a developer exploring new technologies, this project offers valuable learning experiences.
If you have any ideas or suggestions, please feel free to raise Issues or join the QQ group: 1011329060
Everyone is welcome to participate in the project's development and improvement. If you have any ideas or suggestions, please feel free to raise an Issue or join the chat group.
### Control Everything with MCP
Learning & Discussion QQ Group: 376893254
As a voice interaction entry, the XiaoZhi AI chatbot leverages the AI capabilities of large models like Qwen / DeepSeek, and achieves multi-terminal control via the MCP protocol.
## Implemented Features
![Control everything via MCP](docs/mcp-based-graph.jpg)
### Features Implemented
- Wi-Fi / ML307 Cat.1 4G
- BOOT button wake-up and interruption, supporting both click and long-press triggers
- Offline voice wake-up [ESP-SR](https://github.com/espressif/esp-sr)
- Streaming voice dialogue (WebSocket or UDP protocol)
- Support for 5 languages: Mandarin, Cantonese, English, Japanese, Korean [SenseVoice](https://github.com/FunAudioLLM/SenseVoice)
- Voice print recognition to identify who's calling AI's name [3D Speaker](https://github.com/modelscope/3D-Speaker)
- Large model TTS (Volcano Engine or CosyVoice)
- Large Language Models (Qwen, DeepSeek, Doubao)
- Configurable prompts and voice tones (custom characters)
- Short-term memory, self-summarizing after each conversation round
- OLED / LCD display showing signal strength or conversation content
- Support for LCD image expressions
- Multi-language support (Chinese, English)
- Supports two communication protocols ([Websocket](docs/websocket.md) or MQTT+UDP)
- Uses OPUS audio codec
- Voice interaction based on streaming ASR + LLM + TTS architecture
- Speaker recognition, identifies the current speaker [3D Speaker](https://github.com/modelscope/3D-Speaker)
- OLED / LCD display, supports emoji display
- Battery display and power management
- Multi-language support (Chinese, English, Japanese)
- Supports ESP32-C3, ESP32-S3, ESP32-P4 chip platforms
- Device-side MCP for device control (Speaker, LED, Servo, GPIO, etc.)
- Cloud-side MCP to extend large model capabilities (smart home control, PC desktop operation, knowledge search, email, etc.)
## Hardware Section
## Hardware
### Breadboard DIY Practice
See the Feishu document tutorial:
👉 [XiaoZhi AI Chatbot Encyclopedia](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb?from=from_copylink)
👉 ["XiaoZhi AI Chatbot Encyclopedia"](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb?from=from_copylink)
Breadboard demonstration:
Breadboard demo:
![Breadboard Demo](docs/wiring2.jpg)
![Breadboard Demo](docs/v1/wiring2.jpg)
### Supported Open Source Hardware
### Supports 70+ Open Source Hardware (Partial List)
- <a href="https://oshwhub.com/li-chuang-kai-fa-ban/li-chuang-shi-zhan-pai-esp32-s3-kai-fa-ban" target="_blank" title="LiChuang ESP32-S3 Development Board">LiChuang ESP32-S3 Development Board</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="Espressif ESP32-S3-BOX3">Espressif ESP32-S3-BOX3</a>
- <a href="https://docs.m5stack.com/zh_CN/core/CoreS3" target="_blank" title="M5Stack CoreS3">M5Stack CoreS3</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">AtomS3R + Echo Base</a>
- <a href="https://docs.m5stack.com/en/core/ATOM%20Matrix" target="_blank" title="AtomMatrix + Echo Base">AtomMatrix + Echo Base</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">M5Stack AtomS3R + Echo Base</a>
- <a href="https://gf.bilibili.com/item/detail/1108782064" target="_blank" title="Magic Button 2.4">Magic Button 2.4</a>
- <a href="https://www.waveshare.net/shop/ESP32-S3-Touch-AMOLED-1.8.htm" target="_blank" title="Waveshare ESP32-S3-Touch-AMOLED-1.8">Waveshare ESP32-S3-Touch-AMOLED-1.8</a>
- <a href="https://github.com/Xinyuan-LilyGO/T-Circle-S3" target="_blank" title="LILYGO T-Circle-S3">LILYGO T-Circle-S3</a>
- <a href="https://oshwhub.com/tenclass01/xmini_c3" target="_blank" title="XiaGe Mini C3">XiaGe Mini C3</a>
- <a href="https://oshwhub.com/movecall/moji-xiaozhi-ai-derivative-editi" target="_blank" title="Movecall Moji ESP32S3">Moji XiaoZhi AI Derivative Version</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">CuiCan AI pendant</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">CuiCan AI Pendant</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="WMnologo-Xingzhi-1.54">WMnologo-Xingzhi-1.54TFT</a>
- <a href="https://www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html" target="_blank" title="SenseCAP Watcher">SenseCAP Watcher</a>
- <a href="https://www.bilibili.com/video/BV1BHJtz6E2S/" target="_blank" title="ESP-HI Low Cost Robot Dog">ESP-HI Low Cost Robot Dog</a>
<div style="display: flex; justify-content: space-between;">
<a href="docs/v1/lichuang-s3.jpg" target="_blank" title="LiChuang ESP32-S3 Development Board">
@@ -76,60 +77,74 @@ Breadboard demonstration:
<a href="docs/v1/atoms3r.jpg" target="_blank" title="AtomS3R + Echo Base">
<img src="docs/v1/atoms3r.jpg" width="240" />
</a>
<a href="docs/AtomMatrix-echo-base.jpg" target="_blank" title="AtomMatrix-echo-base + Echo Base">
<img src="docs/AtomMatrix-echo-base.jpg" width="240" />
</a>
<a href="docs/v1/magiclick.jpg" target="_blank" title="MagiClick 2.4">
<a href="docs/v1/magiclick.jpg" target="_blank" title="Magic Button 2.4">
<img src="docs/v1/magiclick.jpg" width="240" />
</a>
<a href="docs/v1/waveshare.jpg" target="_blank" title="Waveshare ESP32-S3-Touch-AMOLED-1.8">
<img src="docs/v1/waveshare.jpg" width="240" />
</a>
<a href="docs/lilygo-t-circle-s3.jpg" target="_blank" title="LILYGO T-Circle-S3">
<img src="docs/lilygo-t-circle-s3.jpg" width="240" />
<a href="docs/v1/lilygo-t-circle-s3.jpg" target="_blank" title="LILYGO T-Circle-S3">
<img src="docs/v1/lilygo-t-circle-s3.jpg" width="240" />
</a>
<a href="docs/xmini-c3.jpg" target="_blank" title="Xmini C3">
<img src="docs/xmini-c3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-moji-esp32s3.jpg" target="_blank" title="Moji">
<img src="docs/v1/movecall-moji-esp32s3.jpg" width="240" />
<a href="docs/v1/xmini-c3.jpg" target="_blank" title="XiaGe Mini C3">
<img src="docs/v1/xmini-c3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-cuican-esp32s3.jpg" target="_blank" title="CuiCan">
<img src="docs/v1/movecall-cuican-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/wmnologo_xingzhi_1.54.jpg" target="_blank" title="WMnologo-Xingzhi-1.54">
<img src="docs/v1/wmnologo_xingzhi_1.54.jpg" width="240" />
</a>
<a href="docs/v1/sensecap_watcher.jpg" target="_blank" title="SenseCAP Watcher">
<img src="docs/v1/sensecap_watcher.jpg" width="240" />
</a>
<a href="docs/v1/esp-hi.jpg" target="_blank" title="ESP-HI Low Cost Robot Dog">
<img src="docs/v1/esp-hi.jpg" width="240" />
</a>
</div>
## Firmware Section
## Software
### Flashing Without Development Environment
### Firmware Flashing
For beginners, it's recommended to first use the firmware that can be flashed without setting up a development environment.
For beginners, it is recommended to use the firmware that can be flashed without setting up a development environment.
The firmware connects to the official [xiaozhi.me](https://xiaozhi.me) server by default. Currently, personal users can register an account to use the Qwen real-time model for free.
The firmware connects to the official [xiaozhi.me](https://xiaozhi.me) server by default. Personal users can register an account to use the Qwen real-time model for free.
👉 [Flash Firmware Guide (No IDF Environment)](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)
👉 [Beginner's Firmware Flashing Guide](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)
### Development Environment
- Cursor or VSCode
- Install ESP-IDF plugin, select SDK version 5.3 or above
- Linux is preferred over Windows for faster compilation and fewer driver issues
- Use Google C++ code style, ensure compliance when submitting code
- Install ESP-IDF plugin, select SDK version 5.4 or above
- Linux is better than Windows for faster compilation and fewer driver issues
- This project uses Google C++ code style, please ensure compliance when submitting code
## AI Agent Configuration
### Developer Documentation
If you already have a XiaoZhi AI chatbot device, you can configure it through the [xiaozhi.me](https://xiaozhi.me) console.
- [Custom Board Guide](main/boards/README.md) - Learn how to create custom boards for XiaoZhi AI
- [MCP Protocol IoT Control Usage](docs/mcp-usage.md) - Learn how to control IoT devices via MCP protocol
- [MCP Protocol Interaction Flow](docs/mcp-protocol.md) - Device-side MCP protocol implementation
- [A detailed WebSocket communication protocol document](docs/websocket.md)
👉 [Backend Operation Tutorial (Old Interface)](https://www.bilibili.com/video/BV1jUCUY2EKM/)
## Large Model Configuration
## Technical Principles and Private Deployment
If you already have a XiaoZhi AI chatbot device and have connected to the official server, you can log in to the [xiaozhi.me](https://xiaozhi.me) console for configuration.
👉 [Detailed WebSocket Communication Protocol Documentation](docs/websocket.md)
👉 [Backend Operation Video Tutorial (Old Interface)](https://www.bilibili.com/video/BV1jUCUY2EKM/)
For server deployment on personal computers, refer to another MIT-licensed project [xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server)
## Related Open Source Projects
For server deployment on personal computers, refer to the following open-source projects:
- [xinnan-tech/xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) Python server
- [joey-zhou/xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) Java server
- [AnimeAIChat/xiaozhi-server-go](https://github.com/AnimeAIChat/xiaozhi-server-go) Golang server
Other client projects using the XiaoZhi communication protocol:
- [huangjunsen0406/py-xiaozhi](https://github.com/huangjunsen0406/py-xiaozhi) Python client
- [TOM88812/xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) Android client
## Star History

View File

@@ -1,73 +1,74 @@
# シャオジー AI チャットボット
# MCP ベースのチャットボット
([中文](README.md) | [English](README_en.md) | 日本語)
(日本語 | [中文](README.md) | [English](README_en.md)
これは シャーガーTerrenceの最初のハードウェア作品です。
## 動画
👉 [ESP32+SenseVoice+Qwen72Bで AI チャット仲間を作ろう!【bilibili】](https://www.bilibili.com/video/BV11msTenEH3/)
👉 [人間AIにカメラを装着 vs AIその場で飼い主が3日間髪を洗っていないことを発見【bilibili】](https://www.bilibili.com/video/BV1bpjgzKEhd/)
👉 [シャオジーに DeepSeek のスマートな頭脳を搭載【bilibili】](https://www.bilibili.com/video/BV1GQP6eNEFG/)
👉 [手作りでAIガールフレンドを作る、初心者入門チュートリアル【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
👉 [自分だけの AI パートナーを作る、初心者向けガイド【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
## イントロダクション
## プロジェクトの目的
これはエビ兄さんがオープンソースで公開しているESP32プロジェクトで、MITライセンスのもと、誰でも無料で、商用利用も可能です。
このプロジェクトは MIT ライセンスの下で公開されているオープンソースプロジェクトで、商用利用を含め、誰でも自由に使用することができます。
このプロジェクトを通じて、AIハードウェア開発を理解し、急速に進化する大規模言語モデルを実際のハードウェアデバイスに応用できるようになることを目指しています。
このプロジェクトを通じて、より多くの人々が AI ハードウェア開発を始め、急速に進化している大規模言語モデルを実際のハードウェアデバイスに実装する方法を理解できるようになることを目指しています。AI に興味のある学生でも、新しい技術を探求する開発者でも、このプロジェクトから貴重な学習経験を得ることができます
ご意見やご提案があれば、いつでもIssueを提出するか、QQグループ1011329060 にご参加ください
プロジェクトの開発と改善には誰でも参加できます。アイデアや提案がありましたら、Issue を立てるかチャットグループにご参加ください。
### MCPであらゆるものを制御
学習・交流 QQ グループ376893254
シャオジーAIチャットボットは音声インタラクションの入口として、Qwen / DeepSeekなどの大規模モデルのAI能力を活用し、MCPプロトコルを通じてマルチエンド制御を実現します。
## 実装済みの機能
![MCPであらゆるものを制御](docs/mcp-based-graph.jpg)
### 実装済み機能
- Wi-Fi / ML307 Cat.1 4G
- BOOT ボタンによる起動と中断、クリックと長押しの2種類のトリガーに対応
- オフライン音声起動 [ESP-SR](https://github.com/espressif/esp-sr)
- ストリーミング音声対話WebSocket または UDP プロトコル)
- 5言語対応標準中国語、広東語、英語、日本語、韓国語 [SenseVoice](https://github.com/FunAudioLLM/SenseVoice)
- 話者認識、AI の名前を呼んでいる人を識別 [3D Speaker](https://github.com/modelscope/3D-Speaker)
- 大規模モデル TTSVolcano Engine または CosyVoice
- 大規模言語モデルQwen, DeepSeek, Doubao
- 設定可能なプロンプトと音声トーン(カスタムキャラクター
- 短期記憶、各会話ラウンド後の自己要約
- OLED / LCD ディスプレイ、信号強度や会話内容を表示
- LCD での画像表情表示に対応
- 多言語対応(中国語、英語)
- オフライン音声ウェイクアップ [ESP-SR](https://github.com/espressif/esp-sr)
- 2種類の通信プロトコルに対応[Websocket](docs/websocket.md) または MQTT+UDP
- OPUSオーディオコーデックを採用
- ストリーミングASR + LLM + TTSアーキテクチャに基づく音声インタラクション
- 話者認識、現在話している人を識別 [3D Speaker](https://github.com/modelscope/3D-Speaker)
- OLED / LCDディスプレイ、表情表示対応
- バッテリー表示と電源管理
- 多言語対応(中国語、英語、日本語
- ESP32-C3、ESP32-S3、ESP32-P4チッププラットフォーム対応
- デバイス側MCPによるデバイス制御音量・明るさ調整、アクション制御など
- クラウド側MCPで大規模モデル能力を拡張スマートホーム制御、PCデスクトップ操作、知識検索、メール送受信など
## ハードウェア部分
## ハードウェア
### ブレッドボード DIY 実践
### ブレッドボード手作り実践
Feishu ドキュメントチュートリアルをご覧ください:
Feishuドキュメントチュートリアルをご覧ください
👉 [シャオジー AI チャットボット百科事典](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb?from=from_copylink)
👉 [シャオジーAIチャットボット百科事典](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb?from=from_copylink)
ブレッドボードのデモ:
![ブレッドボードデモ](docs/wiring2.jpg)
![ブレッドボードデモ](docs/v1/wiring2.jpg)
### サポートされているオープンソースハードウェア
### 70種類以上のオープンソースハードウェアに対応(一部のみ表示)
- <a href="https://oshwhub.com/li-chuang-kai-fa-ban/li-chuang-shi-zhan-pai-esp32-s3-kai-fa-ban" target="_blank" title="LiChuang ESP32-S3 開発ボード">LiChuang ESP32-S3 開発ボード</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="Espressif ESP32-S3-BOX3">Espressif ESP32-S3-BOX3</a>
- <a href="https://oshwhub.com/li-chuang-kai-fa-ban/li-chuang-shi-zhan-pai-esp32-s3-kai-fa-ban" target="_blank" title="立創・実戦派 ESP32-S3 開発ボード">立創・実戦派 ESP32-S3 開発ボード</a>
- <a href="https://github.com/espressif/esp-box" target="_blank" title="楽鑫 ESP32-S3-BOX3">楽鑫 ESP32-S3-BOX3</a>
- <a href="https://docs.m5stack.com/zh_CN/core/CoreS3" target="_blank" title="M5Stack CoreS3">M5Stack CoreS3</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">AtomS3R + Echo Base</a>
- <a href="https://docs.m5stack.com/en/core/ATOM%20Matrix" target="_blank" title="AtomMatrix + Echo Base">AtomMatrix + Echo Base</a>
- <a href="https://gf.bilibili.com/item/detail/1108782064" target="_blank" title="マジックボタン 2.4">マジックボタン 2.4</a>
- <a href="https://www.waveshare.net/shop/ESP32-S3-Touch-AMOLED-1.8.htm" target="_blank" title="Waveshare ESP32-S3-Touch-AMOLED-1.8">Waveshare ESP32-S3-Touch-AMOLED-1.8</a>
- <a href="https://docs.m5stack.com/en/atom/Atomic%20Echo%20Base" target="_blank" title="AtomS3R + Echo Base">M5Stack AtomS3R + Echo Base</a>
- <a href="https://gf.bilibili.com/item/detail/1108782064" target="_blank" title="マジックボタン2.4">マジックボタン2.4</a>
- <a href="https://www.waveshare.net/shop/ESP32-S3-Touch-AMOLED-1.8.htm" target="_blank" title="微雪電子 ESP32-S3-Touch-AMOLED-1.8">微雪電子 ESP32-S3-Touch-AMOLED-1.8</a>
- <a href="https://github.com/Xinyuan-LilyGO/T-Circle-S3" target="_blank" title="LILYGO T-Circle-S3">LILYGO T-Circle-S3</a>
- <a href="https://oshwhub.com/tenclass01/xmini_c3" target="_blank" title="XiaGe Mini C3">XiaGe Mini C3</a>
- <a href="https://oshwhub.com/movecall/moji-xiaozhi-ai-derivative-editi" target="_blank" title="Movecall Moji ESP32S3">Moji シャオジー AI 派生版</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">Cuican AI ペンダント</a>
- <a href="https://oshwhub.com/tenclass01/xmini_c3" target="_blank" title="エビ兄さん Mini C3">エビ兄さん Mini C3</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">CuiCan AIペンダント</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="無名科技Nologo-星智-1.54">無名科技Nologo-星智-1.54TFT</a>
- <a href="https://www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html" target="_blank" title="SenseCAP Watcher">SenseCAP Watcher</a>
- <a href="https://www.bilibili.com/video/BV1BHJtz6E2S/" target="_blank" title="ESP-HI 超低コストロボット犬">ESP-HI 超低コストロボット犬</a>
<div style="display: flex; justify-content: space-between;">
<a href="docs/v1/lichuang-s3.jpg" target="_blank" title="LiChuang ESP32-S3 開発ボード">
<a href="docs/v1/lichuang-s3.jpg" target="_blank" title="立創・実戦派 ESP32-S3 開発ボード">
<img src="docs/v1/lichuang-s3.jpg" width="240" />
</a>
<a href="docs/v1/espbox3.jpg" target="_blank" title="Espressif ESP32-S3-BOX3">
<a href="docs/v1/espbox3.jpg" target="_blank" title="楽鑫 ESP32-S3-BOX3">
<img src="docs/v1/espbox3.jpg" width="240" />
</a>
<a href="docs/v1/m5cores3.jpg" target="_blank" title="M5Stack CoreS3">
@@ -76,57 +77,74 @@ Feishu ドキュメントチュートリアルをご覧ください:
<a href="docs/v1/atoms3r.jpg" target="_blank" title="AtomS3R + Echo Base">
<img src="docs/v1/atoms3r.jpg" width="240" />
</a>
<a href="docs/v1/magiclick.jpg" target="_blank" title="MagiClick 2.4">
<a href="docs/v1/magiclick.jpg" target="_blank" title="マジックボタン2.4">
<img src="docs/v1/magiclick.jpg" width="240" />
</a>
<a href="docs/v1/waveshare.jpg" target="_blank" title="Waveshare ESP32-S3-Touch-AMOLED-1.8">
<a href="docs/v1/waveshare.jpg" target="_blank" title="微雪電子 ESP32-S3-Touch-AMOLED-1.8">
<img src="docs/v1/waveshare.jpg" width="240" />
</a>
<a href="docs/lilygo-t-circle-s3.jpg" target="_blank" title="LILYGO T-Circle-S3">
<img src="docs/lilygo-t-circle-s3.jpg" width="240" />
<a href="docs/v1/lilygo-t-circle-s3.jpg" target="_blank" title="LILYGO T-Circle-S3">
<img src="docs/v1/lilygo-t-circle-s3.jpg" width="240" />
</a>
<a href="docs/xmini-c3.jpg" target="_blank" title="Xmini C3">
<img src="docs/xmini-c3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-moji-esp32s3.jpg" target="_blank" title="Moji">
<img src="docs/v1/movecall-moji-esp32s3.jpg" width="240" />
<a href="docs/v1/xmini-c3.jpg" target="_blank" title="エビ兄さん Mini C3">
<img src="docs/v1/xmini-c3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-cuican-esp32s3.jpg" target="_blank" title="CuiCan">
<img src="docs/v1/movecall-cuican-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/wmnologo_xingzhi_1.54.jpg" target="_blank" title="無名科技Nologo-星智-1.54">
<img src="docs/v1/wmnologo_xingzhi_1.54.jpg" width="240" />
</a>
<a href="docs/v1/sensecap_watcher.jpg" target="_blank" title="SenseCAP Watcher">
<img src="docs/v1/sensecap_watcher.jpg" width="240" />
</a>
<a href="docs/v1/esp-hi.jpg" target="_blank" title="ESP-HI 超低コストロボット犬">
<img src="docs/v1/esp-hi.jpg" width="240" />
</a>
</div>
## ファームウェア部分
## ソフトウェア
### 開発環境なしのフラッシュ
### ファームウェア書き込み
初心者の方は、まず開発環境のセットアップなしでフラッシュできるファームウェアを使用することをおめします。
初心者の方は、まず開発環境を構築せずに書き込み可能なファームウェアを使用することをおすすめします。
ファームウェアはデフォルトで公式 [xiaozhi.me](https://xiaozhi.me) サーバーに接続します。現在、個人ユーザーはアカウント登録することで、Qwen リアルタイムモデルを無料で使用できます。
ファームウェアはデフォルトで公式 [xiaozhi.me](https://xiaozhi.me) サーバーに接続します。個人ユーザーはアカウント登録Qwenリアルタイムモデルを無料で用できます。
👉 [フラッシュファームウェアガイドIDF環境なし](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)
👉 [初心者向けファームウェア書き込みガイド](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)
### 開発環境
- Cursor または VSCode
- ESP-IDF プラグインをインストール、SDK バージョン 5.3 以上を選択
- LinuxWindows より好ましい(コンパイルが速く、ドライバの問題も少ない
- Google C++ コードスタイルを使用、コード提出時にはコンプライアンスを確認
- ESP-IDFプラグインをインストール、SDKバージョン5.4以上を選択
- LinuxWindowsよりも優れており、コンパイルが速く、ドライバの問題も少ない
- 本プロジェクトはGoogle C++コードスタイルを用、コード提出時は準拠を確認してください
## AI エージェント設定
### 開発者ドキュメント
シャオジー AI チャットボットデバイスをお持ちの場合は、[xiaozhi.me](https://xiaozhi.me) コンソールで設定できます。
- [カスタム開発ボードガイド](main/boards/README.md) - シャオジーAI用のカスタム開発ボード作成方法
- [MCPプロトコルIoT制御使用法](docs/mcp-usage.md) - MCPプロトコルでIoTデバイスを制御する方法
- [MCPプロトコルインタラクションフロー](docs/mcp-protocol.md) - デバイス側MCPプロトコルの実装方法
- [詳細なWebSocket通信プロトコルドキュメント](docs/websocket.md)
👉 [バックエンド操作チュートリアル(旧インターフェース)](https://www.bilibili.com/video/BV1jUCUY2EKM/)
## 大規模モデル設定
## 技術原理とプライベートデプロイメント
すでにシャオジーAIチャットボットデバイスをお持ちで、公式サーバーに接続済みの場合は、[xiaozhi.me](https://xiaozhi.me) コンソールで設定できます。
👉 [詳細な WebSocket 通信プロトコルドキュメント](docs/websocket.md)
👉 [バックエンド操作ビデオチュートリアル(旧インターフェース)](https://www.bilibili.com/video/BV1jUCUY2EKM/)
個人のコンピュータでのサーバーデプロイメントについては、同じく MIT ライセンスで公開されている別のプロジェクト [xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) を参照してください。
## 関連オープンソースプロジェクト
個人PCでサーバーをデプロイする場合は、以下のオープンソースプロジェクトを参照してください
- [xinnan-tech/xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) Pythonサーバー
- [joey-zhou/xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) Javaサーバー
- [AnimeAIChat/xiaozhi-server-go](https://github.com/AnimeAIChat/xiaozhi-server-go) Golangサーバー
シャオジー通信プロトコルを利用した他のクライアントプロジェクト:
- [huangjunsen0406/py-xiaozhi](https://github.com/huangjunsen0406/py-xiaozhi) Pythonクライアント
- [TOM88812/xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) Androidクライアント
## スター履歴

BIN
docs/mcp-based-graph.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 96 KiB

269
docs/mcp-protocol.md Normal file
View File

@@ -0,0 +1,269 @@
# MCP (Model Context Protocol) 交互流程
NOTICE: AI 辅助生成, 在实现后台服务时, 请参照代码确认细节!!
本项目中的 MCP 协议用于后台 APIMCP 客户端)与 ESP32 设备MCP 服务器)之间的通信,以便后台能够发现和调用设备提供的功能(工具)。
## 协议格式
根据代码 (`main/protocols/protocol.cc`, `main/mcp_server.cc`)MCP 消息是封装在基础通信协议(如 WebSocket 或 MQTT的消息体中的。其内部结构遵循 [JSON-RPC 2.0](https://www.jsonrpc.org/specification) 规范。
整体消息结构示例:
```json
{
"session_id": "...", // 会话 ID
"type": "mcp", // 消息类型,固定为 "mcp"
"payload": { // JSON-RPC 2.0 负载
"jsonrpc": "2.0",
"method": "...", // 方法名 (如 "initialize", "tools/list", "tools/call")
"params": { ... }, // 方法参数 (对于 request)
"id": ..., // 请求 ID (对于 request 和 response)
"result": { ... }, // 方法执行结果 (对于 success response)
"error": { ... } // 错误信息 (对于 error response)
}
}
```
其中,`payload` 部分是标准的 JSON-RPC 2.0 消息:
- `jsonrpc`: 固定的字符串 "2.0"。
- `method`: 要调用的方法名称 (对于 Request)。
- `params`: 方法的参数,一个结构化值,通常为对象 (对于 Request)。
- `id`: 请求的标识符,客户端发送请求时提供,服务器响应时原样返回。用于匹配请求和响应。
- `result`: 方法成功执行时的结果 (对于 Success Response)。
- `error`: 方法执行失败时的错误信息 (对于 Error Response)。
## 交互流程及发送时机
MCP 的交互主要围绕客户端(后台 API发现和调用设备上的“工具”Tool进行。
1. **连接建立与能力通告**
- **时机:** 设备启动并成功连接到后台 API 后。
- **发送方:** 设备。
- **消息:** 设备发送基础协议的 "hello" 消息给后台 API消息中包含设备支持的能力列表例如通过支持 MCP 协议 (`"mcp": true`)。
- **示例 (非 MCP 负载,而是基础协议消息):**
```json
{
"type": "hello",
"version": ...,
"features": {
"mcp": true,
...
},
"transport": "websocket", // 或 "mqtt"
"audio_params": { ... },
"session_id": "..." // 设备收到服务器hello后可能设置
}
```
2. **初始化 MCP 会话**
- **时机:** 后台 API 收到设备 "hello" 消息,确认设备支持 MCP 后,通常作为 MCP 会话的第一个请求发送。
- **发送方:** 后台 API (客户端)。
- **方法:** `initialize`
- **消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"method": "initialize",
"params": {
"capabilities": {
// 客户端能力,可选
// 摄像头视觉相关
"vision": {
"url": "...", //摄像头: 图片处理地址(必须是http地址, 不是websocket地址)
"token": "..." // url token
}
// ... 其他客户端能力
}
},
"id": 1 // 请求 ID
}
```
- **设备响应时机:** 设备收到 `initialize` 请求并处理后。
- **设备响应消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"id": 1, // 匹配请求 ID
"result": {
"protocolVersion": "2024-11-05",
"capabilities": {
"tools": {} // 这里的 tools 似乎不列出详细信息,需要 tools/list
},
"serverInfo": {
"name": "...", // 设备名称 (BOARD_NAME)
"version": "..." // 设备固件版本
}
}
}
```
3. **发现设备工具列表**
- **时机:** 后台 API 需要获取设备当前支持的具体功能(工具)列表及其调用方式时。
- **发送方:** 后台 API (客户端)。
- **方法:** `tools/list`
- **消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"method": "tools/list",
"params": {
"cursor": "" // 用于分页,首次请求为空字符串
},
"id": 2 // 请求 ID
}
```
- **设备响应时机:** 设备收到 `tools/list` 请求并生成工具列表后。
- **设备响应消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"id": 2, // 匹配请求 ID
"result": {
"tools": [ // 工具对象列表
{
"name": "self.get_device_status",
"description": "...",
"inputSchema": { ... } // 参数 schema
},
{
"name": "self.audio_speaker.set_volume",
"description": "...",
"inputSchema": { ... } // 参数 schema
}
// ... 更多工具
],
"nextCursor": "..." // 如果列表很大需要分页,这里会包含下一个请求的 cursor 值
}
}
```
- **分页处理:** 如果 `nextCursor` 字段非空,客户端需要再次发送 `tools/list` 请求,并在 `params` 中带上这个 `cursor` 值以获取下一页工具。
4. **调用设备工具**
- **时机:** 后台 API 需要执行设备上的某个具体功能时。
- **发送方:** 后台 API (客户端)。
- **方法:** `tools/call`
- **消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.audio_speaker.set_volume", // 要调用的工具名称
"arguments": {
// 工具参数,对象格式
"volume": 50 // 参数名及其值
}
},
"id": 3 // 请求 ID
}
```
- **设备响应时机:** 设备收到 `tools/call` 请求,执行相应的工具函数后。
- **设备成功响应消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"id": 3, // 匹配请求 ID
"result": {
"content": [
// 工具执行结果内容
{ "type": "text", "text": "true" } // 示例set_volume 返回 bool
],
"isError": false // 表示成功
}
}
```
- **设备失败响应消息 (MCP payload):**
```json
{
"jsonrpc": "2.0",
"id": 3, // 匹配请求 ID
"error": {
"code": -32601, // JSON-RPC 错误码,例如 Method not found (-32601)
"message": "Unknown tool: self.non_existent_tool" // 错误描述
}
}
```
5. **设备主动发送消息 (Notifications)**
- **时机:** 设备内部发生需要通知后台 API 的事件时(例如,状态变化,虽然代码示例中没有明确的工具发送此类消息,但 `Application::SendMcpMessage` 的存在暗示了设备可能主动发送 MCP 消息)。
- **发送方:** 设备 (服务器)。
- **方法:** 可能是以 `notifications/` 开头的方法名,或者其他自定义方法。
- **消息 (MCP payload):** 遵循 JSON-RPC Notification 格式,没有 `id` 字段。
```json
{
"jsonrpc": "2.0",
"method": "notifications/state_changed", // 示例方法名
"params": {
"newState": "idle",
"oldState": "connecting"
}
// 没有 id 字段
}
```
- **后台 API 处理:** 接收到 Notification 后,后台 API 进行相应的处理,但不回复。
## 交互图
下面是一个简化的交互序列图,展示了主要的 MCP 消息流程:
```mermaid
sequenceDiagram
participant Device as ESP32 Device
participant BackendAPI as 后台 API (Client)
Note over Device, BackendAPI: 建立 WebSocket / MQTT 连接
Device->>BackendAPI: Hello Message (包含 "mcp": true)
BackendAPI->>Device: MCP Initialize Request
Note over BackendAPI: method: initialize
Note over BackendAPI: params: { capabilities: ... }
Device->>BackendAPI: MCP Initialize Response
Note over Device: result: { protocolVersion: ..., serverInfo: ... }
BackendAPI->>Device: MCP Get Tools List Request
Note over BackendAPI: method: tools/list
Note over BackendAPI: params: { cursor: "" }
Device->>BackendAPI: MCP Get Tools List Response
Note over Device: result: { tools: [...], nextCursor: ... }
loop Optional Pagination
BackendAPI->>Device: MCP Get Tools List Request
Note over BackendAPI: method: tools/list
Note over BackendAPI: params: { cursor: "..." }
Device->>BackendAPI: MCP Get Tools List Response
Note over Device: result: { tools: [...], nextCursor: "" }
end
BackendAPI->>Device: MCP Call Tool Request
Note over BackendAPI: method: tools/call
Note over BackendAPI: params: { name: "...", arguments: { ... } }
alt Tool Call Successful
Device->>BackendAPI: MCP Tool Call Success Response
Note over Device: result: { content: [...], isError: false }
else Tool Call Failed
Device->>BackendAPI: MCP Tool Call Error Response
Note over Device: error: { code: ..., message: ... }
end
opt Device Notification
Device->>BackendAPI: MCP Notification
Note over Device: method: notifications/...
Note over Device: params: { ... }
end
```
这份文档概述了该项目中 MCP 协议的主要交互流程。具体的参数细节和工具功能需要参考 `main/mcp_server.cc` 中 `McpServer::AddCommonTools` 以及各个工具的实现。

115
docs/mcp-usage.md Normal file
View File

@@ -0,0 +1,115 @@
# MCP 协议物联网控制用法说明
> 本文档介绍如何基于 MCP 协议实现 ESP32 设备的物联网控制。详细协议流程请参考 [`mcp-protocol.md`](./mcp-protocol.md)。
## 简介
MCPModel Context Protocol是新一代推荐用于物联网控制的协议通过标准 JSON-RPC 2.0 格式在后台与设备间发现和调用"工具"Tool实现灵活的设备控制。
## 典型使用流程
1. 设备启动后通过基础协议(如 WebSocket/MQTT与后台建立连接。
2. 后台通过 MCP 协议的 `initialize` 方法初始化会话。
3. 后台通过 `tools/list` 获取设备支持的所有工具(功能)及参数说明。
4. 后台通过 `tools/call` 调用具体工具,实现对设备的控制。
详细协议格式与交互请见 [`mcp-protocol.md`](./mcp-protocol.md)。
## 设备端工具注册方法说明
设备通过 `McpServer::AddTool` 方法注册可被后台调用的"工具"。其常用函数签名如下:
```cpp
void AddTool(
const std::string& name, // 工具名称,建议唯一且有层次感,如 self.dog.forward
const std::string& description, // 工具描述,简明说明功能,便于大模型理解
const PropertyList& properties, // 输入参数列表(可为空),支持类型:布尔、整数、字符串
std::function<ReturnValue(const PropertyList&)> callback // 工具被调用时的回调实现
);
```
- name工具唯一标识建议用"模块.功能"命名风格。
- description自然语言描述便于 AI/用户理解。
- properties参数列表支持类型有布尔、整数、字符串可指定范围和默认值。
- callback收到调用请求时的实际执行逻辑返回值可为 bool/int/string。
## 典型注册示例(以 ESP-Hi 为例)
```cpp
void InitializeTools() {
auto& mcp_server = McpServer::GetInstance();
// 例1无参数控制机器人前进
mcp_server.AddTool("self.dog.forward", "机器人向前移动", PropertyList(), [this](const PropertyList&) -> ReturnValue {
servo_dog_ctrl_send(DOG_STATE_FORWARD, NULL);
return true;
});
// 例2带参数设置灯光 RGB 颜色
mcp_server.AddTool("self.light.set_rgb", "设置RGB颜色", PropertyList({
Property("r", kPropertyTypeInteger, 0, 255),
Property("g", kPropertyTypeInteger, 0, 255),
Property("b", kPropertyTypeInteger, 0, 255)
}), [this](const PropertyList& properties) -> ReturnValue {
int r = properties["r"].value<int>();
int g = properties["g"].value<int>();
int b = properties["b"].value<int>();
led_on_ = true;
SetLedColor(r, g, b);
return true;
});
}
```
## 常见工具调用 JSON-RPC 示例
### 1. 获取工具列表
```json
{
"jsonrpc": "2.0",
"method": "tools/list",
"params": { "cursor": "" },
"id": 1
}
```
### 2. 控制底盘前进
```json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.chassis.go_forward",
"arguments": {}
},
"id": 2
}
```
### 3. 切换灯光模式
```json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.chassis.switch_light_mode",
"arguments": { "light_mode": 3 }
},
"id": 3
}
```
### 4. 摄像头翻转
```json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.camera.set_camera_flipped",
"arguments": {}
},
"id": 4
}
```
## 备注
- 工具名称、参数及返回值请以设备端 `AddTool` 注册为准。
- 推荐所有新项目统一采用 MCP 协议进行物联网控制。
- 详细协议与进阶用法请查阅 [`mcp-protocol.md`](./mcp-protocol.md)。

393
docs/mqtt-udp.md Normal file
View File

@@ -0,0 +1,393 @@
# MQTT + UDP 混合通信协议文档
基于代码实现整理的 MQTT + UDP 混合通信协议文档,概述设备端与服务器之间如何通过 MQTT 进行控制消息传输,通过 UDP 进行音频数据传输的交互方式。
---
## 1. 协议概览
本协议采用混合传输方式:
- **MQTT**用于控制消息、状态同步、JSON 数据交换
- **UDP**:用于实时音频数据传输,支持加密
### 1.1 协议特点
- **双通道设计**:控制与数据分离,确保实时性
- **加密传输**UDP 音频数据使用 AES-CTR 加密
- **序列号保护**:防止数据包重放和乱序
- **自动重连**MQTT 连接断开时自动重连
---
## 2. 总体流程概览
```mermaid
sequenceDiagram
participant Device as ESP32 设备
participant MQTT as MQTT 服务器
participant UDP as UDP 服务器
Note over Device, UDP: 1. 建立 MQTT 连接
Device->>MQTT: MQTT Connect
MQTT->>Device: Connected
Note over Device, UDP: 2. 请求音频通道
Device->>MQTT: Hello Message (type: "hello", transport: "udp")
MQTT->>Device: Hello Response (UDP 连接信息 + 加密密钥)
Note over Device, UDP: 3. 建立 UDP 连接
Device->>UDP: UDP Connect
UDP->>Device: Connected
Note over Device, UDP: 4. 音频数据传输
loop 音频流传输
Device->>UDP: 加密音频数据 (Opus)
UDP->>Device: 加密音频数据 (Opus)
end
Note over Device, UDP: 5. 控制消息交换
par 控制消息
Device->>MQTT: Listen/TTS/MCP 消息
MQTT->>Device: STT/TTS/MCP 响应
end
Note over Device, UDP: 6. 关闭连接
Device->>MQTT: Goodbye Message
Device->>UDP: Disconnect
```
---
## 3. MQTT 控制通道
### 3.1 连接建立
设备通过 MQTT 连接到服务器,连接参数包括:
- **Endpoint**MQTT 服务器地址和端口
- **Client ID**:设备唯一标识符
- **Username/Password**:认证凭据
- **Keep Alive**心跳间隔默认240秒
### 3.2 Hello 消息交换
#### 3.2.1 设备端发送 Hello
```json
{
"type": "hello",
"version": 3,
"transport": "udp",
"features": {
"mcp": true
},
"audio_params": {
"format": "opus",
"sample_rate": 16000,
"channels": 1,
"frame_duration": 60
}
}
```
#### 3.2.2 服务器响应 Hello
```json
{
"type": "hello",
"transport": "udp",
"session_id": "xxx",
"audio_params": {
"format": "opus",
"sample_rate": 24000,
"channels": 1,
"frame_duration": 60
},
"udp": {
"server": "192.168.1.100",
"port": 8888,
"key": "0123456789ABCDEF0123456789ABCDEF",
"nonce": "0123456789ABCDEF0123456789ABCDEF"
}
}
```
**字段说明:**
- `udp.server`UDP 服务器地址
- `udp.port`UDP 服务器端口
- `udp.key`AES 加密密钥(十六进制字符串)
- `udp.nonce`AES 加密随机数(十六进制字符串)
### 3.3 JSON 消息类型
#### 3.3.1 设备端→服务器
1. **Listen 消息**
```json
{
"session_id": "xxx",
"type": "listen",
"state": "start",
"mode": "manual"
}
```
2. **Abort 消息**
```json
{
"session_id": "xxx",
"type": "abort",
"reason": "wake_word_detected"
}
```
3. **MCP 消息**
```json
{
"session_id": "xxx",
"type": "mcp",
"payload": {
"jsonrpc": "2.0",
"id": 1,
"result": {...}
}
}
```
4. **Goodbye 消息**
```json
{
"session_id": "xxx",
"type": "goodbye"
}
```
#### 3.3.2 服务器→设备端
支持的消息类型与 WebSocket 协议一致,包括:
- **STT**:语音识别结果
- **TTS**:语音合成控制
- **LLM**:情感表达控制
- **MCP**:物联网控制
- **System**:系统控制
- **Custom**:自定义消息(可选)
---
## 4. UDP 音频通道
### 4.1 连接建立
设备收到 MQTT Hello 响应后,使用其中的 UDP 连接信息建立音频通道:
1. 解析 UDP 服务器地址和端口
2. 解析加密密钥和随机数
3. 初始化 AES-CTR 加密上下文
4. 建立 UDP 连接
### 4.2 音频数据格式
#### 4.2.1 加密音频包结构
```
|type 1byte|flags 1byte|payload_len 2bytes|ssrc 4bytes|timestamp 4bytes|sequence 4bytes|
|payload payload_len bytes|
```
**字段说明:**
- `type`:数据包类型,固定为 0x01
- `flags`:标志位,当前未使用
- `payload_len`:负载长度(网络字节序)
- `ssrc`:同步源标识符
- `timestamp`:时间戳(网络字节序)
- `sequence`:序列号(网络字节序)
- `payload`:加密的 Opus 音频数据
#### 4.2.2 加密算法
使用 **AES-CTR** 模式加密:
- **密钥**128位由服务器提供
- **随机数**128位由服务器提供
- **计数器**:包含时间戳和序列号信息
### 4.3 序列号管理
- **发送端**`local_sequence_` 单调递增
- **接收端**`remote_sequence_` 验证连续性
- **防重放**:拒绝序列号小于期望值的数据包
- **容错处理**:允许轻微的序列号跳跃,记录警告
### 4.4 错误处理
1. **解密失败**:记录错误,丢弃数据包
2. **序列号异常**:记录警告,但仍处理数据包
3. **数据包格式错误**:记录错误,丢弃数据包
---
## 5. 状态管理
### 5.1 连接状态
```mermaid
stateDiagram
direction TB
[*] --> Disconnected
Disconnected --> MqttConnecting: StartMqttClient()
MqttConnecting --> MqttConnected: MQTT Connected
MqttConnecting --> Disconnected: Connect Failed
MqttConnected --> RequestingChannel: OpenAudioChannel()
RequestingChannel --> ChannelOpened: Hello Exchange Success
RequestingChannel --> MqttConnected: Hello Timeout/Failed
ChannelOpened --> UdpConnected: UDP Connect Success
UdpConnected --> AudioStreaming: Start Audio Transfer
AudioStreaming --> UdpConnected: Stop Audio Transfer
UdpConnected --> ChannelOpened: UDP Disconnect
ChannelOpened --> MqttConnected: CloseAudioChannel()
MqttConnected --> Disconnected: MQTT Disconnect
```
### 5.2 状态检查
设备通过以下条件判断音频通道是否可用:
```cpp
bool IsAudioChannelOpened() const {
return udp_ != nullptr && !error_occurred_ && !IsTimeout();
}
```
---
## 6. 配置参数
### 6.1 MQTT 配置
从设置中读取的配置项:
- `endpoint`MQTT 服务器地址
- `client_id`:客户端标识符
- `username`:用户名
- `password`:密码
- `keepalive`心跳间隔默认240秒
- `publish_topic`:发布主题
### 6.2 音频参数
- **格式**Opus
- **采样率**16000 Hz设备端/ 24000 Hz服务器端
- **声道数**1单声道
- **帧时长**60ms
---
## 7. 错误处理与重连
### 7.1 MQTT 重连机制
- 连接失败时自动重试
- 支持错误上报控制
- 断线时触发清理流程
### 7.2 UDP 连接管理
- 连接失败时不自动重试
- 依赖 MQTT 通道重新协商
- 支持连接状态查询
### 7.3 超时处理
基类 `Protocol` 提供超时检测:
- 默认超时时间120 秒
- 基于最后接收时间计算
- 超时时自动标记为不可用
---
## 8. 安全考虑
### 8.1 传输加密
- **MQTT**:支持 TLS/SSL 加密端口8883
- **UDP**:使用 AES-CTR 加密音频数据
### 8.2 认证机制
- **MQTT**:用户名/密码认证
- **UDP**:通过 MQTT 通道分发密钥
### 8.3 防重放攻击
- 序列号单调递增
- 拒绝过期数据包
- 时间戳验证
---
## 9. 性能优化
### 9.1 并发控制
使用互斥锁保护 UDP 连接:
```cpp
std::lock_guard<std::mutex> lock(channel_mutex_);
```
### 9.2 内存管理
- 动态创建/销毁网络对象
- 智能指针管理音频数据包
- 及时释放加密上下文
### 9.3 网络优化
- UDP 连接复用
- 数据包大小优化
- 序列号连续性检查
---
## 10. 与 WebSocket 协议的比较
| 特性 | MQTT + UDP | WebSocket |
|------|------------|-----------|
| 控制通道 | MQTT | WebSocket |
| 音频通道 | UDP (加密) | WebSocket (二进制) |
| 实时性 | 高 (UDP) | 中等 |
| 可靠性 | 中等 | 高 |
| 复杂度 | 高 | 低 |
| 加密 | AES-CTR | TLS |
| 防火墙友好度 | 低 | 高 |
---
## 11. 部署建议
### 11.1 网络环境
- 确保 UDP 端口可达
- 配置防火墙规则
- 考虑 NAT 穿透
### 11.2 服务器配置
- MQTT Broker 配置
- UDP 服务器部署
- 密钥管理系统
### 11.3 监控指标
- 连接成功率
- 音频传输延迟
- 数据包丢失率
- 解密失败率
---
## 12. 总结
MQTT + UDP 混合协议通过以下设计实现高效的音视频通信:
- **分离式架构**:控制与数据通道分离,各司其职
- **加密保护**AES-CTR 确保音频数据安全传输
- **序列化管理**:防止重放攻击和数据乱序
- **自动恢复**:支持连接断开后的自动重连
- **性能优化**UDP 传输保证音频数据的实时性
该协议适用于对实时性要求较高的语音交互场景,但需要在网络复杂度和传输性能之间做出权衡。

View File

Before

Width:  |  Height:  |  Size: 36 KiB

After

Width:  |  Height:  |  Size: 36 KiB

View File

Before

Width:  |  Height:  |  Size: 94 KiB

After

Width:  |  Height:  |  Size: 94 KiB

View File

Before

Width:  |  Height:  |  Size: 25 KiB

After

Width:  |  Height:  |  Size: 25 KiB

View File

Before

Width:  |  Height:  |  Size: 17 KiB

After

Width:  |  Height:  |  Size: 17 KiB

View File

Before

Width:  |  Height:  |  Size: 20 KiB

After

Width:  |  Height:  |  Size: 20 KiB

View File

Before

Width:  |  Height:  |  Size: 19 KiB

After

Width:  |  Height:  |  Size: 19 KiB

View File

Before

Width:  |  Height:  |  Size: 12 KiB

After

Width:  |  Height:  |  Size: 12 KiB

View File

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 46 KiB

View File

Before

Width:  |  Height:  |  Size: 121 KiB

After

Width:  |  Height:  |  Size: 121 KiB

BIN
docs/v1/electron-bot.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB

BIN
docs/v1/esp-hi.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

View File

Before

Width:  |  Height:  |  Size: 35 KiB

After

Width:  |  Height:  |  Size: 35 KiB

View File

Before

Width:  |  Height:  |  Size: 47 KiB

After

Width:  |  Height:  |  Size: 47 KiB

BIN
docs/v1/otto-robot.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

View File

Before

Width:  |  Height:  |  Size: 57 KiB

After

Width:  |  Height:  |  Size: 57 KiB

View File

Before

Width:  |  Height:  |  Size: 30 KiB

After

Width:  |  Height:  |  Size: 30 KiB

View File

@@ -1,4 +1,6 @@
以下是一份基于代码实现整理的 WebSocket 通信协议文档,概述客户端(设备与服务器之间如何通过 WebSocket 进行交互。该文档仅基于所提供的代码推断,实际部署时可能需要结合服务器端实现进行进一步确认或补充。
以下是一份基于代码实现整理的 WebSocket 通信协议文档,概述设备与服务器之间如何通过 WebSocket 进行交互。
该文档仅基于所提供的代码推断,实际部署时可能需要结合服务器端实现进行进一步确认或补充。
---
@@ -13,16 +15,19 @@
2. **建立 WebSocket 连接**
- 当设备需要开始语音会话时(例如用户唤醒、手动按键触发等),调用 `OpenAudioChannel()`
- 根据编译配置获取 WebSocket URL`CONFIG_WEBSOCKET_URL`
- 根据配置获取 WebSocket URL
- 设置若干请求头(`Authorization`, `Protocol-Version`, `Device-Id`, `Client-Id`
- 调用 `Connect()` 与服务器建立 WebSocket 连接
3. **发送客户端 “hello 消息**
3. **设备端发送 "hello" 消息**
- 连接成功后,设备会发送一条 JSON 消息,示例结构如下:
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true
},
"transport": "websocket",
"audio_params": {
"format": "opus",
@@ -32,22 +37,38 @@
}
}
```
- 其中 `"frame_duration"` 的值对应 `OPUS_FRAME_DURATION_MS`(例如 60ms
- 其中 `features` 字段为可选,内容根据设备编译配置自动生成。例如:`"mcp": true` 表示支持 MCP 协议
- `frame_duration` 的值对应 `OPUS_FRAME_DURATION_MS`(例如 60ms
4. **服务器回复 hello**
4. **服务器回复 "hello"**
- 设备等待服务器返回一条包含 `"type": "hello"` 的 JSON 消息,并检查 `"transport": "websocket"` 是否匹配。
- 服务器可选下发 `session_id` 字段,设备端收到后会自动记录。
- 示例:
```json
{
"type": "hello",
"transport": "websocket",
"session_id": "xxx",
"audio_params": {
"format": "opus",
"sample_rate": 24000,
"channels": 1,
"frame_duration": 60
}
}
```
- 如果匹配,则认为服务器已就绪,标记音频通道打开成功。
- 如果在超时时间(默认 10 秒)内未收到正确回复,认为连接失败并触发网络错误回调。
5. **后续消息交互**
- 设备端和服务器端之间可发送两种主要类型的数据:
1. **二进制音频数据**Opus 编码)
2. **文本 JSON 消息**用于传输聊天状态、TTS/STT 事件、IoT 命令等)
2. **文本 JSON 消息**用于传输聊天状态、TTS/STT 事件、MCP 协议消息等)
- 在代码里,接收回调主要分为:
- `OnData(...)`:
- 当 `binary` 为 `true` 时,认为是音频帧;设备会将其当作 Opus 数据进行解码。
- 当 `binary` 为 `false` 时,认为是 JSON 文本,需要在设备端用 cJSON 进行解析并做相应业务逻辑处理(见下文消息结构)。
- 当 `binary` 为 `false` 时,认为是 JSON 文本,需要在设备端用 cJSON 进行解析并做相应业务逻辑处理(如聊天、TTS、MCP 协议消息等)。
- 当服务器或网络出现断连,回调 `OnDisconnected()` 被触发:
- 设备会调用 `on_audio_channel_closed_()`,并最终回到空闲状态。
@@ -63,27 +84,63 @@
在建立 WebSocket 连接时,代码示例中设置了以下请求头:
- `Authorization`: 用于存放访问令牌,形如 `"Bearer <token>"`
- `Protocol-Version`: 固定示例中为 `"1"`
- `Device-Id`: 设备物理网卡 MAC 地址
- `Client-Id`: 设备 UUID可在应用中唯一标识设备
- `Protocol-Version`: 协议版本号,与 hello 消息体内的 `version` 字段保持一致
- `Device-Id`: 设备物理网卡 MAC 地址
- `Client-Id`: 软件生成的 UUID擦除 NVS 或重新烧录完整固件会重置
这些头会随着 WebSocket 握手一起发送到服务器,服务器可根据需求进行校验、认证等。
---
## 3. JSON 消息结构
## 3. 二进制协议版本
设备支持多种二进制协议版本,通过配置中的 `version` 字段指定:
### 3.1 版本1默认
直接发送 Opus 音频数据无额外元数据。Websocket 协议会区分 text 与 binary。
### 3.2 版本2
使用 `BinaryProtocol2` 结构:
```c
struct BinaryProtocol2 {
uint16_t version; // 协议版本
uint16_t type; // 消息类型 (0: OPUS, 1: JSON)
uint32_t reserved; // 保留字段
uint32_t timestamp; // 时间戳毫秒用于服务器端AEC
uint32_t payload_size; // 负载大小(字节)
uint8_t payload[]; // 负载数据
} __attribute__((packed));
```
### 3.3 版本3
使用 `BinaryProtocol3` 结构:
```c
struct BinaryProtocol3 {
uint8_t type; // 消息类型
uint8_t reserved; // 保留字段
uint16_t payload_size; // 负载大小
uint8_t payload[]; // 负载数据
} __attribute__((packed));
```
---
## 4. JSON 消息结构
WebSocket 文本帧以 JSON 方式传输,以下为常见的 `"type"` 字段及其对应业务逻辑。若消息里包含未列出的字段,可能为可选或特定实现细节。
### 3.1 客户端→服务器
### 4.1 设备端→服务器
1. **Hello**
- 连接成功后,由客户端发送,告知服务器基本参数。
- 连接成功后,由设备端发送,告知服务器基本参数。
- 例:
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true
},
"transport": "websocket",
"audio_params": {
"format": "opus",
@@ -95,7 +152,7 @@ WebSocket 文本帧以 JSON 方式传输,以下为常见的 `"type"` 字段及
```
2. **Listen**
- 表示客户端开始或停止录音监听。
- 表示设备端开始或停止录音监听。
- 常见字段:
- `"session_id"`:会话标识
- `"type": "listen"`
@@ -124,7 +181,8 @@ WebSocket 文本帧以 JSON 方式传输,以下为常见的 `"type"` 字段及
- `reason` 值可为 `"wake_word_detected"` 或其他。
4. **Wake Word Detected**
- 用于客户端向服务器告知检测到唤醒词。
- 用于设备端向服务器告知检测到唤醒词。
- 在发送该消息之前,可提前发送唤醒词的 Opus 音频数据,用于服务器进行声纹检测。
- 例:
```json
{
@@ -135,78 +193,121 @@ WebSocket 文本帧以 JSON 方式传输,以下为常见的 `"type"` 字段及
}
```
5. **IoT**
- 发送当前设备的物联网相关信息:
- **Descriptors**(描述设备功能、属性等)
- **States**(设备状态的实时更新)
- 例:
5. **MCP**
- 推荐用于物联网控制的新一代协议。所有设备能力发现、工具调用等均通过 type: "mcp" 的消息进行payload 内部为标准 JSON-RPC 2.0(详见 [MCP 协议文档](./mcp-protocol.md))。
- **设备端到服务器发送 result 的例子:**
```json
{
"session_id": "xxx",
"type": "iot",
"descriptors": { ... }
}
```
```json
{
"session_id": "xxx",
"type": "iot",
"states": { ... }
"type": "mcp",
"payload": {
"jsonrpc": "2.0",
"id": 1,
"result": {
"content": [
{ "type": "text", "text": "true" }
],
"isError": false
}
}
}
```
---
### 3.2 服务器→客户
### 4.2 服务器→设备
1. **Hello**
- 服务器端返回的握手确认消息。
- 必须包含 `"type": "hello"` 和 `"transport": "websocket"`。
- 可能会带有 `audio_params`,表示服务器期望的音频参数,或与客户端对齐的配置。
- 成功接收后客户端会设置事件标志,表示 WebSocket 通道就绪。
- 可能会带有 `audio_params`,表示服务器期望的音频参数,或与设备端对齐的配置。
- 服务器可选下发 `session_id` 字段,设备端收到后会自动记录。
- 成功接收后设备端会设置事件标志,表示 WebSocket 通道就绪。
2. **STT**
- `{"type": "stt", "text": "..."}`
- `{"session_id": "xxx", "type": "stt", "text": "..."}`
- 表示服务器端识别到了用户语音。(例如语音转文本结果)
- 设备可能将此文本显示到屏幕上,后续再进入回答等流程。
3. **LLM**
- `{"type": "llm", "emotion": "happy", "text": "😀"}`
- `{"session_id": "xxx", "type": "llm", "emotion": "happy", "text": "😀"}`
- 服务器指示设备调整表情动画 / UI 表达。
4. **TTS**
- `{"type": "tts", "state": "start"}`:服务器准备下发 TTS 音频,客户端进入 speaking 播放状态。
- `{"type": "tts", "state": "stop"}`:表示本次 TTS 结束。
- `{"type": "tts", "state": "sentence_start", "text": "..."}`
- `{"session_id": "xxx", "type": "tts", "state": "start"}`:服务器准备下发 TTS 音频,设备端进入 "speaking" 播放状态。
- `{"session_id": "xxx", "type": "tts", "state": "stop"}`:表示本次 TTS 结束。
- `{"session_id": "xxx", "type": "tts", "state": "sentence_start", "text": "..."}`
- 让设备在界面上显示当前要播放或朗读的文本片段(例如用于显示给用户)。
5. **IoT**
- `{"type": "iot", "commands": [ ... ]}`
- 服务器向设备发送物联网的动作指令,设备解析并执行(如打开灯、设置温度等)。
5. **MCP**
- 服务器通过 type: "mcp" 的消息下发物联网相关的控制指令或返回调用结果payload 结构同上。
- **服务器到设备端发送 tools/call 的例子:**
```json
{
"session_id": "xxx",
"type": "mcp",
"payload": {
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.light.set_rgb",
"arguments": { "r": 255, "g": 0, "b": 0 }
},
"id": 1
}
}
```
6. **音频数据:二进制帧**
- 当服务器发送音频二进制帧Opus 编码)时,客户端解码并播放。
- 若客户端正在处于 “listening” (录音)状态,收到的音频帧会被忽略或清空以防冲突。
6. **System**
- 系统控制命令,常用于远程升级更新。
- 例:
```json
{
"session_id": "xxx",
"type": "system",
"command": "reboot"
}
```
- 支持的命令:
- `"reboot"`:重启设备
7. **Custom**(可选)
- 自定义消息,当 `CONFIG_RECEIVE_CUSTOM_MESSAGE` 启用时支持。
- 例:
```json
{
"session_id": "xxx",
"type": "custom",
"payload": {
"message": "自定义内容"
}
}
```
8. **音频数据:二进制帧**
- 当服务器发送音频二进制帧Opus 编码)时,设备端解码并播放。
- 若设备端正在处于 "listening" (录音)状态,收到的音频帧会被忽略或清空以防冲突。
---
## 4. 音频编解码
## 5. 音频编解码
1. **客户端发送录音数据**
1. **设备端发送录音数据**
- 音频输入经过可能的回声消除、降噪或音量增益后,通过 Opus 编码打包为二进制帧发送给服务器。
- 如果客户端每次编码生成的二进制帧大小为 N 字节,则会通过 WebSocket 的 **binary** 消息发送这块数据
- 根据协议版本,可能直接发送 Opus 数据版本1或使用带元数据的二进制协议版本2/3
2. **客户端播放收到的音频**
2. **设备端播放收到的音频**
- 收到服务器的二进制帧时,同样认定是 Opus 数据。
- 设备端会进行解码,然后交由音频输出接口播放。
- 如果服务器的音频采样率与设备不一致,会在解码后再进行重采样。
---
## 5. 常见状态流转
## 6. 常见状态流转
以下简述设备端关键状态流转,与 WebSocket 消息对应:
以下为常见设备端关键状态流转,与 WebSocket 消息对应:
1. **Idle** → **Connecting**
- 用户触发或唤醒后,设备调用 `OpenAudioChannel()` → 建立 WebSocket 连接 → 发送 `"type":"hello"`。
@@ -223,12 +324,52 @@ WebSocket 文本帧以 JSON 方式传输,以下为常见的 `"type"` 字段及
5. **Listening** / **Speaking** → **Idle**(遇到异常或主动中断)
- 调用 `SendAbortSpeaking(...)` 或 `CloseAudioChannel()` → 中断会话 → 关闭 WebSocket → 状态回到 Idle。
### 自动模式状态流转图
```mermaid
stateDiagram
direction TB
[*] --> kDeviceStateUnknown
kDeviceStateUnknown --> kDeviceStateStarting:初始化
kDeviceStateStarting --> kDeviceStateWifiConfiguring:配置WiFi
kDeviceStateStarting --> kDeviceStateActivating:激活设备
kDeviceStateActivating --> kDeviceStateUpgrading:检测到新版本
kDeviceStateActivating --> kDeviceStateIdle:激活完成
kDeviceStateIdle --> kDeviceStateConnecting:开始连接
kDeviceStateConnecting --> kDeviceStateIdle:连接失败
kDeviceStateConnecting --> kDeviceStateListening:连接成功
kDeviceStateListening --> kDeviceStateSpeaking:开始说话
kDeviceStateSpeaking --> kDeviceStateListening:结束说话
kDeviceStateListening --> kDeviceStateIdle:手动终止
kDeviceStateSpeaking --> kDeviceStateIdle:自动终止
```
### 手动模式状态流转图
```mermaid
stateDiagram
direction TB
[*] --> kDeviceStateUnknown
kDeviceStateUnknown --> kDeviceStateStarting:初始化
kDeviceStateStarting --> kDeviceStateWifiConfiguring:配置WiFi
kDeviceStateStarting --> kDeviceStateActivating:激活设备
kDeviceStateActivating --> kDeviceStateUpgrading:检测到新版本
kDeviceStateActivating --> kDeviceStateIdle:激活完成
kDeviceStateIdle --> kDeviceStateConnecting:开始连接
kDeviceStateConnecting --> kDeviceStateIdle:连接失败
kDeviceStateConnecting --> kDeviceStateListening:连接成功
kDeviceStateIdle --> kDeviceStateListening:开始监听
kDeviceStateListening --> kDeviceStateIdle:停止监听
kDeviceStateIdle --> kDeviceStateSpeaking:开始说话
kDeviceStateSpeaking --> kDeviceStateIdle:结束说话
```
---
## 6. 错误处理
## 7. 错误处理
1. **连接失败**
- 如果 `Connect(url)` 返回失败或在等待服务器 hello 消息时超时,触发 `on_network_error_()` 回调。设备会提示无法连接到服务或类似错误信息。
- 如果 `Connect(url)` 返回失败或在等待服务器 "hello" 消息时超时,触发 `on_network_error_()` 回调。设备会提示"无法连接到服务"或类似错误信息。
2. **服务器断开**
- 如果 WebSocket 异常断开,回调 `OnDisconnected()`
@@ -237,35 +378,46 @@ WebSocket 文本帧以 JSON 方式传输,以下为常见的 `"type"` 字段及
---
## 7. 其它注意事项
## 8. 其它注意事项
1. **鉴权**
- 设备通过设置 `Authorization: Bearer <token>` 提供鉴权,服务器端需验证是否有效。
- 如果令牌过期或无效,服务器可拒绝握手或在后续断开。
2. **会话控制**
- 代码中部分消息包含 `session_id`,用于区分独立的对话或操作。服务端可根据需要对不同会话做分离处理WebSocket 协议为空
- 代码中部分消息包含 `session_id`,用于区分独立的对话或操作。服务端可根据需要对不同会话做分离处理。
3. **音频负载**
- 代码里默认使用 Opus 格式,并设置 `sample_rate = 16000`,单声道。帧时长由 `OPUS_FRAME_DURATION_MS` 控制,一般为 60ms。可根据带宽或性能做适当调整。
- 代码里默认使用 Opus 格式,并设置 `sample_rate = 16000`,单声道。帧时长由 `OPUS_FRAME_DURATION_MS` 控制,一般为 60ms。可根据带宽或性能做适当调整。为了获得更好的音乐播放效果,服务器下行音频可能使用 24000 采样率。
4. **IoT 指令**
- `"type":"iot"` 的消息用户端代码对接 `thing_manager` 执行具体命令,因设备定制而不同。服务器端需确保下发格式与客户端保持一致。
4. **协议版本配置**
- 通过设置中的 `version` 字段配置二进制协议版本1、2 或 3
- 版本1直接发送 Opus 数据
- 版本2使用带时间戳的二进制协议适用于服务器端 AEC
- 版本3使用简化的二进制协议
5. **错误或异常 JSON**
- 当 JSON 中缺少必要字段,例如 `{"type": ...}`,客户端会记录错误日志(`ESP_LOGE(TAG, "Missing message type, data: %s", data);`),不会执行任何业务
5. **物联网控制推荐 MCP 协议**
- 设备与服务器之间的物联网能力发现、状态同步、控制指令等,建议全部通过 MCP 协议(type: "mcp")实现。原有的 type: "iot" 方案已废弃
- MCP 协议可在 WebSocket、MQTT 等多种底层协议上传输,具备更好的扩展性和标准化能力。
- 详细用法请参考 [MCP 协议文档](./mcp-protocol.md) 及 [MCP 物联网控制用法](./mcp-usage.md)。
6. **错误或异常 JSON**
- 当 JSON 中缺少必要字段,例如 `{"type": ...}`,设备端会记录错误日志(`ESP_LOGE(TAG, "Missing message type, data: %s", data);`),不会执行任何业务。
---
## 8. 消息示例
## 9. 消息示例
下面给出一个典型的双向消息示例(流程简化示意):
1. **客户端 → 服务器**(握手)
1. **设备端 → 服务器**(握手)
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true
},
"transport": "websocket",
"audio_params": {
"format": "opus",
@@ -276,63 +428,68 @@ WebSocket 文本帧以 JSON 方式传输,以下为常见的 `"type"` 字段及
}
```
2. **服务器 → 客户端**(握手应答)
2. **服务器 → 设备端**(握手应答)
```json
{
"type": "hello",
"transport": "websocket",
"session_id": "xxx",
"audio_params": {
"format": "opus",
"sample_rate": 16000
}
}
```
3. **客户端 → 服务器**(开始监听)
3. **设备端 → 服务器**(开始监听)
```json
{
"session_id": "",
"session_id": "xxx",
"type": "listen",
"state": "start",
"mode": "auto"
}
```
同时客户端开始发送二进制帧Opus 数据)。
同时设备端开始发送二进制帧Opus 数据)。
4. **服务器 → 客户端**ASR 结果)
4. **服务器 → 设备端**ASR 结果)
```json
{
"session_id": "xxx",
"type": "stt",
"text": "用户说的话"
}
```
5. **服务器 → 客户端**TTS开始
5. **服务器 → 设备端**TTS开始
```json
{
"session_id": "xxx",
"type": "tts",
"state": "start"
}
```
接着服务器发送二进制音频帧给客户端播放。
接着服务器发送二进制音频帧给设备端播放。
6. **服务器 → 客户端**TTS结束
6. **服务器 → 设备端**TTS结束
```json
{
"session_id": "xxx",
"type": "tts",
"state": "stop"
}
```
客户端停止播放音频,若无更多指令,则回到空闲状态。
设备端停止播放音频,若无更多指令,则回到空闲状态。
---
## 9. 总结
## 10. 总结
本协议通过在 WebSocket 上层传输 JSON 文本与二进制音频帧完成功能包括音频流上传、TTS 音频播放、语音识别与状态管理、IoT 指令下发等。其核心特征:
本协议通过在 WebSocket 上层传输 JSON 文本与二进制音频帧完成功能包括音频流上传、TTS 音频播放、语音识别与状态管理、MCP 指令下发等。其核心特征:
- **握手阶段**:发送 `"type":"hello"`,等待服务器返回。
- **音频通道**:采用 Opus 编码的二进制帧双向传输语音流。
- **JSON 消息**:使用 `"type"` 为核心字段标识不同业务逻辑,包括 TTS、STT、IoT、WakeWord 等。
- **音频通道**:采用 Opus 编码的二进制帧双向传输语音流,支持多种协议版本
- **JSON 消息**:使用 `"type"` 为核心字段标识不同业务逻辑,包括 TTS、STT、MCP、WakeWord、System、Custom 等。
- **扩展性**:可根据实际需求在 JSON 消息中添加字段,或在 headers 里进行额外鉴权。
服务器与客户端需提前约定各类消息的字段含义、时序逻辑以及错误处理规则,方能保证通信顺畅。上述信息可作为基础文档,便于后续对接、开发或扩展。
服务器与设备端需提前约定各类消息的字段含义、时序逻辑以及错误处理规则,方能保证通信顺畅。上述信息可作为基础文档,便于后续对接、开发或扩展。

View File

@@ -1,8 +1,13 @@
set(SOURCES "audio_codecs/audio_codec.cc"
"audio_codecs/no_audio_codec.cc"
"audio_codecs/box_audio_codec.cc"
"audio_codecs/es8311_audio_codec.cc"
"audio_codecs/es8388_audio_codec.cc"
set(SOURCES "audio/audio_codec.cc"
"audio/audio_service.cc"
"audio/codecs/no_audio_codec.cc"
"audio/codecs/box_audio_codec.cc"
"audio/codecs/es8311_audio_codec.cc"
"audio/codecs/es8374_audio_codec.cc"
"audio/codecs/es8388_audio_codec.cc"
"audio/codecs/es8389_audio_codec.cc"
"audio/codecs/dummy_audio_codec.cc"
"audio/processors/audio_debugger.cc"
"led/single_led.cc"
"led/circular_strip.cc"
"led/gpio_led.cc"
@@ -10,21 +15,18 @@ set(SOURCES "audio_codecs/audio_codec.cc"
"display/lcd_display.cc"
"display/oled_display.cc"
"protocols/protocol.cc"
"iot/thing.cc"
"iot/thing_manager.cc"
"protocols/mqtt_protocol.cc"
"protocols/websocket_protocol.cc"
"mcp_server.cc"
"system_info.cc"
"application.cc"
"ota.cc"
"settings.cc"
"background_task.cc"
"device_state_event.cc"
"main.cc"
)
set(INCLUDE_DIRS "." "display" "audio_codecs" "protocols" "audio_processing")
# 添加 IOT 相关文件
file(GLOB IOT_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/iot/things/*.cc)
list(APPEND SOURCES ${IOT_SOURCES})
set(INCLUDE_DIRS "." "display" "audio" "protocols")
# 添加板级公共文件
file(GLOB BOARD_COMMON_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/boards/common/*.cc)
@@ -39,9 +41,11 @@ elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ML307)
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ESP32)
set(BOARD_TYPE "bread-compact-esp32")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ESP32_LCD)
set(BOARD_TYPE "bread-compact-esp32-lcd")
set(BOARD_TYPE "bread-compact-esp32-lcd")
elseif(CONFIG_BOARD_TYPE_DF_K10)
set(BOARD_TYPE "df-k10")
elseif(CONFIG_BOARD_TYPE_DF_S3_AI_CAM)
set(BOARD_TYPE "df-s3-ai-cam")
elseif(CONFIG_BOARD_TYPE_ESP_BOX_3)
set(BOARD_TYPE "esp-box-3")
elseif(CONFIG_BOARD_TYPE_ESP_BOX)
@@ -74,22 +78,42 @@ elseif(CONFIG_BOARD_TYPE_MAGICLICK_C3_V2)
set(BOARD_TYPE "magiclick-c3-v2")
elseif(CONFIG_BOARD_TYPE_M5STACK_CORE_S3)
set(BOARD_TYPE "m5stack-core-s3")
elseif(CONFIG_BOARD_TYPE_M5STACK_CORE_TAB5)
set(BOARD_TYPE "m5stack-tab5")
elseif(CONFIG_BOARD_TYPE_ATOMS3_ECHO_BASE)
set(BOARD_TYPE "atoms3-echo-base")
elseif(CONFIG_BOARD_TYPE_ATOMS3R_ECHO_BASE)
set(BOARD_TYPE "atoms3r-echo-base")
elseif(CONFIG_BOARD_TYPE_ATOMS3R_CAM_M12_ECHO_BASE)
set(BOARD_TYPE "atoms3r-cam-m12-echo-base")
elseif(CONFIG_BOARD_TYPE_ATOM_ECHOS3R)
set(BOARD_TYPE "atom-echos3r")
elseif(CONFIG_BOARD_TYPE_ATOMMATRIX_ECHO_BASE)
set(BOARD_TYPE "atommatrix-echo-base")
elseif(CONFIG_BOARD_TYPE_XMINI_C3_V3)
set(BOARD_TYPE "xmini-c3-v3")
elseif(CONFIG_BOARD_TYPE_XMINI_C3_4G)
set(BOARD_TYPE "xmini-c3-4g")
elseif(CONFIG_BOARD_TYPE_XMINI_C3)
set(BOARD_TYPE "xmini-c3")
elseif(CONFIG_BOARD_TYPE_ESP32S3_KORVO2_V3)
set(BOARD_TYPE "esp32s3-korvo2-v3")
elseif(CONFIG_BOARD_TYPE_ESP_SPARKBOT)
set(BOARD_TYPE "esp-sparkbot")
elseif(CONFIG_BOARD_TYPE_ESP_SPOT_S3)
set(BOARD_TYPE "esp-spot-s3")
elseif(CONFIG_BOARD_TYPE_ESP_HI)
set(BOARD_TYPE "esp-hi")
elseif(CONFIG_BOARD_TYPE_ECHOEAR)
set(BOARD_TYPE "echoear")
elseif(CONFIG_BOARD_TYPE_ESP32S3_AUDIO_BOARD)
set(BOARD_TYPE "waveshare-s3-audio-board")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_AMOLED_1_8)
set(BOARD_TYPE "esp32-s3-touch-amoled-1.8")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_AMOLED_2_06)
set(BOARD_TYPE "waveshare-s3-touch-amoled-2.06")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_AMOLED_1_75)
set(BOARD_TYPE "waveshare-s3-touch-amoled-1.75")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_1_85C)
set(BOARD_TYPE "esp32-s3-touch-lcd-1.85c")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_1_85)
@@ -98,14 +122,32 @@ elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_1_46)
set(BOARD_TYPE "esp32-s3-touch-lcd-1.46")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_3_5)
set(BOARD_TYPE "esp32-s3-touch-lcd-3.5")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_3_5B)
set(BOARD_TYPE "waveshare-s3-touch-lcd-3.5b")
elseif(CONFIG_BOARD_TYPE_ESP32C6_LCD_1_69)
set(BOARD_TYPE "waveshare-c6-lcd-1.69")
elseif(CONFIG_BOARD_TYPE_ESP32C6_Touch_AMOLED_1_43)
set(BOARD_TYPE "waveshare-c6-touch-amoled-1.43")
elseif(CONFIG_BOARD_TYPE_ESP32P4_NANO)
set(BOARD_TYPE "waveshare-p4-nano")
elseif(CONFIG_BOARD_TYPE_ESP32P4_WIFI6_Touch_LCD_4B)
set(BOARD_TYPE "waveshare-p4-wifi6-touch-lcd-4b")
elseif(CONFIG_BOARD_TYPE_ESP32P4_WIFI6_Touch_LCD_XC)
set(BOARD_TYPE "waveshare-p4-wifi6-touch-lcd-xc")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI_LCD)
set(BOARD_TYPE "bread-compact-wifi-lcd")
elseif(CONFIG_BOARD_TYPE_TUDOUZI)
set(BOARD_TYPE "tudouzi")
elseif(CONFIG_BOARD_TYPE_LILYGO_T_CIRCLE_S3)
set(BOARD_TYPE "lilygo-t-circle-s3")
elseif(CONFIG_BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3)
elseif(CONFIG_BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_0_V1_1)
set(BOARD_TYPE "lilygo-t-cameraplus-s3")
elseif(CONFIG_BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_2)
set(BOARD_TYPE "lilygo-t-cameraplus-s3")
elseif(CONFIG_BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA)
set(BOARD_TYPE "lilygo-t-display-s3-pro-mvsrlora")
elseif(CONFIG_BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA_NO_BATTERY)
set(BOARD_TYPE "lilygo-t-display-s3-pro-mvsrlora")
elseif(CONFIG_BOARD_TYPE_MOVECALL_MOJI_ESP32S3)
set(BOARD_TYPE "movecall-moji-esp32s3")
elseif(CONFIG_BOARD_TYPE_MOVECALL_CUICAN_ESP32S3)
@@ -114,6 +156,16 @@ elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3)
set(BOARD_TYPE "atk-dnesp32s3")
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX)
set(BOARD_TYPE "atk-dnesp32s3-box")
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX0)
set(BOARD_TYPE "atk-dnesp32s3-box0")
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX2_WIFI)
set(BOARD_TYPE "atk-dnesp32s3-box2-wifi")
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX2_4G)
set(BOARD_TYPE "atk-dnesp32s3-box2-4g")
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3M_WIFI)
set(BOARD_TYPE "atk-dnesp32s3m-wifi")
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3M_4G)
set(BOARD_TYPE "atk-dnesp32s3m-4g")
elseif(CONFIG_BOARD_TYPE_DU_CHATX)
set(BOARD_TYPE "du-chatx")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Taiji_Pi)
@@ -132,8 +184,44 @@ elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_1_54TFT_ML307)
set(BOARD_TYPE "xingzhi-cube-1.54tft-ml307")
elseif(CONFIG_BOARD_TYPE_SENSECAP_WATCHER)
set(BOARD_TYPE "sensecap-watcher")
elseif(CONFIG_BOARD_TYPE_DOIT_S3_AIBOX)
set(BOARD_TYPE "doit-s3-aibox")
elseif(CONFIG_BOARD_TYPE_MIXGO_NOVA)
set(BOARD_TYPE "mixgo-nova")
elseif(CONFIG_BOARD_TYPE_GENJUTECH_S3_1_54TFT)
set(BOARD_TYPE "genjutech-s3-1.54tft")
elseif(CONFIG_BOARD_TYPE_ESP32_CGC)
set(BOARD_TYPE "esp32-cgc")
set(BOARD_TYPE "esp32-cgc")
elseif(CONFIG_BOARD_TYPE_ESP32_CGC_144)
set(BOARD_TYPE "esp32-cgc-144")
elseif(CONFIG_BOARD_TYPE_ESP_S3_LCD_EV_Board)
set(BOARD_TYPE "esp-s3-lcd-ev-board")
elseif(CONFIG_BOARD_TYPE_ESP_S3_LCD_EV_Board_2)
set(BOARD_TYPE "esp-s3-lcd-ev-board-2")
elseif(CONFIG_BOARD_TYPE_ZHENGCHEN_1_54TFT_WIFI)
set(BOARD_TYPE "zhengchen-1.54tft-wifi")
elseif(CONFIG_BOARD_TYPE_MINSI_K08_DUAL)
set(BOARD_TYPE "minsi-k08-dual")
elseif(CONFIG_BOARD_TYPE_ZHENGCHEN_1_54TFT_ML307)
set(BOARD_TYPE "zhengchen-1.54tft-ml307")
elseif(CONFIG_BOARD_TYPE_ESP32_S3_1_54_MUMA)
set(BOARD_TYPE "sp-esp32-s3-1.54-muma")
elseif(CONFIG_BOARD_TYPE_ESP32_S3_1_28_BOX)
set(BOARD_TYPE "sp-esp32-s3-1.28-box")
elseif(CONFIG_BOARD_TYPE_OTTO_ROBOT)
set(BOARD_TYPE "otto-robot")
elseif(CONFIG_BOARD_TYPE_ELECTRON_BOT)
set(BOARD_TYPE "electron-bot")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI_CAM)
set(BOARD_TYPE "bread-compact-wifi-s3cam")
elseif(CONFIG_BOARD_TYPE_JIUCHUAN)
set(BOARD_TYPE "jiuchuan-s3")
elseif(CONFIG_BOARD_TYPE_LABPLUS_MPYTHON_V3)
set(BOARD_TYPE "labplus-mpython-v3")
elseif(CONFIG_BOARD_TYPE_LABPLUS_LEDONG_V2)
set(BOARD_TYPE "labplus-ledong-v2")
elseif(CONFIG_BOARD_TYPE_SURFER_C3_1_14TFT)
set(BOARD_TYPE "surfer-c3-1.14tft")
endif()
file(GLOB BOARD_SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.cc
@@ -141,17 +229,17 @@ file(GLOB BOARD_SOURCES
)
list(APPEND SOURCES ${BOARD_SOURCES})
if(CONFIG_CONNECTION_TYPE_MQTT_UDP)
list(APPEND SOURCES "protocols/mqtt_protocol.cc")
elseif(CONFIG_CONNECTION_TYPE_WEBSOCKET)
list(APPEND SOURCES "protocols/websocket_protocol.cc")
endif()
if(CONFIG_USE_AUDIO_PROCESSOR)
list(APPEND SOURCES "audio_processing/audio_processor.cc")
list(APPEND SOURCES "audio/processors/afe_audio_processor.cc")
else()
list(APPEND SOURCES "audio/processors/no_audio_processor.cc")
endif()
if(CONFIG_USE_WAKE_WORD_DETECT)
list(APPEND SOURCES "audio_processing/wake_word_detect.cc")
if(CONFIG_USE_AFE_WAKE_WORD)
list(APPEND SOURCES "audio/wake_words/afe_wake_word.cc")
elseif(CONFIG_USE_ESP_WAKE_WORD)
list(APPEND SOURCES "audio/wake_words/esp_wake_word.cc")
elseif(CONFIG_USE_CUSTOM_WAKE_WORD)
list(APPEND SOURCES "audio/wake_words/custom_wake_word.cc")
endif()
# 根据Kconfig选择语言目录
@@ -163,18 +251,55 @@ elseif(CONFIG_LANGUAGE_EN_US)
set(LANG_DIR "en-US")
elseif(CONFIG_LANGUAGE_JA_JP)
set(LANG_DIR "ja-JP")
elseif(CONFIG_LANGUAGE_KO_KR)
set(LANG_DIR "ko-KR")
elseif(CONFIG_LANGUAGE_VI_VN)
set(LANG_DIR "vi-VN")
elseif(CONFIG_LANGUAGE_TH_TH)
set(LANG_DIR "th-TH")
elseif(CONFIG_LANGUAGE_DE_DE)
set(LANG_DIR "de-DE")
elseif(CONFIG_LANGUAGE_FR_FR)
set(LANG_DIR "fr-FR")
elseif(CONFIG_LANGUAGE_ES_ES)
set(LANG_DIR "es-ES")
elseif(CONFIG_LANGUAGE_IT_IT)
set(LANG_DIR "it-IT")
elseif(CONFIG_LANGUAGE_RU_RU)
set(LANG_DIR "ru-RU")
elseif(CONFIG_LANGUAGE_AR_SA)
set(LANG_DIR "ar-SA")
elseif(CONFIG_LANGUAGE_HI_IN)
set(LANG_DIR "hi-IN")
elseif(CONFIG_LANGUAGE_PT_PT)
set(LANG_DIR "pt-PT")
elseif(CONFIG_LANGUAGE_PL_PL)
set(LANG_DIR "pl-PL")
elseif(CONFIG_LANGUAGE_CS_CZ)
set(LANG_DIR "cs-CZ")
elseif(CONFIG_LANGUAGE_FI_FI)
set(LANG_DIR "fi-FI")
elseif(CONFIG_LANGUAGE_TR_TR)
set(LANG_DIR "tr-TR")
elseif(CONFIG_LANGUAGE_ID_ID)
set(LANG_DIR "id-ID")
elseif(CONFIG_LANGUAGE_UK_UA)
set(LANG_DIR "uk-UA")
elseif(CONFIG_LANGUAGE_RO_RO)
set(LANG_DIR "ro-RO")
endif()
# 定义生成路径
set(LANG_JSON "${CMAKE_CURRENT_SOURCE_DIR}/assets/${LANG_DIR}/language.json")
set(LANG_JSON "${CMAKE_CURRENT_SOURCE_DIR}/assets/locales/${LANG_DIR}/language.json")
set(LANG_HEADER "${CMAKE_CURRENT_SOURCE_DIR}/assets/lang_config.h")
file(GLOB LANG_SOUNDS ${CMAKE_CURRENT_SOURCE_DIR}/assets/${LANG_DIR}/*.p3)
file(GLOB COMMON_SOUNDS ${CMAKE_CURRENT_SOURCE_DIR}/assets/common/*.p3)
file(GLOB LANG_SOUNDS ${CMAKE_CURRENT_SOURCE_DIR}/assets/locales/${LANG_DIR}/*.ogg)
file(GLOB COMMON_SOUNDS ${CMAKE_CURRENT_SOURCE_DIR}/assets/common/*.ogg)
# 如果目标芯片是 ESP32则排除特定文件
if(CONFIG_IDF_TARGET_ESP32)
list(REMOVE_ITEM SOURCES "audio_codecs/box_audio_codec.cc"
"audio_codecs/es8388_audio_codec.cc"
list(REMOVE_ITEM SOURCES "audio/codecs/box_audio_codec.cc"
"audio/codecs/es8388_audio_codec.cc"
"audio/codecs/es8389_audio_codec.cc"
"led/gpio_led.cc"
)
endif()
@@ -198,7 +323,7 @@ target_compile_definitions(${COMPONENT_LIB}
add_custom_command(
OUTPUT ${LANG_HEADER}
COMMAND python ${PROJECT_DIR}/scripts/gen_lang.py
--input "${LANG_JSON}"
--language "${LANG_DIR}"
--output "${LANG_HEADER}"
DEPENDS
${LANG_JSON}
@@ -210,3 +335,67 @@ add_custom_command(
add_custom_target(lang_header ALL
DEPENDS ${LANG_HEADER}
)
if(CONFIG_BOARD_TYPE_ESP_HI)
set(URL "https://github.com/espressif2022/image_player/raw/main/test_apps/test_8bit")
set(SPIFFS_DIR "${CMAKE_BINARY_DIR}/emoji")
file(MAKE_DIRECTORY ${SPIFFS_DIR})
# List all files to download
set(FILES_TO_DOWNLOAD "")
list(APPEND FILES_TO_DOWNLOAD "Anger_enter.aaf" "Anger_loop.aaf" "Anger_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "happy_enter.aaf" "happy_loop.aaf" "happ_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "sad_enter.aaf" "sad_loop.aaf" "sad_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "scorn_enter.aaf" "scorn_loop.aaf" "scorn_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "left_enter.aaf" "left_loop.aaf" "left_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "right_enter.aaf" "right_loop.aaf" "right_return.aaf")
list(APPEND FILES_TO_DOWNLOAD "asking.aaf" "blink_once.aaf" "blink_quick.aaf")
list(APPEND FILES_TO_DOWNLOAD "connecting.aaf" "panic_enter.aaf" "panic_loop.aaf")
list(APPEND FILES_TO_DOWNLOAD "panic_return.aaf" "wake.aaf")
foreach(FILENAME IN LISTS FILES_TO_DOWNLOAD)
set(REMOTE_FILE "${URL}/${FILENAME}")
set(LOCAL_FILE "${SPIFFS_DIR}/${FILENAME}")
# 检查本地文件是否存在
if(EXISTS ${LOCAL_FILE})
message(STATUS "File ${FILENAME} already exists, skipping download")
else()
message(STATUS "Downloading ${FILENAME}")
file(DOWNLOAD ${REMOTE_FILE} ${LOCAL_FILE}
STATUS DOWNLOAD_STATUS)
list(GET DOWNLOAD_STATUS 0 STATUS_CODE)
if(NOT STATUS_CODE EQUAL 0)
message(FATAL_ERROR "Failed to download ${FILENAME} from ${URL}")
endif()
endif()
endforeach()
spiffs_create_partition_assets(
assets_A
${SPIFFS_DIR}
FLASH_IN_PROJECT
MMAP_FILE_SUPPORT_FORMAT ".aaf"
)
endif()
if(CONFIG_BOARD_TYPE_ECHOEAR)
idf_build_get_property(build_components BUILD_COMPONENTS)
foreach(COMPONENT ${build_components})
if(COMPONENT MATCHES "esp_emote_gfx" OR COMPONENT MATCHES "espressif2022__esp_emote_gfx")
set(EMOTE_GFX_COMPONENT ${COMPONENT})
idf_component_get_property(EMOTE_GFX_COMPONENT_PATH ${EMOTE_GFX_COMPONENT} COMPONENT_DIR)
set(SPIFFS_DIR "${EMOTE_GFX_COMPONENT_PATH}/emoji_normal")
break()
endif()
endforeach()
spiffs_create_partition_assets(
assets_A
${SPIFFS_DIR}
FLASH_IN_PROJECT
MMAP_FILE_SUPPORT_FORMAT ".aaf, ttf, bin"
IMPORT_INC_PATH ${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}
)
endif()

View File

@@ -1,14 +1,14 @@
menu "Xiaozhi Assistant"
config OTA_VERSION_URL
string "OTA Version URL"
config OTA_URL
string "Default OTA URL"
default "https://api.tenclass.net/xiaozhi/ota/"
help
The application will access this URL to check for updates.
The application will access this URL to check for new firmwares and server address.
choice
prompt "语言选择"
prompt "Default Language"
default LANGUAGE_ZH_CN
help
Select device display language
@@ -21,34 +21,44 @@ choice
bool "English"
config LANGUAGE_JA_JP
bool "Japanese"
config LANGUAGE_KO_KR
bool "Korean"
config LANGUAGE_VI_VN
bool "Vietnamese"
config LANGUAGE_TH_TH
bool "Thai"
config LANGUAGE_DE_DE
bool "German"
config LANGUAGE_FR_FR
bool "French"
config LANGUAGE_ES_ES
bool "Spanish"
config LANGUAGE_IT_IT
bool "Italian"
config LANGUAGE_RU_RU
bool "Russian"
config LANGUAGE_AR_SA
bool "Arabic"
config LANGUAGE_HI_IN
bool "Hindi"
config LANGUAGE_PT_PT
bool "Portuguese"
config LANGUAGE_PL_PL
bool "Polish"
config LANGUAGE_CS_CZ
bool "Czech"
config LANGUAGE_FI_FI
bool "Finnish"
config LANGUAGE_TR_TR
bool "Turkish"
config LANGUAGE_ID_ID
bool "Indonesian"
config LANGUAGE_UK_UA
bool "Ukrainian"
config LANGUAGE_RO_RO
bool "Romanian"
endchoice
choice CONNECTION_TYPE
prompt "Connection Type"
default CONNECTION_TYPE_MQTT_UDP
help
网络数据传输协议
config CONNECTION_TYPE_MQTT_UDP
bool "MQTT + UDP"
config CONNECTION_TYPE_WEBSOCKET
bool "Websocket"
endchoice
config WEBSOCKET_URL
depends on CONNECTION_TYPE_WEBSOCKET
string "Websocket URL"
default "wss://api.tenclass.net/xiaozhi/v1/"
help
Communication with the server through websocket after wake up.
config WEBSOCKET_ACCESS_TOKEN
depends on CONNECTION_TYPE_WEBSOCKET
string "Websocket Access Token"
default "test-token"
help
Access token for websocket communication.
choice BOARD_TYPE
prompt "Board Type"
default BOARD_TYPE_BREAD_COMPACT_WIFI
@@ -56,106 +66,301 @@ choice BOARD_TYPE
Board type. 开发板类型
config BOARD_TYPE_BREAD_COMPACT_WIFI
bool "面包板新版接线WiFi"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_WIFI_LCD
bool "面包板新版接线WiFi+ LCD"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_WIFI_CAM
bool "面包板新版接线WiFi+ LCD + Camera"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_ML307
bool "面包板新版接线ML307 AT"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_BREAD_COMPACT_ESP32
bool "面包板WiFi ESP32 DevKit"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_BREAD_COMPACT_ESP32_LCD
bool "面包板WiFi+ LCD ESP32 DevKit"
config BOARD_TYPE_ESP32_CGC
bool "ESP32 CGC"
config BOARD_TYPE_ESP_BOX_3
bool "ESP BOX 3"
config BOARD_TYPE_ESP_BOX
bool "ESP BOX"
config BOARD_TYPE_ESP_BOX_LITE
bool "ESP BOX Lite"
config BOARD_TYPE_KEVIN_BOX_1
bool "Kevin Box 1"
config BOARD_TYPE_KEVIN_BOX_2
bool "Kevin Box 2"
config BOARD_TYPE_KEVIN_C3
bool "Kevin C3"
config BOARD_TYPE_KEVIN_SP_V3_DEV
bool "Kevin SP V3开发板"
config BOARD_TYPE_KEVIN_SP_V4_DEV
bool "Kevin SP V4开发板"
config BOARD_TYPE_KEVIN_YUYING_313LCD
bool "鱼鹰科技3.13LCD开发板"
config BOARD_TYPE_LICHUANG_DEV
bool "立创·实战派ESP32-S3开发板"
config BOARD_TYPE_LICHUANG_C3_DEV
bool "立创·实战派ESP32-C3开发板"
config BOARD_TYPE_DF_K10
bool "DFRobot 行空板 k10"
config BOARD_TYPE_MAGICLICK_2P4
bool "神奇按钮 Magiclick_2.4"
config BOARD_TYPE_MAGICLICK_2P5
bool "神奇按钮 Magiclick_2.5"
config BOARD_TYPE_MAGICLICK_C3
bool "神奇按钮 Magiclick_C3"
config BOARD_TYPE_MAGICLICK_C3_V2
bool "神奇按钮 Magiclick_C3_v2"
config BOARD_TYPE_M5STACK_CORE_S3
bool "M5Stack CoreS3"
config BOARD_TYPE_ATOMS3_ECHO_BASE
bool "AtomS3 + Echo Base"
config BOARD_TYPE_ATOMS3R_ECHO_BASE
bool "AtomS3R + Echo Base"
config BOARD_TYPE_ATOMS3R_CAM_M12_ECHO_BASE
bool "AtomS3R CAM/M12 + Echo Base"
config BOARD_TYPE_ATOMMATRIX_ECHO_BASE
bool "AtomMatrix + Echo Base"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_XMINI_C3_V3
bool "虾哥 Mini C3 V3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_XMINI_C3_4G
bool "虾哥 Mini C3 4G"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_XMINI_C3
bool "虾哥 Mini C3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_ESP32S3_KORVO2_V3
bool "ESP32S3_KORVO2_V3开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_SPARKBOT
bool "ESP-SparkBot开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_SPOT_S3
bool "ESP-Spot-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_HI
bool "ESP-HI"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_ECHOEAR
bool "EchoEar"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_BOX_3
bool "ESP BOX 3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_BOX
bool "ESP BOX"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_BOX_LITE
bool "ESP BOX Lite"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_KEVIN_BOX_1
bool "Kevin Box 1"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_KEVIN_BOX_2
bool "Kevin Box 2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_KEVIN_C3
bool "Kevin C3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_KEVIN_SP_V3_DEV
bool "Kevin SP V3开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_KEVIN_SP_V4_DEV
bool "Kevin SP V4开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_CGC
bool "ESP32 CGC"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_ESP32_CGC_144
bool "ESP32 CGC 144"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_KEVIN_YUYING_313LCD
bool "鱼鹰科技3.13LCD开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LICHUANG_DEV
bool "立创·实战派ESP32-S3开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LICHUANG_C3_DEV
bool "立创·实战派ESP32-C3开发板"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_DF_K10
bool "DFRobot 行空板 k10"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DF_S3_AI_CAM
bool "DFRobot ESP32-S3 AI智能摄像头模块"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MAGICLICK_2P4
bool "神奇按钮 Magiclick_2.4"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MAGICLICK_2P5
bool "神奇按钮 Magiclick_2.5"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MAGICLICK_C3
bool "神奇按钮 Magiclick_C3"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_MAGICLICK_C3_V2
bool "神奇按钮 Magiclick_C3_v2"
depends on IDF_TARGET_ESP32C3
config BOARD_TYPE_M5STACK_CORE_S3
bool "M5Stack CoreS3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_CORE_TAB5
bool "M5Stack Tab5"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_ATOMS3_ECHO_BASE
bool "AtomS3 + Echo Base"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATOMS3R_ECHO_BASE
bool "AtomS3R + Echo Base"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATOMS3R_CAM_M12_ECHO_BASE
bool "AtomS3R CAM/M12 + Echo Base"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATOM_ECHOS3R
bool "AtomEchoS3R"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATOMMATRIX_ECHO_BASE
bool "AtomMatrix + Echo Base"
depends on IDF_TARGET_ESP32
config BOARD_TYPE_ESP32S3_AUDIO_BOARD
bool "Waveshare ESP32-S3-Audio-Board"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32S3_Touch_AMOLED_1_8
bool "Waveshare ESP32-S3-Touch-AMOLED-1.8"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32S3_Touch_AMOLED_2_06
bool "Waveshare ESP32-S3-Touch-AMOLED-2.06"
config BOARD_TYPE_ESP32S3_Touch_AMOLED_1_75
bool "Waveshare ESP32-S3-Touch-AMOLED-1.75"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32S3_Touch_LCD_1_85C
bool "Waveshare ESP32-S3-Touch-LCD-1.85C"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32S3_Touch_LCD_1_85
bool "Waveshare ESP32-S3-Touch-LCD-1.85"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32S3_Touch_LCD_1_46
bool "Waveshare ESP32-S3-Touch-LCD-1.46"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32C6_LCD_1_69
bool "Waveshare ESP32-C6-LCD-1.69"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_ESP32C6_Touch_AMOLED_1_43
bool "Waveshare ESP32-C6-Touch-AMOLOED-1.43"
depends on IDF_TARGET_ESP32C6
config BOARD_TYPE_ESP32S3_Touch_LCD_3_5
bool "Waveshare ESP32-S3-Touch-LCD-3.5"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32S3_Touch_LCD_3_5B
bool "Waveshare ESP32-S3-Touch-LCD-3.5B"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32P4_NANO
bool "Waveshare ESP32-P4-NANO"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_ESP32P4_WIFI6_Touch_LCD_4B
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-4B"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_ESP32P4_WIFI6_Touch_LCD_XC
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C or ESP32-P4-WIFI6-Touch-LCD-4C"
depends on IDF_TARGET_ESP32P4
config BOARD_TYPE_TUDOUZI
bool "土豆子"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_CIRCLE_S3
bool "LILYGO T-Circle-S3"
config BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3
bool "LILYGO T-CameraPlus-S3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_0_V1_1
bool "LILYGO T-CameraPlus-S3_V1_0_V1_1"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3_V1_2
bool "LILYGO T-CameraPlus-S3_V1_2"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA
bool "LILYGO T-Display-S3-Pro-MVSRLora"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LILYGO_T_DISPLAY_S3_PRO_MVSRLORA_NO_BATTERY
bool "LILYGO T-Display-S3-Pro-MVSRLora_No_Battery"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MOVECALL_MOJI_ESP32S3
bool "Movecall Moji 小智AI衍生版"
bool "Movecall Moji 小智AI衍生版"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MOVECALL_CUICAN_ESP32S3
bool "Movecall CuiCan 璀璨·AI吊坠"
bool "Movecall CuiCan 璀璨·AI吊坠"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3
bool "正点原子DNESP32S3开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX
bool "正点原子DNESP32S3-BOX"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX0
bool "正点原子DNESP32S3-BOX0"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX2_WIFI
bool "正点原子DNESP32S3-BOX2-WIFI"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3_BOX2_4G
bool "正点原子DNESP32S3-BOX2-4G"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3M_WIFI
bool "正点原子DNESP32S3M-WIFI"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ATK_DNESP32S3M_4G
bool "正点原子DNESP32S3M-4G"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DU_CHATX
bool "嘟嘟开发板CHATX(wifi)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32S3_Taiji_Pi
bool "太极小派esp32s3"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_Cube_0_85TFT_WIFI
bool "无名科技星智0.85(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_Cube_0_85TFT_ML307
bool "无名科技星智0.85(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_Cube_0_96OLED_WIFI
bool "无名科技星智0.96(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_Cube_0_96OLED_ML307
bool "无名科技星智0.96(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_Cube_1_54TFT_WIFI
bool "无名科技星智1.54(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_XINGZHI_Cube_1_54TFT_ML307
bool "无名科技星智1.54(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SENSECAP_WATCHER
bool "SenseCAP Watcher"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_DOIT_S3_AIBOX
bool "四博智联AI陪伴盒子"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MIXGO_NOVA
bool "元控·青春"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_GENJUTECH_S3_1_54TFT
bool "亘具科技1.54(s3)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_S3_LCD_EV_Board
bool "乐鑫ESP S3 LCD EV Board开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP_S3_LCD_EV_Board_2
bool "乐鑫ESP S3 LCD EV Board 2开发板"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ZHENGCHEN_1_54TFT_WIFI
bool "征辰科技1.54(WIFI)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ZHENGCHEN_1_54TFT_ML307
bool "征辰科技1.54(ML307)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_MINSI_K08_DUAL
bool "敏思科技K08(DUAL)"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_S3_1_54_MUMA
bool "Spotpear ESP32-S3-1.54-MUMA"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_ESP32_S3_1_28_BOX
bool "Spotpear ESP32-S3-1.28-BOX"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_OTTO_ROBOT
bool "ottoRobot"
depends on IDF_TARGET_ESP32S3
select LV_USE_GIF
select LV_GIF_CACHE_DECODE_DATA
config BOARD_TYPE_ELECTRON_BOT
bool "electronBot"
depends on IDF_TARGET_ESP32S3
select LV_USE_GIF
select LV_GIF_CACHE_DECODE_DATA
config BOARD_TYPE_JIUCHUAN
bool "九川智能"
config BOARD_TYPE_LABPLUS_MPYTHON_V3
bool "labplus mpython_v3 board"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_LABPLUS_LEDONG_V2
bool "labplus ledong_v2 board"
depends on IDF_TARGET_ESP32S3
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_SURFER_C3_1_14TFT
bool "Surfer-C3-1-14TFT"
depends on IDF_TARGET_ESP32C3
endchoice
choice ESP_S3_LCD_EV_Board_Version_TYPE
depends on BOARD_TYPE_ESP_S3_LCD_EV_Board
prompt "EV_BOARD Type"
default ESP_S3_LCD_EV_Board_1p4
help
开发板硬件版本型号选择
config ESP_S3_LCD_EV_Board_1p4
bool "乐鑫ESP32_S3_LCD_EV_Board-MB_V1.4"
config ESP_S3_LCD_EV_Board_1p5
bool "乐鑫ESP32_S3_LCD_EV_Board-MB_V1.5"
endchoice
choice DISPLAY_OLED_TYPE
@@ -173,7 +378,7 @@ choice DISPLAY_OLED_TYPE
endchoice
choice DISPLAY_LCD_TYPE
depends on BOARD_TYPE_BREAD_COMPACT_WIFI_LCD || BOARD_TYPE_BREAD_COMPACT_ESP32_LCD || BOARD_TYPE_ESP32_CGC
depends on BOARD_TYPE_BREAD_COMPACT_WIFI_LCD || BOARD_TYPE_BREAD_COMPACT_ESP32_LCD || BOARD_TYPE_ESP32_CGC || BOARD_TYPE_ESP32P4_NANO || BOARD_TYPE_ESP32P4_WIFI6_Touch_LCD_XC || BOARD_TYPE_BREAD_COMPACT_WIFI_CAM
prompt "LCD Type"
default LCD_ST7789_240X320
help
@@ -201,13 +406,21 @@ choice DISPLAY_LCD_TYPE
config LCD_ST7796_320X480
bool "ST7796, 分辨率320*480 IPS"
config LCD_ST7796_320X480_NO_IPS
bool "ST7796, 分辨率320*480, 非IPS"
bool "ST7796, 分辨率320*480, 非IPS"
config LCD_ILI9341_240X320
bool "ILI9341, 分辨率240*320"
config LCD_ILI9341_240X320_NO_IPS
bool "ILI9341, 分辨率240*320, 非IPS"
config LCD_GC9A01_240X240
bool "GC9A01, 分辨率240*240, 圆屏"
config LCD_TYPE_800_1280_10_1_INCH
bool "Waveshare 101M-8001280-IPS-CT-K Display"
config LCD_TYPE_800_1280_10_1_INCH_A
bool "Waveshare 10.1-DSI-TOUCH-A Display"
config LCD_TYPE_800_800_3_4_INCH
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C with 800*800 3.4inch round display"
config LCD_TYPE_720_720_4_INCH
bool "Waveshare ESP32-P4-WIFI6-Touch-LCD-4C with 720*720 4inch round display"
config LCD_CUSTOM
bool "自定义屏幕参数"
endchoice
@@ -215,40 +428,135 @@ endchoice
choice DISPLAY_ESP32S3_KORVO2_V3
depends on BOARD_TYPE_ESP32S3_KORVO2_V3
prompt "ESP32S3_KORVO2_V3 LCD Type"
default LCD_ST7789
default ESP32S3_KORVO2_V3_LCD_ST7789
help
屏幕类型选择
config LCD_ST7789
config ESP32S3_KORVO2_V3_LCD_ST7789
bool "ST7789, 分辨率240*280"
config LCD_ILI9341
config ESP32S3_KORVO2_V3_LCD_ILI9341
bool "ILI9341, 分辨率240*320"
endchoice
choice DISPLAY_ESP32S3_AUDIO_BOARD
depends on BOARD_TYPE_ESP32S3_AUDIO_BOARD
prompt "ESP32S3_AUDIO_BOARD LCD Type"
default AUDIO_BOARD_LCD_JD9853
help
屏幕类型选择
config AUDIO_BOARD_LCD_JD9853
bool "JD9853, 分辨率320*172"
config AUDIO_BOARD_LCD_ST7789
bool "ST7789, 分辨率240*320"
endchoice
config USE_WECHAT_MESSAGE_STYLE
bool "使用微信聊天界面风格"
bool "Enable WeChat Message Style"
default n
help
使用微信聊天界面风格
config USE_WAKE_WORD_DETECT
bool "启用唤醒词检测"
default y
depends on IDF_TARGET_ESP32S3 && SPIRAM
config USE_ESP_WAKE_WORD
bool "Enable Wake Word Detection (without AFE)"
default n
depends on IDF_TARGET_ESP32C3 || IDF_TARGET_ESP32C5 || IDF_TARGET_ESP32C6 || (IDF_TARGET_ESP32 && SPIRAM)
help
需要 ESP32 S3 与 AFE 支持
支持 ESP32 C3、ESP32 C5 与 ESP32 C6增加ESP32支持需要开启PSRAM
config USE_AFE_WAKE_WORD
bool "Enable Wake Word Detection (AFE)"
default y
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM
help
需要 ESP32 S3 与 PSRAM 支持
config USE_CUSTOM_WAKE_WORD
bool "Enable Custom Wake Word Detection"
default n
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM && (!USE_AFE_WAKE_WORD)
help
需要 ESP32 S3 与 PSRAM 支持
config CUSTOM_WAKE_WORD
string "Custom Wake Word"
default "xiao tu dou"
depends on USE_CUSTOM_WAKE_WORD
help
自定义唤醒词,中文用拼音表示,每个字之间用空格隔开
config CUSTOM_WAKE_WORD_DISPLAY
string "Custom Wake Word Display"
default "小土豆"
depends on USE_CUSTOM_WAKE_WORD
help
唤醒后发送给服务器的问候语
config CUSTOM_WAKE_WORD_THRESHOLD
int "Custom Wake Word Threshold (%)"
default 20
range 1 99
depends on USE_CUSTOM_WAKE_WORD
help
自定义唤醒词阈值范围1-99越小越敏感默认10
config USE_AUDIO_PROCESSOR
bool "启用音频降噪、增益处理"
bool "Enable Audio Noise Reduction"
default y
depends on IDF_TARGET_ESP32S3 && SPIRAM
depends on (IDF_TARGET_ESP32S3 || IDF_TARGET_ESP32P4) && SPIRAM
help
需要 ESP32 S3 与 AFE 支持
需要 ESP32 S3 与 PSRAM 支持
config USE_REALTIME_CHAT
bool "启用可语音打断的实时对话模式(需要 AEC 支持)"
config USE_DEVICE_AEC
bool "Enable Device-Side AEC"
default n
depends on USE_AUDIO_PROCESSOR && (BOARD_TYPE_ESP_BOX_3 || BOARD_TYPE_ESP_BOX || BOARD_TYPE_LICHUANG_DEV)
depends on USE_AUDIO_PROCESSOR && (BOARD_TYPE_ESP_BOX_3 || BOARD_TYPE_ESP_BOX || BOARD_TYPE_ESP_BOX_LITE || BOARD_TYPE_LICHUANG_DEV \
|| BOARD_TYPE_ESP32S3_KORVO2_V3 || BOARD_TYPE_ESP32S3_Touch_AMOLED_1_75 || BOARD_TYPE_ESP32S3_Touch_AMOLED_2_06 \
|| BOARD_TYPE_ESP32P4_WIFI6_Touch_LCD_4B || BOARD_TYPE_ESP32P4_WIFI6_Touch_LCD_XC || BOARD_TYPE_ESP_S3_LCD_EV_Board_2 \
|| BOARD_TYPE_ECHOEAR)
help
需要 ESP32 S3 与 AEC 开启,因为性能不够,不建议和微信聊天界面风格同时开启
因为性能不够,不建议和微信聊天界面风格同时开启
config USE_SERVER_AEC
bool "Enable Server-Side AEC (Unstable)"
default n
depends on USE_AUDIO_PROCESSOR
help
启用服务器端 AEC需要服务器支持
config USE_AUDIO_DEBUGGER
bool "Enable Audio Debugger"
default n
help
启用音频调试功能通过UDP发送音频数据
config USE_ACOUSTIC_WIFI_PROVISIONING
bool "Enable Acoustic WiFi Provisioning"
default n
help
启用声波配网功能,使用音频信号传输 WiFi 配置数据
config AUDIO_DEBUG_UDP_SERVER
string "Audio Debug UDP Server Address"
default "192.168.2.100:8000"
depends on USE_AUDIO_DEBUGGER
help
UDP服务器地址格式: IP:PORT用于接收音频调试数据
config RECEIVE_CUSTOM_MESSAGE
bool "Enable Custom Message Reception"
default n
help
启用接收自定义消息功能,允许设备接收来自服务器的自定义消息(最好通过 MQTT 协议)
choice I2S_TYPE_TAIJIPI_S3
depends on BOARD_TYPE_ESP32S3_Taiji_Pi
prompt "taiji-pi-S3 I2S Type"
default TAIJIPAI_I2S_TYPE_STD
help
I2S 类型选择
config TAIJIPAI_I2S_TYPE_STD
bool "I2S Type STD"
config TAIJIPAI_I2S_TYPE_PDM
bool "I2S Type PDM"
endchoice
endmenu

File diff suppressed because it is too large Load Diff

View File

@@ -8,42 +8,30 @@
#include <string>
#include <mutex>
#include <list>
#include <opus_encoder.h>
#include <opus_decoder.h>
#include <opus_resampler.h>
#include <deque>
#include <memory>
#include "protocol.h"
#include "ota.h"
#include "background_task.h"
#include "audio_service.h"
#include "device_state_event.h"
#if CONFIG_USE_WAKE_WORD_DETECT
#include "wake_word_detect.h"
#endif
#if CONFIG_USE_AUDIO_PROCESSOR
#include "audio_processor.h"
#endif
#define SCHEDULE_EVENT (1 << 0)
#define AUDIO_INPUT_READY_EVENT (1 << 1)
#define AUDIO_OUTPUT_READY_EVENT (1 << 2)
#define MAIN_EVENT_SCHEDULE (1 << 0)
#define MAIN_EVENT_SEND_AUDIO (1 << 1)
#define MAIN_EVENT_WAKE_WORD_DETECTED (1 << 2)
#define MAIN_EVENT_VAD_CHANGE (1 << 3)
#define MAIN_EVENT_ERROR (1 << 4)
#define MAIN_EVENT_CHECK_NEW_VERSION_DONE (1 << 5)
#define MAIN_EVENT_CLOCK_TICK (1 << 6)
enum DeviceState {
kDeviceStateUnknown,
kDeviceStateStarting,
kDeviceStateWifiConfiguring,
kDeviceStateIdle,
kDeviceStateConnecting,
kDeviceStateListening,
kDeviceStateSpeaking,
kDeviceStateUpgrading,
kDeviceStateActivating,
kDeviceStateFatalError
enum AecMode {
kAecOff,
kAecOnDeviceSide,
kAecOnServerSide,
};
#define OPUS_FRAME_DURATION_MS 60
class Application {
public:
static Application& GetInstance() {
@@ -55,8 +43,9 @@ public:
Application& operator=(const Application&) = delete;
void Start();
void MainEventLoop();
DeviceState GetDeviceState() const { return device_state_; }
bool IsVoiceDetected() const { return voice_detected_; }
bool IsVoiceDetected() const { return audio_service_.IsVoiceDetected(); }
void Schedule(std::function<void()> callback);
void SetDeviceState(DeviceState state);
void Alert(const char* status, const char* message, const char* emotion = "", const std::string_view& sound = "");
@@ -65,65 +54,55 @@ public:
void ToggleChatState();
void StartListening();
void StopListening();
void UpdateIotStates();
void Reboot();
void WakeWordInvoke(const std::string& wake_word);
void PlaySound(const std::string_view& sound);
bool CanEnterSleepMode();
void SendMcpMessage(const std::string& payload);
void SetAecMode(AecMode mode);
AecMode GetAecMode() const { return aec_mode_; }
void PlaySound(const std::string_view& sound);
AudioService& GetAudioService() { return audio_service_; }
private:
Application();
~Application();
#if CONFIG_USE_WAKE_WORD_DETECT
WakeWordDetect wake_word_detect_;
#endif
#if CONFIG_USE_AUDIO_PROCESSOR
AudioProcessor audio_processor_;
#endif
Ota ota_;
std::mutex mutex_;
std::list<std::function<void()>> main_tasks_;
std::deque<std::function<void()>> main_tasks_;
std::unique_ptr<Protocol> protocol_;
EventGroupHandle_t event_group_ = nullptr;
esp_timer_handle_t clock_timer_handle_ = nullptr;
volatile DeviceState device_state_ = kDeviceStateUnknown;
ListeningMode listening_mode_ = kListeningModeAutoStop;
#if CONFIG_USE_REALTIME_CHAT
bool realtime_chat_enabled_ = true;
#else
bool realtime_chat_enabled_ = false;
#endif
AecMode aec_mode_ = kAecOff;
std::string last_error_message_;
AudioService audio_service_;
bool has_server_time_ = false;
bool aborted_ = false;
bool voice_detected_ = false;
int clock_ticks_ = 0;
TaskHandle_t main_loop_task_handle_ = nullptr;
TaskHandle_t check_new_version_task_handle_ = nullptr;
TaskHandle_t main_event_loop_task_handle_ = nullptr;
// Audio encode / decode
TaskHandle_t audio_loop_task_handle_ = nullptr;
BackgroundTask* background_task_ = nullptr;
std::chrono::steady_clock::time_point last_output_time_;
std::list<std::vector<uint8_t>> audio_decode_queue_;
std::unique_ptr<OpusEncoderWrapper> opus_encoder_;
std::unique_ptr<OpusDecoderWrapper> opus_decoder_;
OpusResampler input_resampler_;
OpusResampler reference_resampler_;
OpusResampler output_resampler_;
void MainLoop();
void OnAudioInput();
void OnAudioOutput();
void ReadAudio(std::vector<int16_t>& data, int sample_rate, int samples);
void ResetDecoder();
void SetDecodeSampleRate(int sample_rate, int frame_duration);
void CheckNewVersion();
void ShowActivationCode();
void OnClockTimer();
void OnWakeWordDetected();
void CheckNewVersion(Ota& ota);
void ShowActivationCode(const std::string& code, const std::string& message);
void SetListeningMode(ListeningMode mode);
void AudioLoop();
};
class TaskPriorityReset {
public:
TaskPriorityReset(BaseType_t priority) {
original_priority_ = uxTaskPriorityGet(NULL);
vTaskPrioritySet(NULL, priority);
}
~TaskPriorityReset() {
vTaskPrioritySet(NULL, original_priority_);
}
private:
BaseType_t original_priority_;
};
#endif // _APPLICATION_H_

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

View File

@@ -0,0 +1,58 @@
{
"language": {
"type" :"ar-SA"
},
"strings": {
"WARNING":"تحذير",
"INFO":"معلومات",
"ERROR":"خطأ",
"VERSION": "الإصدار ",
"LOADING_PROTOCOL":"الاتصال بالخادم...",
"INITIALIZING":"التهيئة...",
"PIN_ERROR":"يرجى إدخال بطاقة SIM",
"REG_ERROR":"لا يمكن الوصول إلى الشبكة، يرجى التحقق من حالة بطاقة البيانات",
"DETECTING_MODULE":"اكتشاف الوحدة...",
"REGISTERING_NETWORK":"انتظار الشبكة...",
"CHECKING_NEW_VERSION":"فحص الإصدار الجديد...",
"CHECK_NEW_VERSION_FAILED":"فشل فحص الإصدار الجديد، سيتم المحاولة خلال %d ثانية: %s",
"SWITCH_TO_WIFI_NETWORK":"التبديل إلى Wi-Fi...",
"SWITCH_TO_4G_NETWORK":"التبديل إلى 4G...",
"STANDBY":"في الانتظار",
"CONNECT_TO":"الاتصال بـ ",
"CONNECTING":"جاري الاتصال...",
"CONNECTED_TO":"متصل بـ ",
"LISTENING":"الاستماع...",
"SPEAKING":"التحدث...",
"SERVER_NOT_FOUND":"البحث عن خدمة متاحة",
"SERVER_NOT_CONNECTED":"لا يمكن الاتصال بالخدمة، يرجى المحاولة لاحقاً",
"SERVER_TIMEOUT":"انتهت مهلة الاستجابة",
"SERVER_ERROR":"فشل الإرسال، يرجى التحقق من الشبكة",
"CONNECT_TO_HOTSPOT":"اتصل الهاتف بنقطة الاتصال ",
"ACCESS_VIA_BROWSER":"،الوصول عبر المتصفح ",
"WIFI_CONFIG_MODE":"وضع تكوين الشبكة",
"ENTERING_WIFI_CONFIG_MODE":"الدخول في وضع تكوين الشبكة...",
"SCANNING_WIFI":"فحص Wi-Fi...",
"NEW_VERSION": "إصدار جديد ",
"OTA_UPGRADE":"تحديث OTA",
"UPGRADING":"تحديث النظام...",
"UPGRADE_FAILED":"فشل التحديث",
"ACTIVATION":"تفعيل الجهاز",
"BATTERY_LOW":"البطارية منخفضة",
"BATTERY_CHARGING":"جاري الشحن",
"BATTERY_FULL":"البطارية ممتلئة",
"BATTERY_NEED_CHARGE":"البطارية منخفضة، يرجى الشحن",
"VOLUME":"الصوت ",
"MUTED":"صامت",
"MAX_VOLUME":"أقصى صوت",
"RTC_MODE_OFF":"AEC مُوقف",
"RTC_MODE_ON":"AEC مُشغل"
}
}

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More