Compare commits

..

26 Commits

Author SHA1 Message Date
Xiaoxia
c0e702ff47 Bump to 1.5.6 2025-04-05 06:39:59 +08:00
Xiaoxia
91f98a49d9 magiclick 2p4/2p5: add click to reconfigure wifi 2025-04-05 03:27:53 +08:00
Xiaoxia
976ad2eac6 atk box build with wechat message style 2025-04-04 15:58:23 +08:00
Xiaoxia
fcd886c217 移除长按对话,改为单击多轮对话 2025-04-04 15:53:50 +08:00
MakerM0
eae0ca9315 增加电量显示和睡眠 (#444) 2025-04-04 15:37:40 +08:00
virgil
464633e7a1 Fix lcd bug (#443)
* fix: fixed the issue that the LCD does not light up when some devices are turned on.

* fix:  fix task sys_evt  stack overflow.
2025-04-04 15:33:17 +08:00
Xiaoxia
70883058e8 fix startup sound 2025-04-04 15:29:44 +08:00
HonestQiao
0ac2f09bb0 GPIO LED更新,可以设置LEDC定时器和控制通道 (#439)
* 添加普通LED支持,支持不同状态亮度设置和呼吸灯效果

* GPIO LED更新,可以设置LEDC定时器和控制通道
2025-04-03 02:25:03 +08:00
Xiaoxia
c9dfcb7c93 更新开发者文档(自定义开发板、IoT设备) 2025-04-02 09:11:41 +08:00
香草味的纳西妲
7c6b39e65d add: LVGL Image Converter Script 增加LVGL图片转换工具脚本,可以自定义小智的表情文件。 (#423)
* add: LVGL Image Converter

* add: README

* Rename README.MD to README.md

* remove readme image

---------

Co-authored-by: Xiaoxia <terrence@tenclass.com>
2025-04-01 06:51:11 +08:00
Kevincoooool
31c1f07481 Korvo开发板支持AEC (#431) 2025-03-31 00:07:12 +08:00
Terrence
488dfa5bf9 Bump to 1.5.5 2025-03-30 16:14:10 +08:00
Terrence
45012e38d4 Fix upgrade failure (main task stack overflow) 2025-03-30 16:13:49 +08:00
Xiaoxia
fa899a310e ESP-BOX-3 / LichuangDev enable AEC to support realtime chat (#429)
* read frame duration from server

* fit wechat style emoji size

* Make Wechat UI look better

* Add Realtime Chat to ESP-BOX-3/LichuangDev

* disable debug log

* Fix Sh1106 Compile Error Bug. IDF 5.3.2 Not supporting sh1106-esp-idf. (#424)

* Fix ESP32 Board Led No Light Bug (#427)

* add board esp32-s3-touch-lcd-3.5 (#415)

* add board esp32-s3-touch-lcd-3.5

* add axp2101

---------

Co-authored-by: flyingtjy <flyingtjy@gmail.com>

---------

Co-authored-by: ooxxU <71391474@qq.com>
Co-authored-by: flying1425 <79792003+flying1425@users.noreply.github.com>
Co-authored-by: flyingtjy <flyingtjy@gmail.com>
2025-03-30 09:07:08 +08:00
flying1425
e4c76eaa46 add board esp32-s3-touch-lcd-3.5 (#415)
* add board esp32-s3-touch-lcd-3.5

* add axp2101

---------

Co-authored-by: flyingtjy <flyingtjy@gmail.com>
2025-03-29 20:15:29 +08:00
ooxxU
69f736ac64 Fix ESP32 Board Led No Light Bug (#427) 2025-03-29 20:14:40 +08:00
ooxxU
2e9ce37321 Fix Sh1106 Compile Error Bug. IDF 5.3.2 Not supporting sh1106-esp-idf. (#424) 2025-03-29 12:01:09 +08:00
Lucinhu
8a0bb68c33 fix: 修复微信ui在消息数量超过上限后,气泡位置向上偏移的问题 (#418) 2025-03-27 20:02:45 +08:00
Eason
7ac252637d SH1106 Supported (#416)
* SH1106 Supported

* OLED_SSD1306_128X64_SH1106 -> OLED_SH1106_128X64
2025-03-26 23:11:08 +08:00
Terrence
4c7e00630b fix sensecap watcher partition alignment 2025-03-26 23:07:58 +08:00
Lucinhu
f05b46a0d9 typo: 删除sense-watcher显示修复的重复以及改掉错误的注释描述 (#417) 2025-03-26 22:15:12 +08:00
MOV
e486d8e91e Add feature cuican (#407)
* fix:Modify the README and add Moji images

* fix: Moji LCD initialization configuration.

* fix: DISPLAY_MIRROR_X false >> true

* fix: Moji GPIO conflict

* fix: Moji GPIO conflict

* add-feature-cuican

* update cuican image

* update config.h
2025-03-26 02:20:30 +08:00
ZhouKe
1f29fa44c2 change SPI mode for 7pin LCD (#405)
Co-authored-by: FanCheng <598592060@qq.com>
2025-03-26 00:39:48 +08:00
WMnologo
ae8aec1879 新增了0.85tft屏幕版本的星智开发板 (#397)
* 新增了0.85tft屏幕版本的星智开发板

* 对调了0.85寸屏幕版本的星智连接ml307模块的TX、RX引脚
2025-03-25 20:28:32 +08:00
Chinsyo
50c49023a7 Support adjust volume with knob in SenseCAP Watcher (#399)
* fix typo, add missing prefix `CONFIG` to `ESP_TASK_WDT_TIMEOUT_S`

* add KNOB gpio spec to sensecap config

* create new knob component

* implement ajust output volume with knob

* modify function name to UpperCamelCase

* Tidy up comments and logs
2025-03-25 02:37:43 +08:00
Terrence
af879d7806 Bump to 1.5.2 2025-03-22 23:12:04 +08:00
70 changed files with 4404 additions and 381 deletions

View File

@@ -4,7 +4,7 @@
# CMakeLists in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.16)
set(PROJECT_VER "1.5.1")
set(PROJECT_VER "1.5.6")
# Add this line to disable the specific warning
add_compile_options(-Wno-missing-field-initializers)

View File

@@ -2,7 +2,7 @@
(中文 | [English](README_en.md) | [日本語](README_ja.md)
这是虾哥的第一个硬件作品。
## 视频介绍
👉 [ESP32+SenseVoice+Qwen72B打造你的AI聊天伴侣【bilibili】](https://www.bilibili.com/video/BV11msTenEH3/)
@@ -12,7 +12,7 @@
## 项目目的
本项目是一个开源项目,以 MIT 许可证发布,允许任何人免费使用,并可以用于商业用途。
本项目是由虾哥开源的一个开源项目,以 MIT 许可证发布,允许任何人免费使用,并可以用于商业用途。
我们希望通过这个项目,能够帮助更多人入门 AI 硬件开发,了解如何将当下飞速发展的大语言模型应用到实际的硬件设备中。无论你是对 AI 感兴趣的学生,还是想要探索新技术的开发者,都可以通过这个项目获得宝贵的学习经验。
@@ -60,8 +60,8 @@
- <a href="https://github.com/Xinyuan-LilyGO/T-Circle-S3" target="_blank" title="LILYGO T-Circle-S3">LILYGO T-Circle-S3</a>
- <a href="https://oshwhub.com/tenclass01/xmini_c3" target="_blank" title="虾哥 Mini C3">虾哥 Mini C3</a>
- <a href="https://oshwhub.com/movecall/moji-xiaozhi-ai-derivative-editi" target="_blank" title="Movecall Moji ESP32S3">Moji 小智AI衍生版</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">璀璨·AI吊坠</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="无名科技Nologo-星智-1.54">无名科技Nologo-星智-1.54TFT</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="无名科技Nologo-星智-0.96">无名科技Nologo-星智-0.96TFT</a>
- <a href="https://www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html" target="_blank" title="SenseCAP Watcher">SenseCAP Watcher</a>
<div style="display: flex; justify-content: space-between;">
<a href="docs/v1/lichuang-s3.jpg" target="_blank" title="立创·实战派 ESP32-S3 开发板">
@@ -91,12 +91,12 @@
<a href="docs/v1/movecall-moji-esp32s3.jpg" target="_blank" title="Movecall Moji 小智AI衍生版">
<img src="docs/v1/movecall-moji-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-cuican-esp32s3.jpg" target="_blank" title="CuiCan">
<img src="docs/v1/movecall-cuican-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/wmnologo_xingzhi_1.54.jpg" target="_blank" title="无名科技Nologo-星智-1.54">
<img src="docs/v1/wmnologo_xingzhi_1.54.jpg" width="240" />
</a>
<a href="docs/v1/wmnologo_xingzhi_0.96.jpg" target="_blank" title="无名科技Nologo-星智-0.96">
<img src="docs/v1/wmnologo_xingzhi_0.96.jpg" width="240" />
</a>
<a href="docs/v1/sensecap_watcher.jpg" target="_blank" title="SenseCAP Watcher">
<img src="docs/v1/sensecap_watcher.jpg" width="240" />
</a>
@@ -120,6 +120,11 @@
- Linux 比 Windows 更好,编译速度快,也免去驱动问题的困扰
- 使用 Google C++ 代码风格,提交代码时请确保符合规范
### 开发者文档
- [开发板定制指南](main/boards/README.md) - 学习如何为小智创建自定义开发板适配
- [物联网控制模块](main/iot/README.md) - 了解如何通过AI语音控制物联网设备
## 智能体配置

View File

@@ -2,7 +2,7 @@
([中文](README.md) | English | [日本語](README_ja.md))
This is Terrence's first hardware project.
## Introduction
👉 [Build your AI chat companion with ESP32+SenseVoice+Qwen72B!【bilibili】](https://www.bilibili.com/video/BV11msTenEH3/)
@@ -60,6 +60,8 @@ Breadboard demonstration:
- <a href="https://github.com/Xinyuan-LilyGO/T-Circle-S3" target="_blank" title="LILYGO T-Circle-S3">LILYGO T-Circle-S3</a>
- <a href="https://oshwhub.com/tenclass01/xmini_c3" target="_blank" title="XiaGe Mini C3">XiaGe Mini C3</a>
- <a href="https://oshwhub.com/movecall/moji-xiaozhi-ai-derivative-editi" target="_blank" title="Movecall Moji ESP32S3">Moji XiaoZhi AI Derivative Version</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">CuiCan AI pendant</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="WMnologo-Xingzhi-1.54">WMnologo-Xingzhi-1.54TFT</a>
- <a href="https://www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html" target="_blank" title="SenseCAP Watcher">SenseCAP Watcher</a>
<div style="display: flex; justify-content: space-between;">
@@ -93,6 +95,12 @@ Breadboard demonstration:
<a href="docs/v1/movecall-moji-esp32s3.jpg" target="_blank" title="Moji">
<img src="docs/v1/movecall-moji-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-cuican-esp32s3.jpg" target="_blank" title="CuiCan">
<img src="docs/v1/movecall-cuican-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/wmnologo_xingzhi_1.54.jpg" target="_blank" title="WMnologo-Xingzhi-1.54">
<img src="docs/v1/wmnologo_xingzhi_1.54.jpg" width="240" />
</a>
<a href="docs/v1/sensecap_watcher.jpg" target="_blank" title="SenseCAP Watcher">
<img src="docs/v1/sensecap_watcher.jpg" width="240" />
</a>
@@ -115,6 +123,11 @@ The firmware connects to the official [xiaozhi.me](https://xiaozhi.me) server by
- Linux is preferred over Windows for faster compilation and fewer driver issues
- Use Google C++ code style, ensure compliance when submitting code
### Developer Documentation
- [Board Customization Guide](main/boards/README.md) - Learn how to create custom board adaptations for XiaoZhi
- [IoT Control Module](main/iot/README.md) - Understand how to control IoT devices through AI voice commands
## AI Agent Configuration
If you already have a XiaoZhi AI chatbot device, you can configure it through the [xiaozhi.me](https://xiaozhi.me) console.

View File

@@ -2,7 +2,7 @@
([中文](README.md) | [English](README_en.md) | 日本語)
これは シャーガーTerrenceの最初のハードウェア作品です。
## プロジェクト紹介
👉 [ESP32+SenseVoice+Qwen72Bで AI チャット仲間を作ろう【bilibili】](https://www.bilibili.com/video/BV11msTenEH3/)
@@ -60,6 +60,8 @@ Feishu ドキュメントチュートリアルをご覧ください:
- <a href="https://github.com/Xinyuan-LilyGO/T-Circle-S3" target="_blank" title="LILYGO T-Circle-S3">LILYGO T-Circle-S3</a>
- <a href="https://oshwhub.com/tenclass01/xmini_c3" target="_blank" title="XiaGe Mini C3">XiaGe Mini C3</a>
- <a href="https://oshwhub.com/movecall/moji-xiaozhi-ai-derivative-editi" target="_blank" title="Movecall Moji ESP32S3">Moji シャオジー AI 派生版</a>
- <a href="https://oshwhub.com/movecall/cuican-ai-pendant-lights-up-y" target="_blank" title="Movecall CuiCan ESP32S3">Cuican AI ペンダント</a>
- <a href="https://github.com/WMnologo/xingzhi-ai" target="_blank" title="無名科技Nologo-星智-1.54">無名科技Nologo-星智-1.54TFT</a>
- <a href="https://www.seeedstudio.com/SenseCAP-Watcher-W1-A-p-5979.html" target="_blank" title="SenseCAP Watcher">SenseCAP Watcher</a>
<div style="display: flex; justify-content: space-between;">
@@ -90,6 +92,12 @@ Feishu ドキュメントチュートリアルをご覧ください:
<a href="docs/v1/movecall-moji-esp32s3.jpg" target="_blank" title="Moji">
<img src="docs/v1/movecall-moji-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/movecall-cuican-esp32s3.jpg" target="_blank" title="CuiCan">
<img src="docs/v1/movecall-cuican-esp32s3.jpg" width="240" />
</a>
<a href="docs/v1/wmnologo_xingzhi_1.54.jpg" target="_blank" title="無名科技Nologo-星智-1.54">
<img src="docs/v1/wmnologo_xingzhi_1.54.jpg" width="240" />
</a>
<a href="docs/v1/sensecap_watcher.jpg" target="_blank" title="SenseCAP Watcher">
<img src="docs/v1/sensecap_watcher.jpg" width="240" />
</a>
@@ -112,6 +120,11 @@ Feishu ドキュメントチュートリアルをご覧ください:
- Linux は Windows より好ましい(コンパイルが速く、ドライバーの問題も少ない)
- Google C++ コードスタイルを使用、コード提出時にはコンプライアンスを確認
### 開発者ドキュメント
- [ボードカスタマイズガイド](main/boards/README.md) - シャオジー向けのカスタムボード適応を作成する方法を学ぶ
- [IoT 制御モジュール](main/iot/README.md) - AI 音声コマンドでIoTデバイスを制御する方法を理解する
## AI エージェント設定
シャオジー AI チャットボットデバイスをお持ちの場合は、[xiaozhi.me](https://xiaozhi.me) コンソールで設定できます。

Binary file not shown.

After

Width:  |  Height:  |  Size: 50 KiB

View File

@@ -39,7 +39,7 @@ elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ML307)
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ESP32)
set(BOARD_TYPE "bread-compact-esp32")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ESP32_LCD)
set(BOARD_TYPE "bread-compact-esp32-lcd")
set(BOARD_TYPE "bread-compact-esp32-lcd")
elseif(CONFIG_BOARD_TYPE_DF_K10)
set(BOARD_TYPE "df-k10")
elseif(CONFIG_BOARD_TYPE_ESP_BOX_3)
@@ -96,6 +96,8 @@ elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_1_85)
set(BOARD_TYPE "esp32-s3-touch-lcd-1.85")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_1_46)
set(BOARD_TYPE "esp32-s3-touch-lcd-1.46")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Touch_LCD_3_5)
set(BOARD_TYPE "esp32-s3-touch-lcd-3.5")
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI_LCD)
set(BOARD_TYPE "bread-compact-wifi-lcd")
elseif(CONFIG_BOARD_TYPE_TUDOUZI)
@@ -106,6 +108,8 @@ elseif(CONFIG_BOARD_TYPE_LILYGO_T_CAMERAPLUS_S3)
set(BOARD_TYPE "lilygo-t-cameraplus-s3")
elseif(CONFIG_BOARD_TYPE_MOVECALL_MOJI_ESP32S3)
set(BOARD_TYPE "movecall-moji-esp32s3")
elseif(CONFIG_BOARD_TYPE_MOVECALL_CUICAN_ESP32S3)
set(BOARD_TYPE "movecall-cuican-esp32s3")
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3)
set(BOARD_TYPE "atk-dnesp32s3")
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX)
@@ -114,6 +118,10 @@ elseif(CONFIG_BOARD_TYPE_DU_CHATX)
set(BOARD_TYPE "du-chatx")
elseif(CONFIG_BOARD_TYPE_ESP32S3_Taiji_Pi)
set(BOARD_TYPE "taiji-pi-s3")
elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_0_85TFT_WIFI)
set(BOARD_TYPE "xingzhi-cube-0.85tft-wifi")
elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_0_85TFT_ML307)
set(BOARD_TYPE "xingzhi-cube-0.85tft-ml307")
elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_0_96OLED_WIFI)
set(BOARD_TYPE "xingzhi-cube-0.96oled-wifi")
elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_0_96OLED_ML307)
@@ -125,7 +133,7 @@ elseif(CONFIG_BOARD_TYPE_XINGZHI_Cube_1_54TFT_ML307)
elseif(CONFIG_BOARD_TYPE_SENSECAP_WATCHER)
set(BOARD_TYPE "sensecap-watcher")
elseif(CONFIG_BOARD_TYPE_ESP32_CGC)
set(BOARD_TYPE "esp32-cgc")
set(BOARD_TYPE "esp32-cgc")
endif()
file(GLOB BOARD_SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/boards/${BOARD_TYPE}/*.cc

View File

@@ -122,6 +122,8 @@ choice BOARD_TYPE
bool "Waveshare ESP32-S3-Touch-LCD-1.85"
config BOARD_TYPE_ESP32S3_Touch_LCD_1_46
bool "Waveshare ESP32-S3-Touch-LCD-1.46"
config BOARD_TYPE_ESP32S3_Touch_LCD_3_5
bool "Waveshare ESP32-S3-Touch-LCD-3.5"
config BOARD_TYPE_TUDOUZI
bool "土豆子"
config BOARD_TYPE_LILYGO_T_CIRCLE_S3
@@ -130,6 +132,8 @@ choice BOARD_TYPE
bool "LILYGO T-CameraPlus-S3"
config BOARD_TYPE_MOVECALL_MOJI_ESP32S3
bool "Movecall Moji 小智AI衍生版"
config BOARD_TYPE_MOVECALL_CUICAN_ESP32S3
bool "Movecall CuiCan 璀璨·AI吊坠"
config BOARD_TYPE_ATK_DNESP32S3
bool "正点原子DNESP32S3开发板"
config BOARD_TYPE_ATK_DNESP32S3_BOX
@@ -138,6 +142,10 @@ choice BOARD_TYPE
bool "嘟嘟开发板CHATX(wifi)"
config BOARD_TYPE_ESP32S3_Taiji_Pi
bool "太极小派esp32s3"
config BOARD_TYPE_XINGZHI_Cube_0_85TFT_WIFI
bool "无名科技星智0.85(WIFI)"
config BOARD_TYPE_XINGZHI_Cube_0_85TFT_ML307
bool "无名科技星智0.85(ML307)"
config BOARD_TYPE_XINGZHI_Cube_0_96OLED_WIFI
bool "无名科技星智0.96(WIFI)"
config BOARD_TYPE_XINGZHI_Cube_0_96OLED_ML307
@@ -160,6 +168,8 @@ choice DISPLAY_OLED_TYPE
bool "SSD1306, 分辨率128*32"
config OLED_SSD1306_128X64
bool "SSD1306, 分辨率128*64"
config OLED_SH1106_128X64
bool "SH1106, 分辨率128*64"
endchoice
choice DISPLAY_LCD_TYPE
@@ -218,7 +228,14 @@ config USE_WECHAT_MESSAGE_STYLE
bool "使用微信聊天界面风格"
default n
help
使用微信聊天界面风格
使用微信聊天界面风格
config USE_WAKE_WORD_DETECT
bool "启用唤醒词检测"
default y
depends on IDF_TARGET_ESP32S3 && SPIRAM
help
需要 ESP32 S3 与 AFE 支持
config USE_AUDIO_PROCESSOR
bool "启用音频降噪、增益处理"
@@ -227,10 +244,11 @@ config USE_AUDIO_PROCESSOR
help
需要 ESP32 S3 与 AFE 支持
config USE_WAKE_WORD_DETECT
bool "启用唤醒词检测"
default y
depends on IDF_TARGET_ESP32S3 && SPIRAM
config USE_REALTIME_CHAT
bool "启用可语音打断的实时对话模式(需要 AEC 支持)"
default n
depends on USE_AUDIO_PROCESSOR && (BOARD_TYPE_ESP_BOX_3 || BOARD_TYPE_ESP_BOX || BOARD_TYPE_LICHUANG_DEV || BOARD_TYPE_ESP32S3_KORVO2_V3)
help
需要 ESP32 S3 与 AFE 支持
需要 ESP32 S3 与 AEC 开启,因为性能不够,不建议和微信聊天界面风格同时开启
endmenu

View File

@@ -155,6 +155,7 @@ void Application::CheckNewVersion() {
SetDeviceState(kDeviceStateIdle);
display->SetChatMessage("system", "");
ResetDecoder();
PlaySound(Lang::Sounds::P3_SUCCESS);
// Exit the loop if upgrade or idle
break;
@@ -203,6 +204,7 @@ void Application::Alert(const char* status, const char* message, const char* emo
display->SetEmotion(emotion);
display->SetChatMessage("system", message);
if (!sound.empty()) {
ResetDecoder();
PlaySound(sound);
}
}
@@ -217,9 +219,8 @@ void Application::DismissAlert() {
}
void Application::PlaySound(const std::string_view& sound) {
auto codec = Board::GetInstance().GetAudioCodec();
codec->EnableOutput(true);
SetDecodeSampleRate(16000);
// The assets are encoded at 16000Hz, 60ms frame duration
SetDecodeSampleRate(16000, 60);
const char* data = sound.data();
size_t size = sound.size();
for (const char* p = data; p < data + size; ) {
@@ -255,9 +256,7 @@ void Application::ToggleChatState() {
return;
}
keep_listening_ = true;
protocol_->SendStartListening(kListeningModeAutoStop);
SetDeviceState(kDeviceStateListening);
SetListeningMode(realtime_chat_enabled_ ? kListeningModeRealtime : kListeningModeAutoStop);
});
} else if (device_state_ == kDeviceStateSpeaking) {
Schedule([this]() {
@@ -281,7 +280,6 @@ void Application::StartListening() {
return;
}
keep_listening_ = false;
if (device_state_ == kDeviceStateIdle) {
Schedule([this]() {
if (!protocol_->IsAudioChannelOpened()) {
@@ -290,14 +288,13 @@ void Application::StartListening() {
return;
}
}
protocol_->SendStartListening(kListeningModeManualStop);
SetDeviceState(kDeviceStateListening);
SetListeningMode(kListeningModeManualStop);
});
} else if (device_state_ == kDeviceStateSpeaking) {
Schedule([this]() {
AbortSpeaking(kAbortReasonNone);
protocol_->SendStartListening(kListeningModeManualStop);
SetDeviceState(kDeviceStateListening);
SetListeningMode(kListeningModeManualStop);
});
}
}
@@ -320,12 +317,12 @@ void Application::Start() {
/* Setup the audio codec */
auto codec = board.GetAudioCodec();
opus_decode_sample_rate_ = codec->output_sample_rate();
opus_decoder_ = std::make_unique<OpusDecoderWrapper>(opus_decode_sample_rate_, 1);
opus_decoder_ = std::make_unique<OpusDecoderWrapper>(codec->output_sample_rate(), 1, OPUS_FRAME_DURATION_MS);
opus_encoder_ = std::make_unique<OpusEncoderWrapper>(16000, 1, OPUS_FRAME_DURATION_MS);
// For ML307 boards, we use complexity 5 to save bandwidth
// For other boards, we use complexity 3 to save CPU
if (board.GetBoardType() == "ml307") {
if (realtime_chat_enabled_) {
ESP_LOGI(TAG, "Realtime chat enabled, setting opus encoder complexity to 0");
opus_encoder_->SetComplexity(0);
} else if (board.GetBoardType() == "ml307") {
ESP_LOGI(TAG, "ML307 board detected, setting opus encoder complexity to 5");
opus_encoder_->SetComplexity(5);
} else {
@@ -337,24 +334,20 @@ void Application::Start() {
input_resampler_.Configure(codec->input_sample_rate(), 16000);
reference_resampler_.Configure(codec->input_sample_rate(), 16000);
}
codec->OnInputReady([this, codec]() {
BaseType_t higher_priority_task_woken = pdFALSE;
xEventGroupSetBitsFromISR(event_group_, AUDIO_INPUT_READY_EVENT, &higher_priority_task_woken);
return higher_priority_task_woken == pdTRUE;
});
codec->OnOutputReady([this]() {
BaseType_t higher_priority_task_woken = pdFALSE;
xEventGroupSetBitsFromISR(event_group_, AUDIO_OUTPUT_READY_EVENT, &higher_priority_task_woken);
return higher_priority_task_woken == pdTRUE;
});
codec->Start();
xTaskCreatePinnedToCore([](void* arg) {
Application* app = (Application*)arg;
app->AudioLoop();
vTaskDelete(NULL);
}, "audio_loop", 4096 * 2, this, 8, &audio_loop_task_handle_, realtime_chat_enabled_ ? 1 : 0);
/* Start the main loop */
xTaskCreate([](void* arg) {
xTaskCreatePinnedToCore([](void* arg) {
Application* app = (Application*)arg;
app->MainLoop();
vTaskDelete(NULL);
}, "main_loop", 4096 * 2, this, 4, nullptr);
}, "main_loop", 4096 * 2, this, 4, &main_loop_task_handle_, 0);
/* Wait for the network to be ready */
board.StartNetwork();
@@ -372,9 +365,7 @@ void Application::Start() {
});
protocol_->OnIncomingAudio([this](std::vector<uint8_t>&& data) {
std::lock_guard<std::mutex> lock(mutex_);
if (device_state_ == kDeviceStateSpeaking) {
audio_decode_queue_.emplace_back(std::move(data));
}
audio_decode_queue_.emplace_back(std::move(data));
});
protocol_->OnAudioChannelOpened([this, codec, &board]() {
board.SetPowerSaveMode(false);
@@ -382,7 +373,7 @@ void Application::Start() {
ESP_LOGW(TAG, "Server sample rate %d does not match device output sample rate %d, resampling may cause distortion",
protocol_->server_sample_rate(), codec->output_sample_rate());
}
SetDecodeSampleRate(protocol_->server_sample_rate());
SetDecodeSampleRate(protocol_->server_sample_rate(), protocol_->server_frame_duration());
auto& thing_manager = iot::ThingManager::GetInstance();
protocol_->SendIotDescriptors(thing_manager.GetDescriptorsJson());
std::string states;
@@ -412,13 +403,12 @@ void Application::Start() {
});
} else if (strcmp(state->valuestring, "stop") == 0) {
Schedule([this]() {
background_task_->WaitForCompletion();
if (device_state_ == kDeviceStateSpeaking) {
background_task_->WaitForCompletion();
if (keep_listening_) {
protocol_->SendStartListening(kListeningModeAutoStop);
SetDeviceState(kDeviceStateListening);
} else {
if (listening_mode_ == kListeningModeManualStop) {
SetDeviceState(kDeviceStateIdle);
} else {
SetDeviceState(kDeviceStateListening);
}
}
});
@@ -474,7 +464,7 @@ void Application::Start() {
}, "check_new_version", 4096 * 2, this, 2, nullptr);
#if CONFIG_USE_AUDIO_PROCESSOR
audio_processor_.Initialize(codec->input_channels(), codec->input_reference());
audio_processor_.Initialize(codec, realtime_chat_enabled_);
audio_processor_.OnOutput([this](std::vector<int16_t>&& data) {
background_task_->Schedule([this, data = std::move(data)]() mutable {
opus_encoder_->Encode(std::move(data), [this](std::vector<uint8_t>&& opus) {
@@ -500,7 +490,7 @@ void Application::Start() {
#endif
#if CONFIG_USE_WAKE_WORD_DETECT
wake_word_detect_.Initialize(codec->input_channels(), codec->input_reference());
wake_word_detect_.Initialize(codec);
wake_word_detect_.OnWakeWordDetected([this](const std::string& wake_word) {
Schedule([this, &wake_word]() {
if (device_state_ == kDeviceStateIdle) {
@@ -520,8 +510,7 @@ void Application::Start() {
// Set the chat state to wake word detected
protocol_->SendWakeWordDetected(wake_word);
ESP_LOGI(TAG, "Wake word detected: %s", wake_word.c_str());
keep_listening_ = true;
SetDeviceState(kDeviceStateIdle);
SetListeningMode(realtime_chat_enabled_ ? kListeningModeRealtime : kListeningModeAutoStop);
} else if (device_state_ == kDeviceStateSpeaking) {
AbortSpeaking(kAbortReasonWakeWordDetected);
} else if (device_state_ == kDeviceStateActivating) {
@@ -534,6 +523,13 @@ void Application::Start() {
SetDeviceState(kDeviceStateIdle);
esp_timer_start_periodic(clock_timer_handle_, 1000000);
#if 0
while (true) {
SystemInfo::PrintRealTimeStats(pdMS_TO_TICKS(1000));
vTaskDelay(pdMS_TO_TICKS(10000));
}
#endif
}
void Application::OnClockTimer() {
@@ -541,7 +537,6 @@ void Application::OnClockTimer() {
// Print the debug info every 10 seconds
if (clock_ticks_ % 10 == 0) {
// SystemInfo::PrintRealTimeStats(pdMS_TO_TICKS(1000));
int free_sram = heap_caps_get_free_size(MALLOC_CAP_INTERNAL);
int min_free_sram = heap_caps_get_minimum_free_size(MALLOC_CAP_INTERNAL);
ESP_LOGI(TAG, "Free internal: %u minimal internal: %u", free_sram, min_free_sram);
@@ -561,6 +556,7 @@ void Application::OnClockTimer() {
}
}
// Add a async task to MainLoop
void Application::Schedule(std::function<void()> callback) {
{
std::lock_guard<std::mutex> lock(mutex_);
@@ -574,16 +570,8 @@ void Application::Schedule(std::function<void()> callback) {
// they should use Schedule to call this function
void Application::MainLoop() {
while (true) {
auto bits = xEventGroupWaitBits(event_group_,
SCHEDULE_EVENT | AUDIO_INPUT_READY_EVENT | AUDIO_OUTPUT_READY_EVENT,
pdTRUE, pdFALSE, portMAX_DELAY);
auto bits = xEventGroupWaitBits(event_group_, SCHEDULE_EVENT, pdTRUE, pdFALSE, portMAX_DELAY);
if (bits & AUDIO_INPUT_READY_EVENT) {
InputAudio();
}
if (bits & AUDIO_OUTPUT_READY_EVENT) {
OutputAudio();
}
if (bits & SCHEDULE_EVENT) {
std::unique_lock<std::mutex> lock(mutex_);
std::list<std::function<void()>> tasks = std::move(main_tasks_);
@@ -595,14 +583,18 @@ void Application::MainLoop() {
}
}
void Application::ResetDecoder() {
std::lock_guard<std::mutex> lock(mutex_);
opus_decoder_->ResetState();
audio_decode_queue_.clear();
last_output_time_ = std::chrono::steady_clock::now();
// The Audio Loop is used to input and output audio data
void Application::AudioLoop() {
auto codec = Board::GetInstance().GetAudioCodec();
while (true) {
OnAudioInput();
if (codec->output_enabled()) {
OnAudioOutput();
}
}
}
void Application::OutputAudio() {
void Application::OnAudioOutput() {
auto now = std::chrono::steady_clock::now();
auto codec = Board::GetInstance().GetAudioCodec();
const int max_silence_seconds = 10;
@@ -624,7 +616,6 @@ void Application::OutputAudio() {
return;
}
last_output_time_ = now;
auto opus = std::move(audio_decode_queue_.front());
audio_decode_queue_.pop_front();
lock.unlock();
@@ -638,27 +629,57 @@ void Application::OutputAudio() {
if (!opus_decoder_->Decode(std::move(opus), pcm)) {
return;
}
// Resample if the sample rate is different
if (opus_decode_sample_rate_ != codec->output_sample_rate()) {
if (opus_decoder_->sample_rate() != codec->output_sample_rate()) {
int target_size = output_resampler_.GetOutputSamples(pcm.size());
std::vector<int16_t> resampled(target_size);
output_resampler_.Process(pcm.data(), pcm.size(), resampled.data());
pcm = std::move(resampled);
}
codec->OutputData(pcm);
last_output_time_ = std::chrono::steady_clock::now();
});
}
void Application::InputAudio() {
auto codec = Board::GetInstance().GetAudioCodec();
void Application::OnAudioInput() {
std::vector<int16_t> data;
if (!codec->InputData(data)) {
#if CONFIG_USE_WAKE_WORD_DETECT
if (wake_word_detect_.IsDetectionRunning()) {
ReadAudio(data, 16000, wake_word_detect_.GetFeedSize());
wake_word_detect_.Feed(data);
return;
}
#endif
#if CONFIG_USE_AUDIO_PROCESSOR
if (audio_processor_.IsRunning()) {
ReadAudio(data, 16000, audio_processor_.GetFeedSize());
audio_processor_.Feed(data);
return;
}
#else
if (device_state_ == kDeviceStateListening) {
ReadAudio(data, 16000, 30 * 16000 / 1000);
background_task_->Schedule([this, data = std::move(data)]() mutable {
opus_encoder_->Encode(std::move(data), [this](std::vector<uint8_t>&& opus) {
Schedule([this, opus = std::move(opus)]() {
protocol_->SendAudio(opus);
});
});
});
return;
}
#endif
vTaskDelay(pdMS_TO_TICKS(30));
}
if (codec->input_sample_rate() != 16000) {
void Application::ReadAudio(std::vector<int16_t>& data, int sample_rate, int samples) {
auto codec = Board::GetInstance().GetAudioCodec();
if (codec->input_sample_rate() != sample_rate) {
data.resize(samples * codec->input_sample_rate() / sample_rate);
if (!codec->InputData(data)) {
return;
}
if (codec->input_channels() == 2) {
auto mic_channel = std::vector<int16_t>(data.size() / 2);
auto reference_channel = std::vector<int16_t>(data.size() / 2);
@@ -680,28 +701,12 @@ void Application::InputAudio() {
input_resampler_.Process(data.data(), data.size(), resampled.data());
data = std::move(resampled);
}
} else {
data.resize(samples);
if (!codec->InputData(data)) {
return;
}
}
#if CONFIG_USE_WAKE_WORD_DETECT
if (wake_word_detect_.IsDetectionRunning()) {
wake_word_detect_.Feed(data);
}
#endif
#if CONFIG_USE_AUDIO_PROCESSOR
if (audio_processor_.IsRunning()) {
audio_processor_.Input(data);
}
#else
if (device_state_ == kDeviceStateListening) {
background_task_->Schedule([this, data = std::move(data)]() mutable {
opus_encoder_->Encode(std::move(data), [this](std::vector<uint8_t>&& opus) {
Schedule([this, opus = std::move(opus)]() {
protocol_->SendAudio(opus);
});
});
});
}
#endif
}
void Application::AbortSpeaking(AbortReason reason) {
@@ -710,6 +715,11 @@ void Application::AbortSpeaking(AbortReason reason) {
protocol_->SendAbortSpeaking(reason);
}
void Application::SetListeningMode(ListeningMode mode) {
listening_mode_ = mode;
SetDeviceState(kDeviceStateListening);
}
void Application::SetDeviceState(DeviceState state) {
if (device_state_ == state) {
return;
@@ -723,7 +733,6 @@ void Application::SetDeviceState(DeviceState state) {
background_task_->WaitForCompletion();
auto& board = Board::GetInstance();
auto codec = board.GetAudioCodec();
auto display = board.GetDisplay();
auto led = board.GetLed();
led->OnStateChanged();
@@ -747,30 +756,43 @@ void Application::SetDeviceState(DeviceState state) {
case kDeviceStateListening:
display->SetStatus(Lang::Strings::LISTENING);
display->SetEmotion("neutral");
ResetDecoder();
opus_encoder_->ResetState();
#if CONFIG_USE_AUDIO_PROCESSOR
audio_processor_.Start();
#endif
#if CONFIG_USE_WAKE_WORD_DETECT
wake_word_detect_.StopDetection();
#endif
// Update the IoT states before sending the start listening command
UpdateIotStates();
if (previous_state == kDeviceStateSpeaking) {
// FIXME: Wait for the speaker to empty the buffer
vTaskDelay(pdMS_TO_TICKS(120));
// Make sure the audio processor is running
#if CONFIG_USE_AUDIO_PROCESSOR
if (!audio_processor_.IsRunning()) {
#else
if (true) {
#endif
// Send the start listening command
protocol_->SendStartListening(listening_mode_);
if (listening_mode_ == kListeningModeAutoStop && previous_state == kDeviceStateSpeaking) {
// FIXME: Wait for the speaker to empty the buffer
vTaskDelay(pdMS_TO_TICKS(120));
}
opus_encoder_->ResetState();
#if CONFIG_USE_WAKE_WORD_DETECT
wake_word_detect_.StopDetection();
#endif
#if CONFIG_USE_AUDIO_PROCESSOR
audio_processor_.Start();
#endif
}
break;
case kDeviceStateSpeaking:
display->SetStatus(Lang::Strings::SPEAKING);
ResetDecoder();
codec->EnableOutput(true);
if (listening_mode_ != kListeningModeRealtime) {
#if CONFIG_USE_AUDIO_PROCESSOR
audio_processor_.Stop();
audio_processor_.Stop();
#endif
#if CONFIG_USE_WAKE_WORD_DETECT
wake_word_detect_.StartDetection();
wake_word_detect_.StartDetection();
#endif
}
ResetDecoder();
break;
default:
// Do nothing
@@ -778,19 +800,28 @@ void Application::SetDeviceState(DeviceState state) {
}
}
void Application::SetDecodeSampleRate(int sample_rate) {
if (opus_decode_sample_rate_ == sample_rate) {
void Application::ResetDecoder() {
std::lock_guard<std::mutex> lock(mutex_);
opus_decoder_->ResetState();
audio_decode_queue_.clear();
last_output_time_ = std::chrono::steady_clock::now();
auto codec = Board::GetInstance().GetAudioCodec();
codec->EnableOutput(true);
}
void Application::SetDecodeSampleRate(int sample_rate, int frame_duration) {
if (opus_decoder_->sample_rate() == sample_rate && opus_decoder_->duration_ms() == frame_duration) {
return;
}
opus_decode_sample_rate_ = sample_rate;
opus_decoder_.reset();
opus_decoder_ = std::make_unique<OpusDecoderWrapper>(opus_decode_sample_rate_, 1);
opus_decoder_ = std::make_unique<OpusDecoderWrapper>(sample_rate, 1, frame_duration);
auto codec = Board::GetInstance().GetAudioCodec();
if (opus_decode_sample_rate_ != codec->output_sample_rate()) {
ESP_LOGI(TAG, "Resampling audio from %d to %d", opus_decode_sample_rate_, codec->output_sample_rate());
output_resampler_.Configure(opus_decode_sample_rate_, codec->output_sample_rate());
if (opus_decoder_->sample_rate() != codec->output_sample_rate()) {
ESP_LOGI(TAG, "Resampling audio from %d to %d", opus_decoder_->sample_rate(), codec->output_sample_rate());
output_resampler_.Configure(opus_decoder_->sample_rate(), codec->output_sample_rate());
}
}

View File

@@ -88,12 +88,20 @@ private:
EventGroupHandle_t event_group_ = nullptr;
esp_timer_handle_t clock_timer_handle_ = nullptr;
volatile DeviceState device_state_ = kDeviceStateUnknown;
bool keep_listening_ = false;
ListeningMode listening_mode_ = kListeningModeAutoStop;
#if CONFIG_USE_REALTIME_CHAT
bool realtime_chat_enabled_ = true;
#else
bool realtime_chat_enabled_ = false;
#endif
bool aborted_ = false;
bool voice_detected_ = false;
int clock_ticks_ = 0;
TaskHandle_t main_loop_task_handle_ = nullptr;
TaskHandle_t check_new_version_task_handle_ = nullptr;
// Audio encode / decode
TaskHandle_t audio_loop_task_handle_ = nullptr;
BackgroundTask* background_task_ = nullptr;
std::chrono::steady_clock::time_point last_output_time_;
std::list<std::vector<uint8_t>> audio_decode_queue_;
@@ -101,19 +109,21 @@ private:
std::unique_ptr<OpusEncoderWrapper> opus_encoder_;
std::unique_ptr<OpusDecoderWrapper> opus_decoder_;
int opus_decode_sample_rate_ = -1;
OpusResampler input_resampler_;
OpusResampler reference_resampler_;
OpusResampler output_resampler_;
void MainLoop();
void InputAudio();
void OutputAudio();
void OnAudioInput();
void OnAudioOutput();
void ReadAudio(std::vector<int16_t>& data, int sample_rate, int samples);
void ResetDecoder();
void SetDecodeSampleRate(int sample_rate);
void SetDecodeSampleRate(int sample_rate, int frame_duration);
void CheckNewVersion();
void ShowActivationCode();
void OnClockTimer();
void SetListeningMode(ListeningMode mode);
void AudioLoop();
};
#endif // _APPLICATION_H_

View File

@@ -14,23 +14,11 @@ AudioCodec::AudioCodec() {
AudioCodec::~AudioCodec() {
}
void AudioCodec::OnInputReady(std::function<bool()> callback) {
on_input_ready_ = callback;
}
void AudioCodec::OnOutputReady(std::function<bool()> callback) {
on_output_ready_ = callback;
}
void AudioCodec::OutputData(std::vector<int16_t>& data) {
Write(data.data(), data.size());
}
bool AudioCodec::InputData(std::vector<int16_t>& data) {
int duration = 30;
int input_frame_size = input_sample_rate_ / 1000 * duration * input_channels_;
data.resize(input_frame_size);
int samples = Read(data.data(), data.size());
if (samples > 0) {
return true;
@@ -38,22 +26,6 @@ bool AudioCodec::InputData(std::vector<int16_t>& data) {
return false;
}
IRAM_ATTR bool AudioCodec::on_sent(i2s_chan_handle_t handle, i2s_event_data_t *event, void *user_ctx) {
auto audio_codec = (AudioCodec*)user_ctx;
if (audio_codec->output_enabled_ && audio_codec->on_output_ready_) {
return audio_codec->on_output_ready_();
}
return false;
}
IRAM_ATTR bool AudioCodec::on_recv(i2s_chan_handle_t handle, i2s_event_data_t *event, void *user_ctx) {
auto audio_codec = (AudioCodec*)user_ctx;
if (audio_codec->input_enabled_ && audio_codec->on_input_ready_) {
return audio_codec->on_input_ready_();
}
return false;
}
void AudioCodec::Start() {
Settings settings("audio", false);
output_volume_ = settings.GetInt("output_volume", output_volume_);
@@ -62,15 +34,6 @@ void AudioCodec::Start() {
output_volume_ = 10;
}
// 注册音频数据回调
i2s_event_callbacks_t rx_callbacks = {};
rx_callbacks.on_recv = on_recv;
i2s_channel_register_event_callback(rx_handle_, &rx_callbacks, this);
i2s_event_callbacks_t tx_callbacks = {};
tx_callbacks.on_sent = on_sent;
i2s_channel_register_event_callback(tx_handle_, &tx_callbacks, this);
ESP_ERROR_CHECK(i2s_channel_enable(tx_handle_));
ESP_ERROR_CHECK(i2s_channel_enable(rx_handle_));

View File

@@ -23,8 +23,6 @@ public:
void Start();
void OutputData(std::vector<int16_t>& data);
bool InputData(std::vector<int16_t>& data);
void OnOutputReady(std::function<bool()> callback);
void OnInputReady(std::function<bool()> callback);
inline bool duplex() const { return duplex_; }
inline bool input_reference() const { return input_reference_; }
@@ -33,13 +31,8 @@ public:
inline int input_channels() const { return input_channels_; }
inline int output_channels() const { return output_channels_; }
inline int output_volume() const { return output_volume_; }
private:
std::function<bool()> on_input_ready_;
std::function<bool()> on_output_ready_;
IRAM_ATTR static bool on_recv(i2s_chan_handle_t handle, i2s_event_data_t *event, void *user_ctx);
IRAM_ATTR static bool on_sent(i2s_chan_handle_t handle, i2s_event_data_t *event, void *user_ctx);
inline bool input_enabled() const { return input_enabled_; }
inline bool output_enabled() const { return output_enabled_; }
protected:
i2s_chan_handle_t tx_handle_ = nullptr;

View File

@@ -10,31 +10,41 @@ AudioProcessor::AudioProcessor()
event_group_ = xEventGroupCreate();
}
void AudioProcessor::Initialize(int channels, bool reference) {
channels_ = channels;
reference_ = reference;
int ref_num = reference_ ? 1 : 0;
void AudioProcessor::Initialize(AudioCodec* codec, bool realtime_chat) {
codec_ = codec;
int ref_num = codec_->input_reference() ? 1 : 0;
std::string input_format;
for (int i = 0; i < channels_ - ref_num; i++) {
for (int i = 0; i < codec_->input_channels() - ref_num; i++) {
input_format.push_back('M');
}
for (int i = 0; i < ref_num; i++) {
input_format.push_back('R');
}
srmodel_list_t *models = esp_srmodel_init("model");
char* ns_model_name = esp_srmodel_filter(models, ESP_NSNET_PREFIX, NULL);
afe_config_t* afe_config = afe_config_init(input_format.c_str(), NULL, AFE_TYPE_VC, AFE_MODE_HIGH_PERF);
afe_config->aec_init = false;
afe_config->aec_mode = AEC_MODE_VOIP_HIGH_PERF;
if (realtime_chat) {
afe_config->aec_init = true;
afe_config->aec_mode = AEC_MODE_VOIP_LOW_COST;
} else {
afe_config->aec_init = false;
}
afe_config->ns_init = true;
afe_config->vad_init = true;
afe_config->vad_mode = VAD_MODE_0;
afe_config->vad_min_noise_ms = 100;
afe_config->ns_model_name = ns_model_name;
afe_config->afe_ns_mode = AFE_NS_MODE_NET;
if (realtime_chat) {
afe_config->vad_init = false;
} else {
afe_config->vad_init = true;
afe_config->vad_mode = VAD_MODE_0;
afe_config->vad_min_noise_ms = 100;
}
afe_config->afe_perferred_core = 1;
afe_config->afe_perferred_priority = 1;
afe_config->agc_init = true;
afe_config->agc_mode = AFE_AGC_MODE_WEBRTC;
afe_config->agc_compression_gain_db = 10;
afe_config->agc_init = false;
afe_config->memory_alloc_mode = AFE_MEMORY_ALLOC_MORE_PSRAM;
afe_iface_ = esp_afe_handle_from_config(afe_config);
@@ -54,15 +64,12 @@ AudioProcessor::~AudioProcessor() {
vEventGroupDelete(event_group_);
}
void AudioProcessor::Input(const std::vector<int16_t>& data) {
input_buffer_.insert(input_buffer_.end(), data.begin(), data.end());
size_t AudioProcessor::GetFeedSize() {
return afe_iface_->get_feed_chunksize(afe_data_) * codec_->input_channels();
}
auto feed_size = afe_iface_->get_feed_chunksize(afe_data_) * channels_;
while (input_buffer_.size() >= feed_size) {
auto chunk = input_buffer_.data();
afe_iface_->feed(afe_data_, chunk);
input_buffer_.erase(input_buffer_.begin(), input_buffer_.begin() + feed_size);
}
void AudioProcessor::Feed(const std::vector<int16_t>& data) {
afe_iface_->feed(afe_data_, data.data());
}
void AudioProcessor::Start() {

View File

@@ -10,28 +10,29 @@
#include <vector>
#include <functional>
#include "audio_codec.h"
class AudioProcessor {
public:
AudioProcessor();
~AudioProcessor();
void Initialize(int channels, bool reference);
void Input(const std::vector<int16_t>& data);
void Initialize(AudioCodec* codec, bool realtime_chat);
void Feed(const std::vector<int16_t>& data);
void Start();
void Stop();
bool IsRunning();
void OnOutput(std::function<void(std::vector<int16_t>&& data)> callback);
void OnVadStateChange(std::function<void(bool speaking)> callback);
size_t GetFeedSize();
private:
EventGroupHandle_t event_group_ = nullptr;
esp_afe_sr_iface_t* afe_iface_ = nullptr;
esp_afe_sr_data_t* afe_data_ = nullptr;
std::vector<int16_t> input_buffer_;
std::function<void(std::vector<int16_t>&& data)> output_callback_;
std::function<void(bool speaking)> vad_state_change_callback_;
int channels_;
bool reference_;
AudioCodec* codec_ = nullptr;
bool is_speaking_ = false;
void AudioProcessorTask();

View File

@@ -30,10 +30,9 @@ WakeWordDetect::~WakeWordDetect() {
vEventGroupDelete(event_group_);
}
void WakeWordDetect::Initialize(int channels, bool reference) {
channels_ = channels;
reference_ = reference;
int ref_num = reference_ ? 1 : 0;
void WakeWordDetect::Initialize(AudioCodec* codec) {
codec_ = codec;
int ref_num = codec_->input_reference() ? 1 : 0;
srmodel_list_t *models = esp_srmodel_init("model");
for (int i = 0; i < models->num; i++) {
@@ -51,14 +50,14 @@ void WakeWordDetect::Initialize(int channels, bool reference) {
}
std::string input_format;
for (int i = 0; i < channels_ - ref_num; i++) {
for (int i = 0; i < codec_->input_channels() - ref_num; i++) {
input_format.push_back('M');
}
for (int i = 0; i < ref_num; i++) {
input_format.push_back('R');
}
afe_config_t* afe_config = afe_config_init(input_format.c_str(), models, AFE_TYPE_SR, AFE_MODE_HIGH_PERF);
afe_config->aec_init = reference_;
afe_config->aec_init = codec_->input_reference();
afe_config->aec_mode = AEC_MODE_SR_HIGH_PERF;
afe_config->afe_perferred_core = 1;
afe_config->afe_perferred_priority = 1;
@@ -92,13 +91,11 @@ bool WakeWordDetect::IsDetectionRunning() {
}
void WakeWordDetect::Feed(const std::vector<int16_t>& data) {
input_buffer_.insert(input_buffer_.end(), data.begin(), data.end());
afe_iface_->feed(afe_data_, data.data());
}
auto feed_size = afe_iface_->get_feed_chunksize(afe_data_) * channels_;
while (input_buffer_.size() >= feed_size) {
afe_iface_->feed(afe_data_, input_buffer_.data());
input_buffer_.erase(input_buffer_.begin(), input_buffer_.begin() + feed_size);
}
size_t WakeWordDetect::GetFeedSize() {
return afe_iface_->get_feed_chunksize(afe_data_) * codec_->input_channels();
}
void WakeWordDetect::AudioDetectionTask() {

View File

@@ -15,18 +15,20 @@
#include <mutex>
#include <condition_variable>
#include "audio_codec.h"
class WakeWordDetect {
public:
WakeWordDetect();
~WakeWordDetect();
void Initialize(int channels, bool reference);
void Initialize(AudioCodec* codec);
void Feed(const std::vector<int16_t>& data);
void OnWakeWordDetected(std::function<void(const std::string& wake_word)> callback);
void StartDetection();
void StopDetection();
bool IsDetectionRunning();
size_t GetFeedSize();
void EncodeWakeWordData();
bool GetWakeWordOpus(std::vector<uint8_t>& opus);
const std::string& GetLastDetectedWakeWord() const { return last_detected_wake_word_; }
@@ -36,11 +38,9 @@ private:
esp_afe_sr_data_t* afe_data_ = nullptr;
char* wakenet_model_ = NULL;
std::vector<std::string> wake_words_;
std::vector<int16_t> input_buffer_;
EventGroupHandle_t event_group_;
std::function<void(const std::string& wake_word)> wake_word_detected_callback_;
int channels_;
bool reference_;
AudioCodec* codec_ = nullptr;
std::string last_detected_wake_word_;
TaskHandle_t wake_word_encode_task_ = nullptr;

337
main/boards/README.md Normal file
View File

@@ -0,0 +1,337 @@
# 自定义开发板指南
本指南介绍如何为小智AI语音聊天机器人项目定制一个新的开发板初始化程序。小智AI支持50多种ESP32系列开发板每个开发板的初始化代码都放在对应的目录下。
## 重要提示
> **警告**: 对于自定义开发板当IO配置与原有开发板不同时切勿直接覆盖原有开发板的配置编译固件。必须创建新的开发板类型或者通过config.json文件中的builds配置不同的name和sdkconfig宏定义来区分。使用 `python scripts/release.py [开发板目录名字]` 来编译打包固件。
>
> 如果直接覆盖原有配置将来OTA升级时您的自定义固件可能会被原有开发板的标准固件覆盖导致您的设备无法正常工作。每个开发板有唯一的标识和对应的固件升级通道保持开发板标识的唯一性非常重要。
## 目录结构
每个开发板的目录结构通常包含以下文件:
- `xxx_board.cc` - 主要的板级初始化代码,实现了板子相关的初始化和功能
- `config.h` - 板级配置文件,定义了硬件管脚映射和其他配置项
- `config.json` - 编译配置,指定目标芯片和特殊的编译选项
- `README.md` - 开发板相关的说明文档
## 定制开发板步骤
### 1. 创建新的开发板目录
首先在`boards/`目录下创建一个新的目录,例如`my-custom-board/`
```bash
mkdir main/boards/my-custom-board
```
### 2. 创建配置文件
#### config.h
`config.h`中定义所有的硬件配置,包括:
- 音频采样率和I2S引脚配置
- 音频编解码芯片地址和I2C引脚配置
- 按钮和LED引脚配置
- 显示屏参数和引脚配置
参考示例来自lichuang-c3-dev
```c
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
// 音频配置
#define AUDIO_INPUT_SAMPLE_RATE 24000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_10
#define AUDIO_I2S_GPIO_WS GPIO_NUM_12
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_8
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_7
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_11
#define AUDIO_CODEC_PA_PIN GPIO_NUM_13
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_0
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_1
#define AUDIO_CODEC_ES8311_ADDR ES8311_CODEC_DEFAULT_ADDR
// 按钮配置
#define BOOT_BUTTON_GPIO GPIO_NUM_9
// 显示屏配置
#define DISPLAY_SPI_SCK_PIN GPIO_NUM_3
#define DISPLAY_SPI_MOSI_PIN GPIO_NUM_5
#define DISPLAY_DC_PIN GPIO_NUM_6
#define DISPLAY_SPI_CS_PIN GPIO_NUM_4
#define DISPLAY_WIDTH 320
#define DISPLAY_HEIGHT 240
#define DISPLAY_MIRROR_X true
#define DISPLAY_MIRROR_Y false
#define DISPLAY_SWAP_XY true
#define DISPLAY_OFFSET_X 0
#define DISPLAY_OFFSET_Y 0
#define DISPLAY_BACKLIGHT_PIN GPIO_NUM_2
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT true
#endif // _BOARD_CONFIG_H_
```
#### config.json
`config.json`中定义编译配置:
```json
{
"target": "esp32s3", // 目标芯片型号: esp32, esp32s3, esp32c3等
"builds": [
{
"name": "my-custom-board", // 开发板名称
"sdkconfig_append": [
// 额外需要的编译配置
"CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y",
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions_8M.csv\""
]
}
]
}
```
### 3. 编写板级初始化代码
创建一个`my_custom_board.cc`文件,实现开发板的所有初始化逻辑。
一个基本的开发板类定义包含以下几个部分:
1. **类定义**:继承自`WifiBoard``ML307Board`
2. **初始化函数**包括I2C、显示屏、按钮、IoT等组件的初始化
3. **虚函数重写**:如`GetAudioCodec()``GetDisplay()``GetBacklight()`
4. **注册开发板**:使用`DECLARE_BOARD`宏注册开发板
```cpp
#include "wifi_board.h"
#include "audio_codecs/es8311_audio_codec.h"
#include "display/lcd_display.h"
#include "application.h"
#include "button.h"
#include "config.h"
#include "iot/thing_manager.h"
#include <esp_log.h>
#include <driver/i2c_master.h>
#include <driver/spi_common.h>
#define TAG "MyCustomBoard"
// 声明字体
LV_FONT_DECLARE(font_puhui_16_4);
LV_FONT_DECLARE(font_awesome_16_4);
class MyCustomBoard : public WifiBoard {
private:
i2c_master_bus_handle_t codec_i2c_bus_;
Button boot_button_;
LcdDisplay* display_;
// I2C初始化
void InitializeI2c() {
i2c_master_bus_config_t i2c_bus_cfg = {
.i2c_port = I2C_NUM_0,
.sda_io_num = AUDIO_CODEC_I2C_SDA_PIN,
.scl_io_num = AUDIO_CODEC_I2C_SCL_PIN,
.clk_source = I2C_CLK_SRC_DEFAULT,
.glitch_ignore_cnt = 7,
.intr_priority = 0,
.trans_queue_depth = 0,
.flags = {
.enable_internal_pullup = 1,
},
};
ESP_ERROR_CHECK(i2c_new_master_bus(&i2c_bus_cfg, &codec_i2c_bus_));
}
// SPI初始化用于显示屏
void InitializeSpi() {
spi_bus_config_t buscfg = {};
buscfg.mosi_io_num = DISPLAY_SPI_MOSI_PIN;
buscfg.miso_io_num = GPIO_NUM_NC;
buscfg.sclk_io_num = DISPLAY_SPI_SCK_PIN;
buscfg.quadwp_io_num = GPIO_NUM_NC;
buscfg.quadhd_io_num = GPIO_NUM_NC;
buscfg.max_transfer_sz = DISPLAY_WIDTH * DISPLAY_HEIGHT * sizeof(uint16_t);
ESP_ERROR_CHECK(spi_bus_initialize(SPI2_HOST, &buscfg, SPI_DMA_CH_AUTO));
}
// 按钮初始化
void InitializeButtons() {
boot_button_.OnClick([this]() {
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
app.ToggleChatState();
});
}
// 显示屏初始化以ST7789为例
void InitializeDisplay() {
esp_lcd_panel_io_handle_t panel_io = nullptr;
esp_lcd_panel_handle_t panel = nullptr;
esp_lcd_panel_io_spi_config_t io_config = {};
io_config.cs_gpio_num = DISPLAY_SPI_CS_PIN;
io_config.dc_gpio_num = DISPLAY_DC_PIN;
io_config.spi_mode = 2;
io_config.pclk_hz = 80 * 1000 * 1000;
io_config.trans_queue_depth = 10;
io_config.lcd_cmd_bits = 8;
io_config.lcd_param_bits = 8;
ESP_ERROR_CHECK(esp_lcd_new_panel_io_spi(SPI2_HOST, &io_config, &panel_io));
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.reset_gpio_num = GPIO_NUM_NC;
panel_config.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_RGB;
panel_config.bits_per_pixel = 16;
ESP_ERROR_CHECK(esp_lcd_new_panel_st7789(panel_io, &panel_config, &panel));
esp_lcd_panel_reset(panel);
esp_lcd_panel_init(panel);
esp_lcd_panel_invert_color(panel, true);
esp_lcd_panel_swap_xy(panel, DISPLAY_SWAP_XY);
esp_lcd_panel_mirror(panel, DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y);
// 创建显示屏对象
display_ = new SpiLcdDisplay(panel_io, panel,
DISPLAY_WIDTH, DISPLAY_HEIGHT,
DISPLAY_OFFSET_X, DISPLAY_OFFSET_Y,
DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y, DISPLAY_SWAP_XY,
{
.text_font = &font_puhui_16_4,
.icon_font = &font_awesome_16_4,
.emoji_font = font_emoji_32_init(),
});
}
// IoT设备初始化
void InitializeIot() {
auto& thing_manager = iot::ThingManager::GetInstance();
thing_manager.AddThing(iot::CreateThing("Speaker"));
thing_manager.AddThing(iot::CreateThing("Screen"));
// 可以添加更多IoT设备
}
public:
// 构造函数
MyCustomBoard() : boot_button_(BOOT_BUTTON_GPIO) {
InitializeI2c();
InitializeSpi();
InitializeDisplay();
InitializeButtons();
InitializeIot();
GetBacklight()->SetBrightness(100);
}
// 获取音频编解码器
virtual AudioCodec* GetAudioCodec() override {
static Es8311AudioCodec audio_codec(
codec_i2c_bus_,
I2C_NUM_0,
AUDIO_INPUT_SAMPLE_RATE,
AUDIO_OUTPUT_SAMPLE_RATE,
AUDIO_I2S_GPIO_MCLK,
AUDIO_I2S_GPIO_BCLK,
AUDIO_I2S_GPIO_WS,
AUDIO_I2S_GPIO_DOUT,
AUDIO_I2S_GPIO_DIN,
AUDIO_CODEC_PA_PIN,
AUDIO_CODEC_ES8311_ADDR);
return &audio_codec;
}
// 获取显示屏
virtual Display* GetDisplay() override {
return display_;
}
// 获取背光控制
virtual Backlight* GetBacklight() override {
static PwmBacklight backlight(DISPLAY_BACKLIGHT_PIN, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
return &backlight;
}
};
// 注册开发板
DECLARE_BOARD(MyCustomBoard);
```
### 4. 创建README.md
在README.md中说明开发板的特性、硬件要求、编译和烧录步骤
## 常见开发板组件
### 1. 显示屏
项目支持多种显示屏驱动,包括:
- ST7789 (SPI)
- ILI9341 (SPI)
- SH8601 (QSPI)
- 等...
### 2. 音频编解码器
支持的编解码器包括:
- ES8311 (常用)
- ES7210 (麦克风阵列)
- AW88298 (功放)
- 等...
### 3. 电源管理
一些开发板使用电源管理芯片:
- AXP2101
- 其他可用的PMIC
### 4. IoT设备
可以添加各种IoT设备让AI能够"看到"和控制:
- Speaker (扬声器)
- Screen (屏幕)
- Battery (电池)
- Light (灯光)
- 等...
## 开发板类继承关系
- `Board` - 基础板级类
- `WifiBoard` - WiFi连接的开发板
- `ML307Board` - 使用4G模块的开发板
## 开发技巧
1. **参考相似的开发板**:如果您的新开发板与现有开发板有相似之处,可以参考现有实现
2. **分步调试**:先实现基础功能(如显示),再添加更复杂的功能(如音频)
3. **管脚映射**确保在config.h中正确配置所有管脚映射
4. **检查硬件兼容性**:确认所有芯片和驱动程序的兼容性
## 可能遇到的问题
1. **显示屏不正常**检查SPI配置、镜像设置和颜色反转设置
2. **音频无输出**检查I2S配置、PA使能引脚和编解码器地址
3. **无法连接网络**检查WiFi凭据和网络配置
4. **无法与服务器通信**检查MQTT或WebSocket配置
## 参考资料
- ESP-IDF 文档: https://docs.espressif.com/projects/esp-idf/
- LVGL 文档: https://docs.lvgl.io/
- ESP-SR 文档: https://github.com/espressif/esp-sr

View File

@@ -171,12 +171,7 @@ private:
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
});
boot_button_.OnPressDown([this]() {
Application::GetInstance().StartListening();
});
boot_button_.OnPressUp([this]() {
Application::GetInstance().StopListening();
app.ToggleChatState();
});
}

View File

@@ -3,7 +3,9 @@
"builds": [
{
"name": "atk-dnesp32s3-box",
"sdkconfig_append": []
"sdkconfig_append": [
"CONFIG_USE_WECHAT_MESSAGE_STYLE=y"
]
}
]
}

View File

@@ -91,12 +91,7 @@ private:
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
});
boot_button_.OnPressDown([this]() {
Application::GetInstance().StartListening();
});
boot_button_.OnPressUp([this]() {
Application::GetInstance().StopListening();
app.ToggleChatState();
});
}

View File

@@ -195,11 +195,6 @@ public:
}
virtual Led* GetLed() override {
static SingleLed led(BUILTIN_LED_GPIO);
return &led;
}
virtual AudioCodec* GetAudioCodec() override {
#ifdef AUDIO_I2S_METHOD_SIMPLEX
static NoAudioCodecSimplex audio_codec(AUDIO_INPUT_SAMPLE_RATE, AUDIO_OUTPUT_SAMPLE_RATE,

View File

@@ -145,7 +145,7 @@
#define DISPLAY_OFFSET_X 0
#define DISPLAY_OFFSET_Y 0
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT false
#define DISPLAY_SPI_MODE 2
#define DISPLAY_SPI_MODE 3
#endif
#ifdef CONFIG_LCD_ST7789_240X135

View File

@@ -15,6 +15,10 @@
#include <esp_lcd_panel_ops.h>
#include <esp_lcd_panel_vendor.h>
#ifdef SH1106
#include <esp_lcd_panel_sh1106.h>
#endif
#define TAG "CompactWifiBoard"
LV_FONT_DECLARE(font_puhui_14_1);
@@ -76,7 +80,11 @@ private:
};
panel_config.vendor_config = &ssd1306_config;
#ifdef SH1106
ESP_ERROR_CHECK(esp_lcd_new_panel_sh1106(panel_io_, &panel_config, &panel_));
#else
ESP_ERROR_CHECK(esp_lcd_new_panel_ssd1306(panel_io_, &panel_config, &panel_));
#endif
ESP_LOGI(TAG, "SSD1306 driver installed");
// Reset the display

View File

@@ -42,6 +42,9 @@
#define DISPLAY_HEIGHT 32
#elif CONFIG_OLED_SSD1306_128X64
#define DISPLAY_HEIGHT 64
#elif CONFIG_OLED_SH1106_128X64
#define DISPLAY_HEIGHT 64
#define SH1106
#else
#error "未选择 OLED 屏幕类型"
#endif

View File

@@ -0,0 +1,52 @@
#include "knob.h"
static const char* TAG = "Knob";
Knob::Knob(gpio_num_t pin_a, gpio_num_t pin_b) {
knob_config_t config = {
.default_direction = 0,
.gpio_encoder_a = static_cast<uint8_t>(pin_a),
.gpio_encoder_b = static_cast<uint8_t>(pin_b),
};
esp_err_t err = ESP_OK;
knob_handle_ = iot_knob_create(&config);
if (knob_handle_ == NULL) {
ESP_LOGE(TAG, "Failed to create knob instance");
return;
}
err = iot_knob_register_cb(knob_handle_, KNOB_LEFT, knob_callback, this);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to register left callback: %s", esp_err_to_name(err));
return;
}
err = iot_knob_register_cb(knob_handle_, KNOB_RIGHT, knob_callback, this);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to register right callback: %s", esp_err_to_name(err));
return;
}
ESP_LOGI(TAG, "Knob initialized with pins A:%d B:%d", pin_a, pin_b);
}
Knob::~Knob() {
if (knob_handle_ != NULL) {
iot_knob_delete(knob_handle_);
knob_handle_ = NULL;
}
}
void Knob::OnRotate(std::function<void(bool)> callback) {
on_rotate_ = callback;
}
void Knob::knob_callback(void* arg, void* data) {
Knob* knob = static_cast<Knob*>(data);
knob_event_t event = iot_knob_get_event(arg);
if (knob->on_rotate_) {
knob->on_rotate_(event == KNOB_RIGHT);
}
}

25
main/boards/common/knob.h Normal file
View File

@@ -0,0 +1,25 @@
#ifndef KNOB_H_
#define KNOB_H_
#include <driver/gpio.h>
#include <functional>
#include <esp_log.h>
#include <iot_knob.h>
class Knob {
public:
Knob(gpio_num_t pin_a, gpio_num_t pin_b);
~Knob();
void OnRotate(std::function<void(bool)> callback);
private:
static void knob_callback(void* arg, void* data);
knob_handle_t knob_handle_;
gpio_num_t pin_a_;
gpio_num_t pin_b_;
std::function<void(bool)> on_rotate_;
};
#endif // KNOB_H_

View File

@@ -126,7 +126,11 @@ private:
{
.text_font = &font_puhui_20_4,
.icon_font = &font_awesome_20_4,
#if CONFIG_USE_WECHAT_MESSAGE_STYLE
.emoji_font = font_emoji_32_init(),
#else
.emoji_font = font_emoji_64_init(),
#endif
});
}

View File

@@ -3,7 +3,9 @@
"builds": [
{
"name": "esp32-s3-touch-amoled-1.8",
"sdkconfig_append": []
"sdkconfig_append": [
"CONFIG_USE_WECHAT_MESSAGE_STYLE=y"
]
}
]
}

View File

@@ -87,7 +87,11 @@ public:
{
.text_font = &font_puhui_30_4,
.icon_font = &font_awesome_30_4,
#if CONFIG_USE_WECHAT_MESSAGE_STYLE
.emoji_font = font_emoji_32_init(),
#else
.emoji_font = font_emoji_64_init(),
#endif
}) {
DisplayLockGuard lock(this);
lv_obj_set_style_pad_left(status_bar_, LV_HOR_RES * 0.1, 0);

View File

@@ -114,7 +114,7 @@ private:
ESP_ERROR_CHECK(spi_bus_initialize(QSPI_LCD_HOST, &bus_config, SPI_DMA_CH_AUTO));
}
void Initializespd2010Display() {
void InitializeSpd2010Display() {
esp_lcd_panel_io_handle_t panel_io = nullptr;
esp_lcd_panel_handle_t panel = nullptr;
@@ -224,7 +224,7 @@ public:
InitializeI2c();
InitializeTca9554();
InitializeSpi();
Initializespd2010Display();
InitializeSpd2010Display();
InitializeButtons();
InitializeIot();
GetBacklight()->RestoreBrightness();

View File

@@ -0,0 +1,3 @@
新增 微雪 开发板: ESP32-S3-Touch-LCD-3.5
产品链接:
https://www.waveshare.net/shop/ESP32-S3-Touch-LCD-3.5.htm

View File

@@ -0,0 +1,31 @@
#include <freertos/FreeRTOS.h>
#include <freertos/timers.h>
#include <freertos/task.h>
#include <esp_log.h>
#include "board.h"
#include "boards/common/wifi_board.h"
#include "iot/thing.h"
#define TAG "BoardControl"
namespace iot {
class BoardControl : public Thing {
public:
BoardControl() : Thing("BoardControl", "当前 AI 机器人管理和控制") {
// 修改重新配网
methods_.AddMethod("ResetWifiConfiguration", "重新配网", ParameterList(),
[this](const ParameterList& parameters) {
ESP_LOGI(TAG, "ResetWifiConfiguration");
auto board = static_cast<WifiBoard*>(&Board::GetInstance());
if (board && board->GetBoardType() == "wifi") {
board->ResetWifiConfiguration();
}
});
}
};
} // namespace iot
DECLARE_THING(BoardControl);

View File

@@ -0,0 +1,50 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
#include <driver/spi_master.h>
#define AUDIO_INPUT_SAMPLE_RATE 24000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_12
#define AUDIO_I2S_GPIO_WS GPIO_NUM_15
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_13
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_14
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_16
#define AUDIO_CODEC_PA_PIN GPIO_NUM_NC
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_8
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_7
#define AUDIO_CODEC_ES8311_ADDR ES8311_CODEC_DEFAULT_ADDR
#define BUILTIN_LED_GPIO GPIO_NUM_NC
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define VOLUME_UP_BUTTON_GPIO GPIO_NUM_NC
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_NC
#define DISPLAY_SPI_MODE 0
#define DISPLAY_CS_PIN GPIO_NUM_NC
#define DISPLAY_MOSI_PIN GPIO_NUM_1
#define DISPLAY_MISO_PIN GPIO_NUM_2
#define DISPLAY_CLK_PIN GPIO_NUM_5
#define DISPLAY_DC_PIN GPIO_NUM_3
#define DISPLAY_RST_PIN GPIO_NUM_NC
#define DISPLAY_WIDTH 480
#define DISPLAY_HEIGHT 320
#define DISPLAY_MIRROR_X false
#define DISPLAY_MIRROR_Y false
#define DISPLAY_SWAP_XY true
#define DISPLAY_RGB_ORDER LCD_RGB_ELEMENT_ORDER_BGR
#define DISPLAY_INVERT_COLOR true
#define DISPLAY_OFFSET_X 0
#define DISPLAY_OFFSET_Y 0
#define DISPLAY_BACKLIGHT_PIN GPIO_NUM_6
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT false
#endif // _BOARD_CONFIG_H_

View File

@@ -0,0 +1,9 @@
{
"target": "esp32s3",
"builds": [
{
"name": "esp32-s3-touch-lcd-3.5",
"sdkconfig_append": []
}
]
}

View File

@@ -0,0 +1,295 @@
#include "wifi_board.h"
#include "audio_codecs/es8311_audio_codec.h"
#include "display/lcd_display.h"
#include "system_reset.h"
#include "application.h"
#include "button.h"
#include "config.h"
#include "iot/thing_manager.h"
#include <esp_log.h>
#include "i2c_device.h"
#include <driver/i2c.h>
#include <driver/ledc.h>
#include <wifi_station.h>
#include <esp_lcd_panel_vendor.h>
#include <esp_lcd_panel_io.h>
#include <esp_lcd_panel_ops.h>
#include <esp_timer.h>
#include "esp_io_expander_tca9554.h"
#include "axp2101.h"
#include "power_save_timer.h"
#define TAG "waveshare_lcd_3_5"
LV_FONT_DECLARE(font_puhui_16_4);
LV_FONT_DECLARE(font_awesome_16_4);
class Pmic : public Axp2101 {
public:
Pmic(i2c_master_bus_handle_t i2c_bus, uint8_t addr) : Axp2101(i2c_bus, addr) {
WriteReg(0x22, 0b110); // PWRON > OFFLEVEL as POWEROFF Source enable
WriteReg(0x27, 0x10); // hold 4s to power off
// Disable All DCs but DC1
WriteReg(0x80, 0x01);
// Disable All LDOs
WriteReg(0x90, 0x00);
WriteReg(0x91, 0x00);
// Set DC1 to 3.3V
WriteReg(0x82, (3300 - 1500) / 100);
// Set ALDO1 to 3.3V
WriteReg(0x92, (3300 - 500) / 100);
// Enable ALDO1(MIC)
WriteReg(0x90, 0x01);
WriteReg(0x64, 0x02); // CV charger voltage setting to 4.1V
WriteReg(0x61, 0x02); // set Main battery precharge current to 50mA
WriteReg(0x62, 0x08); // set Main battery charger current to 400mA ( 0x08-200mA, 0x09-300mA, 0x0A-400mA )
WriteReg(0x63, 0x01); // set Main battery term charge current to 25mA
}
};
typedef struct {
int cmd; /*<! The specific LCD command */
const void *data; /*<! Buffer that holds the command specific data */
size_t data_bytes; /*<! Size of `data` in memory, in bytes */
unsigned int delay_ms; /*<! Delay in milliseconds after this command */
} st7796_lcd_init_cmd_t;
typedef struct {
const st7796_lcd_init_cmd_t *init_cmds; /*!< Pointer to initialization commands array. Set to NULL if using default commands.
* The array should be declared as `static const` and positioned outside the function.
* Please refer to `vendor_specific_init_default` in source file.
*/
uint16_t init_cmds_size; /*<! Number of commands in above array */
} st7796_vendor_config_t;
st7796_lcd_init_cmd_t st7796_lcd_init_cmds[] = {
{0x11, (uint8_t []){ 0x00 }, 0, 120},
// {0x36, (uint8_t []){ 0x08 }, 1, 0},
{0x3A, (uint8_t []){ 0x05 }, 1, 0},
{0xF0, (uint8_t []){ 0xC3 }, 1, 0},
{0xF0, (uint8_t []){ 0x96 }, 1, 0},
{0xB4, (uint8_t []){ 0x01 }, 1, 0},
{0xB7, (uint8_t []){ 0xC6 }, 1, 0},
{0xC0, (uint8_t []){ 0x80, 0x45 }, 2, 0},
{0xC1, (uint8_t []){ 0x13 }, 1, 0},
{0xC2, (uint8_t []){ 0xA7 }, 1, 0},
{0xC5, (uint8_t []){ 0x0A }, 1, 0},
{0xE8, (uint8_t []){ 0x40, 0x8A, 0x00, 0x00, 0x29, 0x19, 0xA5, 0x33}, 8, 0},
{0xE0, (uint8_t []){ 0xD0, 0x08, 0x0F, 0x06, 0x06, 0x33, 0x30, 0x33, 0x47, 0x17, 0x13, 0x13, 0x2B, 0x31}, 14, 0},
{0xE1, (uint8_t []){ 0xD0, 0x0A, 0x11, 0x0B, 0x09, 0x07, 0x2F, 0x33, 0x47, 0x38, 0x15, 0x16, 0x2C, 0x32},14, 0},
{0xF0, (uint8_t []){ 0x3C }, 1, 0},
{0xF0, (uint8_t []){ 0x69 }, 1, 120},
{0x21, (uint8_t []){ 0x00 }, 0, 0},
{0x29, (uint8_t []){ 0x00 }, 0, 0},
};
class CustomBoard : public WifiBoard {
private:
Button boot_button_;
Pmic* pmic_ = nullptr;
i2c_master_bus_handle_t i2c_bus_;
esp_io_expander_handle_t io_expander = NULL;
LcdDisplay* display_;
PowerSaveTimer* power_save_timer_;
void InitializePowerSaveTimer() {
power_save_timer_ = new PowerSaveTimer(-1, 60, 300);
power_save_timer_->OnEnterSleepMode([this]() {
ESP_LOGI(TAG, "Enabling sleep mode");
auto display = GetDisplay();
display->SetChatMessage("system", "");
display->SetEmotion("sleepy");
GetBacklight()->SetBrightness(20);
});
power_save_timer_->OnExitSleepMode([this]() {
auto display = GetDisplay();
display->SetChatMessage("system", "");
display->SetEmotion("neutral");
GetBacklight()->RestoreBrightness();
});
power_save_timer_->OnShutdownRequest([this]() {
pmic_->PowerOff();
});
power_save_timer_->SetEnabled(true);
}
void InitializeI2c() {
// Initialize I2C peripheral
i2c_master_bus_config_t i2c_bus_cfg = {
.i2c_port = (i2c_port_t)0,
.sda_io_num = AUDIO_CODEC_I2C_SDA_PIN,
.scl_io_num = AUDIO_CODEC_I2C_SCL_PIN,
.clk_source = I2C_CLK_SRC_DEFAULT,
};
ESP_ERROR_CHECK(i2c_new_master_bus(&i2c_bus_cfg, &i2c_bus_));
}
void InitializeTca9554(void)
{
esp_err_t ret = esp_io_expander_new_i2c_tca9554(i2c_bus_, ESP_IO_EXPANDER_I2C_TCA9554_ADDRESS_000, &io_expander);
if(ret != ESP_OK)
ESP_LOGE(TAG, "TCA9554 create returned error");
ret = esp_io_expander_set_dir(io_expander, IO_EXPANDER_PIN_NUM_0 | IO_EXPANDER_PIN_NUM_1, IO_EXPANDER_OUTPUT);
ESP_ERROR_CHECK(ret);
vTaskDelay(pdMS_TO_TICKS(100));
ret = esp_io_expander_set_level(io_expander, IO_EXPANDER_PIN_NUM_1, 0);
ESP_ERROR_CHECK(ret);
vTaskDelay(pdMS_TO_TICKS(100));
ret = esp_io_expander_set_level(io_expander, IO_EXPANDER_PIN_NUM_0 | IO_EXPANDER_PIN_NUM_1, 1);
ESP_ERROR_CHECK(ret);
}
void InitializeAxp2101() {
ESP_LOGI(TAG, "Init AXP2101");
pmic_ = new Pmic(i2c_bus_, 0x34);
}
void InitializeSpi() {
ESP_LOGI(TAG, "Initialize QSPI bus");
spi_bus_config_t buscfg = {};
buscfg.mosi_io_num = DISPLAY_MOSI_PIN;
buscfg.miso_io_num = DISPLAY_MISO_PIN;
buscfg.sclk_io_num = DISPLAY_CLK_PIN;
buscfg.quadwp_io_num = GPIO_NUM_NC;
buscfg.quadhd_io_num = GPIO_NUM_NC;
buscfg.max_transfer_sz = DISPLAY_WIDTH * DISPLAY_HEIGHT * sizeof(uint16_t);
ESP_ERROR_CHECK(spi_bus_initialize(SPI3_HOST, &buscfg, SPI_DMA_CH_AUTO));
}
void InitializeLcdDisplay() {
esp_lcd_panel_io_handle_t panel_io = nullptr;
esp_lcd_panel_handle_t panel = nullptr;
// 液晶屏控制IO初始化
ESP_LOGI(TAG, "Install panel IO");
esp_lcd_panel_io_spi_config_t io_config = {};
io_config.cs_gpio_num = DISPLAY_CS_PIN;
io_config.dc_gpio_num = DISPLAY_DC_PIN;
io_config.spi_mode = DISPLAY_SPI_MODE;
io_config.pclk_hz = 40 * 1000 * 1000;
io_config.trans_queue_depth = 10;
io_config.lcd_cmd_bits = 8;
io_config.lcd_param_bits = 8;
ESP_ERROR_CHECK(esp_lcd_new_panel_io_spi(SPI3_HOST, &io_config, &panel_io));
st7796_vendor_config_t st7796_vendor_config = {
.init_cmds = st7796_lcd_init_cmds,
.init_cmds_size = sizeof(st7796_lcd_init_cmds) / sizeof(st7796_lcd_init_cmd_t),
};
// 初始化液晶屏驱动芯片
ESP_LOGI(TAG, "Install LCD driver");
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.reset_gpio_num = DISPLAY_RST_PIN;
panel_config.rgb_ele_order = DISPLAY_RGB_ORDER;
panel_config.bits_per_pixel = 16;
panel_config.vendor_config = &st7796_vendor_config;
ESP_ERROR_CHECK(esp_lcd_new_panel_st7789(panel_io, &panel_config, &panel));
esp_lcd_panel_reset(panel);
esp_lcd_panel_init(panel);
esp_lcd_panel_invert_color(panel, DISPLAY_INVERT_COLOR);
esp_lcd_panel_swap_xy(panel, DISPLAY_SWAP_XY);
esp_lcd_panel_mirror(panel, DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y);
display_ = new SpiLcdDisplay(panel_io, panel,
DISPLAY_WIDTH, DISPLAY_HEIGHT, DISPLAY_OFFSET_X, DISPLAY_OFFSET_Y, DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y, DISPLAY_SWAP_XY,
{
.text_font = &font_puhui_16_4,
.icon_font = &font_awesome_16_4,
.emoji_font = font_emoji_32_init(),
});
}
void InitializeButtons() {
boot_button_.OnClick([this]() {
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
app.ToggleChatState();
});
}
// 物联网初始化,添加对 AI 可见设备
void InitializeIot() {
auto& thing_manager = iot::ThingManager::GetInstance();
thing_manager.AddThing(iot::CreateThing("Speaker"));
thing_manager.AddThing(iot::CreateThing("Screen"));
thing_manager.AddThing(iot::CreateThing("Battery"));
thing_manager.AddThing(iot::CreateThing("BoardControl"));
}
public:
CustomBoard() :
boot_button_(BOOT_BUTTON_GPIO) {
InitializePowerSaveTimer();
InitializeI2c();
InitializeTca9554();
InitializeAxp2101();
InitializeSpi();
InitializeLcdDisplay();
InitializeButtons();
InitializeIot();
GetBacklight()->RestoreBrightness();
}
virtual AudioCodec* GetAudioCodec() override {
static Es8311AudioCodec audio_codec(i2c_bus_, I2C_NUM_0, AUDIO_INPUT_SAMPLE_RATE, AUDIO_OUTPUT_SAMPLE_RATE,
AUDIO_I2S_GPIO_MCLK, AUDIO_I2S_GPIO_BCLK, AUDIO_I2S_GPIO_WS, AUDIO_I2S_GPIO_DOUT, AUDIO_I2S_GPIO_DIN,
AUDIO_CODEC_PA_PIN, AUDIO_CODEC_ES8311_ADDR);
return &audio_codec;
}
virtual Display* GetDisplay() override {
return display_;
}
virtual Backlight* GetBacklight() override {
static PwmBacklight backlight(DISPLAY_BACKLIGHT_PIN, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
return &backlight;
}
virtual bool GetBatteryLevel(int &level, bool& charging, bool& discharging) override {
static bool last_discharging = false;
charging = pmic_->IsCharging();
discharging = pmic_->IsDischarging();
if (discharging != last_discharging) {
power_save_timer_->SetEnabled(discharging);
last_discharging = discharging;
}
level = pmic_->GetBatteryLevel();
return true;
}
virtual void SetPowerSaveMode(bool enabled) override {
if (!enabled) {
power_save_timer_->WakeUp();
}
WifiBoard::SetPowerSaveMode(enabled);
}
};
DECLARE_BOARD(CustomBoard);

View File

@@ -136,12 +136,7 @@ private:
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
});
boot_button_.OnPressDown([this]() {
Application::GetInstance().StartListening();
});
boot_button_.OnPressUp([this]() {
Application::GetInstance().StopListening();
app.ToggleChatState();
});
}

View File

@@ -78,12 +78,7 @@ private:
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
});
boot_button_.OnPressDown([this]() {
Application::GetInstance().StartListening();
});
boot_button_.OnPressUp([this]() {
Application::GetInstance().StopListening();
app.ToggleChatState();
});
}

View File

@@ -16,6 +16,9 @@
#include <driver/spi_common.h>
#include <esp_lcd_nv3023.h>
#include "../magiclick-2p5/power_manager.h"
#include "power_save_timer.h"
#define TAG "magiclick_2p4"
LV_FONT_DECLARE(font_puhui_16_4);
@@ -64,6 +67,41 @@ private:
Button right_button_;
NV3023Display* display_;
PowerSaveTimer* power_save_timer_;
PowerManager* power_manager_;
esp_lcd_panel_io_handle_t panel_io = nullptr;
esp_lcd_panel_handle_t panel = nullptr;
void InitializePowerManager() {
power_manager_ = new PowerManager(GPIO_NUM_48);
power_manager_->OnChargingStatusChanged([this](bool is_charging) {
if (is_charging) {
power_save_timer_->SetEnabled(false);
} else {
power_save_timer_->SetEnabled(true);
}
});
}
void InitializePowerSaveTimer() {
power_save_timer_ = new PowerSaveTimer(240, 60, -1);
power_save_timer_->OnEnterSleepMode([this]() {
ESP_LOGI(TAG, "Enabling sleep mode");
display_->SetChatMessage("system", "");
display_->SetEmotion("sleepy");
GetBacklight()->SetBrightness(1);
});
power_save_timer_->OnExitSleepMode([this]() {
display_->SetChatMessage("system", "");
display_->SetEmotion("neutral");
GetBacklight()->RestoreBrightness();
});
power_save_timer_->SetEnabled(true);
}
void InitializeCodecI2c() {
// Initialize I2C peripheral
i2c_master_bus_config_t i2c_bus_cfg = {
@@ -82,7 +120,14 @@ private:
}
void InitializeButtons() {
main_button_.OnClick([this]() {
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
});
main_button_.OnPressDown([this]() {
power_save_timer_->WakeUp();
Application::GetInstance().StartListening();
});
main_button_.OnPressUp([this]() {
@@ -90,6 +135,7 @@ private:
});
left_button_.OnClick([this]() {
power_save_timer_->WakeUp();
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
@@ -104,11 +150,13 @@ private:
});
left_button_.OnLongPress([this]() {
power_save_timer_->WakeUp();
GetAudioCodec()->SetOutputVolume(0);
GetDisplay()->ShowNotification(Lang::Strings::MUTED);
});
right_button_.OnClick([this]() {
power_save_timer_->WakeUp();
auto codec = GetAudioCodec();
auto volume = codec->output_volume() + 10;
if (volume > 100) {
@@ -119,6 +167,7 @@ private:
});
right_button_.OnLongPress([this]() {
power_save_timer_->WakeUp();
GetAudioCodec()->SetOutputVolume(100);
GetDisplay()->ShowNotification(Lang::Strings::MAX_VOLUME);
});
@@ -143,8 +192,8 @@ private:
}
void InitializeNv3023Display(){
esp_lcd_panel_io_handle_t panel_io = nullptr;
esp_lcd_panel_handle_t panel = nullptr;
// esp_lcd_panel_io_handle_t panel_io = nullptr;
// esp_lcd_panel_handle_t panel = nullptr;
// 液晶屏控制IO初始化
ESP_LOGD(TAG, "Install panel IO");
esp_lcd_panel_io_spi_config_t io_config = {};
@@ -188,13 +237,16 @@ public:
main_button_(MAIN_BUTTON_GPIO),
left_button_(LEFT_BUTTON_GPIO),
right_button_(RIGHT_BUTTON_GPIO) {
InitializeLedPower();
InitializePowerManager();
InitializePowerSaveTimer();
InitializeCodecI2c();
InitializeButtons();
InitializeLedPower();
InitializeSpi();
InitializeNv3023Display();
InitializeIot();
GetBacklight()->RestoreBrightness();
}
virtual Led* GetLed() override {
@@ -217,6 +269,25 @@ public:
static PwmBacklight backlight(DISPLAY_BACKLIGHT_PIN, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
return &backlight;
}
virtual bool GetBatteryLevel(int& level, bool& charging, bool& discharging) override {
static bool last_discharging = false;
charging = power_manager_->IsCharging();
discharging = power_manager_->IsDischarging();
if (discharging != last_discharging) {
power_save_timer_->SetEnabled(discharging);
last_discharging = discharging;
}
level = power_manager_->GetBatteryLevel();
return true;
}
virtual void SetPowerSaveMode(bool enabled) override {
if (!enabled) {
power_save_timer_->WakeUp();
}
WifiBoard::SetPowerSaveMode(enabled);
}
};
DECLARE_BOARD(magiclick_2p4);

View File

@@ -3,7 +3,10 @@
"builds": [
{
"name": "magiclick-2p5",
"sdkconfig_append": []
"sdkconfig_append": [
"CONFIG_PM_ENABLE=y",
"CONFIG_FREERTOS_USE_TICKLESS_IDLE=y"
]
}
]
}

View File

@@ -18,6 +18,9 @@
#include <esp_lcd_panel_ops.h>
#include <esp_lcd_gc9a01.h>
#include "power_manager.h"
#include "power_save_timer.h"
#define TAG "magiclick_2p5"
LV_FONT_DECLARE(font_puhui_16_4);
@@ -33,28 +36,6 @@ public:
.icon_font = &font_awesome_16_4,
.emoji_font = font_emoji_32_init(),
}) {
DisplayLockGuard lock(this);
// 只需要覆盖颜色相关的样式
auto screen = lv_disp_get_scr_act(lv_disp_get_default());
lv_obj_set_style_text_color(screen, lv_color_black(), 0);
// 设置容器背景色
lv_obj_set_style_bg_color(container_, lv_color_black(), 0);
// 设置状态栏背景色和文本颜色
lv_obj_set_style_bg_color(status_bar_, lv_color_white(), 0);
lv_obj_set_style_text_color(network_label_, lv_color_black(), 0);
lv_obj_set_style_text_color(notification_label_, lv_color_black(), 0);
lv_obj_set_style_text_color(status_label_, lv_color_black(), 0);
lv_obj_set_style_text_color(mute_label_, lv_color_black(), 0);
lv_obj_set_style_text_color(battery_label_, lv_color_black(), 0);
// 设置内容区背景色和文本颜色
lv_obj_set_style_bg_color(content_, lv_color_black(), 0);
lv_obj_set_style_border_width(content_, 0, 0);
lv_obj_set_style_text_color(emotion_label_, lv_color_white(), 0);
lv_obj_set_style_text_color(chat_message_label_, lv_color_white(), 0);
}
};
@@ -100,6 +81,41 @@ private:
Button right_button_;
GC9107Display* display_;
PowerSaveTimer* power_save_timer_;
PowerManager* power_manager_;
esp_lcd_panel_io_handle_t panel_io = nullptr;
esp_lcd_panel_handle_t panel = nullptr;
void InitializePowerManager() {
power_manager_ = new PowerManager(GPIO_NUM_48);
power_manager_->OnChargingStatusChanged([this](bool is_charging) {
if (is_charging) {
power_save_timer_->SetEnabled(false);
} else {
power_save_timer_->SetEnabled(true);
}
});
}
void InitializePowerSaveTimer() {
power_save_timer_ = new PowerSaveTimer(240, 60, -1);
power_save_timer_->OnEnterSleepMode([this]() {
ESP_LOGI(TAG, "Enabling sleep mode");
display_->SetChatMessage("system", "");
display_->SetEmotion("sleepy");
GetBacklight()->SetBrightness(1);
});
power_save_timer_->OnExitSleepMode([this]() {
display_->SetChatMessage("system", "");
display_->SetEmotion("neutral");
GetBacklight()->RestoreBrightness();
});
power_save_timer_->SetEnabled(true);
}
void InitializeCodecI2c() {
// Initialize I2C peripheral
i2c_master_bus_config_t i2c_bus_cfg = {
@@ -118,7 +134,14 @@ private:
}
void InitializeButtons() {
main_button_.OnClick([this]() {
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
});
main_button_.OnPressDown([this]() {
power_save_timer_->WakeUp();
Application::GetInstance().StartListening();
});
main_button_.OnPressUp([this]() {
@@ -126,6 +149,7 @@ private:
});
left_button_.OnClick([this]() {
power_save_timer_->WakeUp();
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
@@ -140,11 +164,13 @@ private:
});
left_button_.OnLongPress([this]() {
power_save_timer_->WakeUp();
GetAudioCodec()->SetOutputVolume(0);
GetDisplay()->ShowNotification(Lang::Strings::MUTED);
});
right_button_.OnClick([this]() {
power_save_timer_->WakeUp();
auto codec = GetAudioCodec();
auto volume = codec->output_volume() + 10;
if (volume > 100) {
@@ -155,11 +181,10 @@ private:
});
right_button_.OnLongPress([this]() {
power_save_timer_->WakeUp();
GetAudioCodec()->SetOutputVolume(100);
GetDisplay()->ShowNotification(Lang::Strings::MAX_VOLUME);
});
}
void InitializeLedPower() {
@@ -181,8 +206,8 @@ private:
}
void InitializeGc9107Display(){
esp_lcd_panel_io_handle_t panel_io = nullptr;
esp_lcd_panel_handle_t panel = nullptr;
// esp_lcd_panel_io_handle_t panel_io = nullptr;
// esp_lcd_panel_handle_t panel = nullptr;
// 液晶屏控制IO初始化
ESP_LOGD(TAG, "Install panel IO");
esp_lcd_panel_io_spi_config_t io_config = {};
@@ -195,7 +220,7 @@ private:
io_config.lcd_param_bits = 8;
ESP_ERROR_CHECK(esp_lcd_new_panel_io_spi(SPI3_HOST, &io_config, &panel_io));
// 初始化液晶屏驱动芯片NV3023
// 初始化液晶屏驱动芯片GC9107
ESP_LOGD(TAG, "Install LCD driver");
gc9a01_vendor_config_t gc9107_vendor_config = {
.init_cmds = gc9107_lcd_init_cmds,
@@ -232,9 +257,11 @@ public:
main_button_(MAIN_BUTTON_GPIO),
left_button_(LEFT_BUTTON_GPIO),
right_button_(RIGHT_BUTTON_GPIO) {
InitializeLedPower();
InitializePowerManager();
InitializePowerSaveTimer();
InitializeCodecI2c();
InitializeButtons();
InitializeLedPower();
InitializeSpi();
InitializeGc9107Display();
InitializeIot();
@@ -261,6 +288,25 @@ public:
static PwmBacklight backlight(DISPLAY_BACKLIGHT_PIN, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
return &backlight;
}
virtual bool GetBatteryLevel(int& level, bool& charging, bool& discharging) override {
static bool last_discharging = false;
charging = power_manager_->IsCharging();
discharging = power_manager_->IsDischarging();
if (discharging != last_discharging) {
power_save_timer_->SetEnabled(discharging);
last_discharging = discharging;
}
level = power_manager_->GetBatteryLevel();
return true;
}
virtual void SetPowerSaveMode(bool enabled) override {
if (!enabled) {
power_save_timer_->WakeUp();
}
WifiBoard::SetPowerSaveMode(enabled);
}
};
DECLARE_BOARD(magiclick_2p5);

View File

@@ -0,0 +1,195 @@
#pragma once
#include <vector>
#include <functional>
#include <esp_timer.h>
#include <driver/gpio.h>
#include <esp_adc/adc_oneshot.h>
#define CHARGING_PIN GPIO_NUM_48
#define CHARGING_ACTIVE_STATE 0
class PowerManager {
private:
esp_timer_handle_t timer_handle_;
std::function<void(bool)> on_charging_status_changed_;
std::function<void(bool)> on_low_battery_status_changed_;
gpio_num_t charging_pin_ = CHARGING_PIN;
std::vector<uint16_t> adc_values_;
uint32_t battery_level_ = 0;
bool is_charging_ = false;
bool is_low_battery_ = false;
int ticks_ = 0;
const int kBatteryAdcInterval = 60;
const int kBatteryAdcDataCount = 3;
const int kLowBatteryLevel = 20;
adc_oneshot_unit_handle_t adc_handle_;
void CheckBatteryStatus() {
// Get charging status
bool new_charging_status = gpio_get_level(charging_pin_) == 0;
// ESP_LOGI("PowerManager", "new_charging_status: %s,is_charging_:%s", new_charging_status?"True":"False",is_charging_?"True":"False");
if (new_charging_status != is_charging_) {
is_charging_ = new_charging_status;
if (on_charging_status_changed_) {
on_charging_status_changed_(is_charging_);
}
ReadBatteryAdcData();
return;
}
// 如果电池电量数据不足,则读取电池电量数据
if (adc_values_.size() < kBatteryAdcDataCount) {
ReadBatteryAdcData();
return;
}
// 如果电池电量数据充足,则每 kBatteryAdcInterval 个 tick 读取一次电池电量数据
ticks_++;
if (ticks_ % kBatteryAdcInterval == 0) {
ReadBatteryAdcData();
}
}
void ReadBatteryAdcData() {
int adc_value;
ESP_ERROR_CHECK(adc_oneshot_read(adc_handle_, ADC_CHANNEL_6, &adc_value));
ESP_LOGI("PowerManager", "ADC value: %d ", adc_value);
// 将 ADC 值添加到队列中
adc_values_.push_back(adc_value);
if (adc_values_.size() > kBatteryAdcDataCount) {
adc_values_.erase(adc_values_.begin());
}
uint32_t average_adc = 0;
for (auto value : adc_values_) {
average_adc += value;
}
average_adc /= adc_values_.size();
// 定义电池电量区间
const struct {
uint16_t adc;
uint8_t level;
} levels[] = {
{1985, 0},
{2079, 20},
{2141, 40},
{2296, 60},
{2420, 80},
{2606, 100}
};
// 低于最低值时
if (average_adc < levels[0].adc) {
battery_level_ = 0;
}
// 高于最高值时
else if (average_adc >= levels[5].adc) {
battery_level_ = 100;
} else {
// 线性插值计算中间值
for (int i = 0; i < 5; i++) {
if (average_adc >= levels[i].adc && average_adc < levels[i+1].adc) {
float ratio = static_cast<float>(average_adc - levels[i].adc) / (levels[i+1].adc - levels[i].adc);
battery_level_ = levels[i].level + ratio * (levels[i+1].level - levels[i].level);
break;
}
}
}
// Check low battery status
if (adc_values_.size() >= kBatteryAdcDataCount) {
bool new_low_battery_status = battery_level_ <= kLowBatteryLevel;
if (new_low_battery_status != is_low_battery_) {
is_low_battery_ = new_low_battery_status;
if (on_low_battery_status_changed_) {
on_low_battery_status_changed_(is_low_battery_);
}
}
}
ESP_LOGI("PowerManager", "ADC value: %d average: %ld level: %ld", adc_value, average_adc, battery_level_);
}
public:
PowerManager(gpio_num_t pin) : charging_pin_(pin) {
// 初始化充电引脚
gpio_config_t io_conf = {};
io_conf.intr_type = GPIO_INTR_DISABLE;
io_conf.mode = GPIO_MODE_INPUT;
io_conf.pin_bit_mask = (1ULL << charging_pin_);
io_conf.pull_down_en = GPIO_PULLDOWN_DISABLE;
io_conf.pull_up_en = GPIO_PULLUP_ENABLE;
gpio_config(&io_conf);
// 创建电池电量检查定时器
esp_timer_create_args_t timer_args = {
.callback = [](void* arg) {
PowerManager* self = static_cast<PowerManager*>(arg);
self->CheckBatteryStatus();
},
.arg = this,
.dispatch_method = ESP_TIMER_TASK,
.name = "battery_check_timer",
.skip_unhandled_events = true,
};
ESP_ERROR_CHECK(esp_timer_create(&timer_args, &timer_handle_));
ESP_ERROR_CHECK(esp_timer_start_periodic(timer_handle_, 1000000));
// 初始化 ADC
adc_oneshot_unit_init_cfg_t init_config = {
.unit_id = ADC_UNIT_1,
.ulp_mode = ADC_ULP_MODE_DISABLE,
};
ESP_ERROR_CHECK(adc_oneshot_new_unit(&init_config, &adc_handle_));
adc_oneshot_chan_cfg_t chan_config = {
.atten = ADC_ATTEN_DB_12,
.bitwidth = ADC_BITWIDTH_12,
};
ESP_ERROR_CHECK(adc_oneshot_config_channel(adc_handle_, ADC_CHANNEL_6, &chan_config));
}
~PowerManager() {
if (timer_handle_) {
esp_timer_stop(timer_handle_);
esp_timer_delete(timer_handle_);
}
if (adc_handle_) {
adc_oneshot_del_unit(adc_handle_);
}
}
bool IsCharging() {
// 检测充电指示引脚
if(gpio_get_level(charging_pin_) != CHARGING_ACTIVE_STATE)
{
return false;
}
return is_charging_;
}
bool IsDischarging() {
// 没有区分充电和放电,所以直接返回相反状态
return !is_charging_;
}
uint8_t GetBatteryLevel() {
return battery_level_;
}
void OnLowBatteryStatusChanged(std::function<void(bool)> callback) {
on_low_battery_status_changed_ = callback;
}
void OnChargingStatusChanged(std::function<void(bool)> callback) {
on_charging_status_changed_ = callback;
}
};

View File

@@ -0,0 +1,44 @@
# ESP32-S3 编译配置指南
## 基本命令
### 设置目标芯片
```bash
idf.py set-target esp32s3
```
### 打开配置界面:
```bash
idf.py menuconfig
```
### Flash 配置:
```
Serial flasher config -> Flash size -> 8 MB
```
### 分区表配置:
```
Partition Table -> Custom partition CSV file -> partitions_8M.csv
```
### 开发板选择:
```
Xiaozhi Assistant -> Board Type -> Movecall CuiCan 璀璨·AI吊坠
```
### 启用编译优化:
```
Component config → Compiler options → Optimization Level → Optimize for size (-Os)
```
### 编译:
```bash
idf.py build
```

View File

@@ -0,0 +1,45 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
// Movecall CuiCan configuration
#include <driver/gpio.h>
#define AUDIO_INPUT_SAMPLE_RATE 24000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_45
#define AUDIO_I2S_GPIO_WS GPIO_NUM_41
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_39
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_40
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_42
#define AUDIO_CODEC_PA_PIN GPIO_NUM_17
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_6
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_7
#define AUDIO_CODEC_ES8311_ADDR ES8311_CODEC_DEFAULT_ADDR
#define BUILTIN_LED_GPIO GPIO_NUM_21
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define DISPLAY_WIDTH 240
#define DISPLAY_HEIGHT 240
#define DISPLAY_MIRROR_X true
#define DISPLAY_MIRROR_Y false
#define DISPLAY_SWAP_XY false
#define DISPLAY_OFFSET_X 0
#define DISPLAY_OFFSET_Y 0
#define DISPLAY_BACKLIGHT_PIN GPIO_NUM_16
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT false
#define DISPLAY_SPI_SCLK_PIN GPIO_NUM_12
#define DISPLAY_SPI_MOSI_PIN GPIO_NUM_10
#define DISPLAY_SPI_CS_PIN GPIO_NUM_13
#define DISPLAY_SPI_DC_PIN GPIO_NUM_14
#define DISPLAY_SPI_RESET_PIN GPIO_NUM_11
#define DISPLAY_SPI_SCLK_HZ (40 * 1000 * 1000)
#endif // _BOARD_CONFIG_H_

View File

@@ -0,0 +1,9 @@
{
"target": "esp32s3",
"builds": [
{
"name": "movecall-cuican-esp32s3",
"sdkconfig_append": []
}
]
}

View File

@@ -0,0 +1,140 @@
#include "wifi_board.h"
#include "audio_codecs/es8311_audio_codec.h"
#include "display/lcd_display.h"
#include "application.h"
#include "button.h"
#include "config.h"
#include "iot/thing_manager.h"
#include "led/single_led.h"
#include <wifi_station.h>
#include <esp_log.h>
#include <esp_efuse_table.h>
#include <driver/i2c_master.h>
#include <esp_lcd_panel_io.h>
#include <esp_lcd_panel_ops.h>
#include <esp_lcd_gc9a01.h>
#include "driver/gpio.h"
#include "driver/spi_master.h"
#define TAG "MovecallCuicanESP32S3"
LV_FONT_DECLARE(font_puhui_16_4);
LV_FONT_DECLARE(font_awesome_16_4);
class MovecallCuicanESP32S3 : public WifiBoard {
private:
i2c_master_bus_handle_t codec_i2c_bus_;
Button boot_button_;
Display* display_;
void InitializeCodecI2c() {
// Initialize I2C peripheral
i2c_master_bus_config_t i2c_bus_cfg = {
.i2c_port = I2C_NUM_0,
.sda_io_num = AUDIO_CODEC_I2C_SDA_PIN,
.scl_io_num = AUDIO_CODEC_I2C_SCL_PIN,
.clk_source = I2C_CLK_SRC_DEFAULT,
.glitch_ignore_cnt = 7,
.intr_priority = 0,
.trans_queue_depth = 0,
.flags = {
.enable_internal_pullup = 1,
},
};
ESP_ERROR_CHECK(i2c_new_master_bus(&i2c_bus_cfg, &codec_i2c_bus_));
}
// SPI初始化
void InitializeSpi() {
ESP_LOGI(TAG, "Initialize SPI bus");
spi_bus_config_t buscfg = GC9A01_PANEL_BUS_SPI_CONFIG(DISPLAY_SPI_SCLK_PIN, DISPLAY_SPI_MOSI_PIN,
DISPLAY_WIDTH * DISPLAY_HEIGHT * sizeof(uint16_t));
ESP_ERROR_CHECK(spi_bus_initialize(SPI3_HOST, &buscfg, SPI_DMA_CH_AUTO));
}
// GC9A01初始化
void InitializeGc9a01Display() {
ESP_LOGI(TAG, "Init GC9A01 display");
ESP_LOGI(TAG, "Install panel IO");
esp_lcd_panel_io_handle_t io_handle = NULL;
esp_lcd_panel_io_spi_config_t io_config = GC9A01_PANEL_IO_SPI_CONFIG(DISPLAY_SPI_CS_PIN, DISPLAY_SPI_DC_PIN, NULL, NULL);
io_config.pclk_hz = DISPLAY_SPI_SCLK_HZ;
ESP_ERROR_CHECK(esp_lcd_new_panel_io_spi(SPI3_HOST, &io_config, &io_handle));
ESP_LOGI(TAG, "Install GC9A01 panel driver");
esp_lcd_panel_handle_t panel_handle = NULL;
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.reset_gpio_num = DISPLAY_SPI_RESET_PIN; // Set to -1 if not use
panel_config.rgb_endian = LCD_RGB_ENDIAN_BGR; //LCD_RGB_ENDIAN_RGB;
panel_config.bits_per_pixel = 16; // Implemented by LCD command `3Ah` (16/18)
ESP_ERROR_CHECK(esp_lcd_new_panel_gc9a01(io_handle, &panel_config, &panel_handle));
ESP_ERROR_CHECK(esp_lcd_panel_reset(panel_handle));
ESP_ERROR_CHECK(esp_lcd_panel_init(panel_handle));
ESP_ERROR_CHECK(esp_lcd_panel_invert_color(panel_handle, true));
ESP_ERROR_CHECK(esp_lcd_panel_mirror(panel_handle, true, false));
ESP_ERROR_CHECK(esp_lcd_panel_disp_on_off(panel_handle, true));
display_ = new SpiLcdDisplay(io_handle, panel_handle,
DISPLAY_WIDTH, DISPLAY_HEIGHT, DISPLAY_OFFSET_X, DISPLAY_OFFSET_Y, DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y, DISPLAY_SWAP_XY,
{
.text_font = &font_puhui_16_4,
.icon_font = &font_awesome_16_4,
.emoji_font = font_emoji_64_init(),
});
}
void InitializeButtons() {
boot_button_.OnClick([this]() {
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
app.ToggleChatState();
});
}
// 物联网初始化,添加对 AI 可见设备
void InitializeIot() {
auto& thing_manager = iot::ThingManager::GetInstance();
thing_manager.AddThing(iot::CreateThing("Speaker"));
thing_manager.AddThing(iot::CreateThing("Screen"));
}
public:
MovecallCuicanESP32S3() : boot_button_(BOOT_BUTTON_GPIO) {
InitializeCodecI2c();
InitializeSpi();
InitializeGc9a01Display();
InitializeButtons();
InitializeIot();
GetBacklight()->RestoreBrightness();
}
virtual Led* GetLed() override {
static SingleLed led_strip(BUILTIN_LED_GPIO);
return &led_strip;
}
virtual Display* GetDisplay() override {
return display_;
}
virtual Backlight* GetBacklight() override {
static PwmBacklight backlight(DISPLAY_BACKLIGHT_PIN, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
return &backlight;
}
virtual AudioCodec* GetAudioCodec() override {
static Es8311AudioCodec audio_codec(codec_i2c_bus_, I2C_NUM_0, AUDIO_INPUT_SAMPLE_RATE, AUDIO_OUTPUT_SAMPLE_RATE,
AUDIO_I2S_GPIO_MCLK, AUDIO_I2S_GPIO_BCLK, AUDIO_I2S_GPIO_WS, AUDIO_I2S_GPIO_DOUT, AUDIO_I2S_GPIO_DIN,
AUDIO_CODEC_PA_PIN, AUDIO_CODEC_ES8311_ADDR);
return &audio_codec;
}
};
DECLARE_BOARD(MovecallCuicanESP32S3);

View File

@@ -54,7 +54,8 @@
#define BSP_PWR_START_UP (BSP_PWR_SDCARD | BSP_PWR_LCD | BSP_PWR_SYSTEM | BSP_PWR_AI_CHIP | BSP_PWR_CODEC_PA | BSP_PWR_GROVE | BSP_PWR_BAT_ADC)
#define BSP_KNOB_BTN (IO_EXPANDER_PIN_NUM_3)
#define BSP_KNOB_A_PIN GPIO_NUM_41
#define BSP_KNOB_B_PIN GPIO_NUM_42
/* QSPI */
#define BSP_SPI3_HOST_PCLK (GPIO_NUM_7)
@@ -81,6 +82,13 @@
#define DISPLAY_BACKLIGHT_PIN GPIO_NUM_8
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT false
/* Touch */
#define BSP_TOUCH_I2C_NUM (1)
#define BSP_TOUCH_GPIO_INT (IO_EXPANDER_PIN_NUM_5)
#define BSP_TOUCH_I2C_SDA (GPIO_NUM_39)
#define BSP_TOUCH_I2C_SCL (GPIO_NUM_38)
#define BSP_TOUCH_I2C_CLK (400000)
/* Settings */
#define DRV_LCD_PIXEL_CLK_HZ (40 * 1000 * 1000)
#define DRV_LCD_CMD_BITS (32)

View File

@@ -6,6 +6,7 @@
#include "font_awesome_symbols.h"
#include "application.h"
#include "button.h"
#include "knob.h"
#include "config.h"
#include "led/single_led.h"
#include "iot/thing_manager.h"
@@ -21,6 +22,7 @@
#include <driver/spi_common.h>
#include <wifi_station.h>
#include <iot_button.h>
#include <iot_knob.h>
#include <esp_io_expander_tca95xx_16bit.h>
#include <esp_sleep.h>
@@ -34,6 +36,7 @@ class SensecapWatcher : public WifiBoard {
private:
i2c_master_bus_handle_t i2c_bus_;
LcdDisplay* display_;
std::unique_ptr<Knob> knob_;
esp_io_expander_handle_t io_exp_handle;
button_handle_t btns;
PowerSaveTimer* power_save_timer_;
@@ -83,6 +86,29 @@ private:
},
};
ESP_ERROR_CHECK(i2c_new_master_bus(&i2c_bus_cfg, &i2c_bus_));
// pulldown for lcd i2c
const gpio_config_t io_config = {
.pin_bit_mask = (1ULL << BSP_TOUCH_I2C_SDA) | (1ULL << BSP_TOUCH_I2C_SCL) | (1ULL << BSP_SPI3_HOST_PCLK) | (1ULL << BSP_SPI3_HOST_DATA0) | (1ULL << BSP_SPI3_HOST_DATA1)
| (1ULL << BSP_SPI3_HOST_DATA2) | (1ULL << BSP_SPI3_HOST_DATA3) | (1ULL << BSP_LCD_SPI_CS) | (1UL << DISPLAY_BACKLIGHT_PIN),
.mode = GPIO_MODE_OUTPUT,
.pull_up_en = GPIO_PULLUP_DISABLE,
.pull_down_en = GPIO_PULLDOWN_DISABLE,
.intr_type = GPIO_INTR_DISABLE,
};
gpio_config(&io_config);
gpio_set_level(BSP_TOUCH_I2C_SDA, 0);
gpio_set_level(BSP_TOUCH_I2C_SCL, 0);
gpio_set_level(BSP_LCD_SPI_CS, 0);
gpio_set_level(DISPLAY_BACKLIGHT_PIN, 0);
gpio_set_level(BSP_SPI3_HOST_PCLK, 0);
gpio_set_level(BSP_SPI3_HOST_DATA0, 0);
gpio_set_level(BSP_SPI3_HOST_DATA1, 0);
gpio_set_level(BSP_SPI3_HOST_DATA2, 0);
gpio_set_level(BSP_SPI3_HOST_DATA3, 0);
}
esp_err_t IoExpanderSetLevel(uint16_t pin_mask, uint8_t level) {
@@ -115,6 +141,41 @@ private:
assert(ret == ESP_OK);
}
void OnKnobRotate(bool clockwise) {
auto codec = GetAudioCodec();
int current_volume = codec->output_volume();
int new_volume = current_volume + (clockwise ? 5 : -5);
// 确保音量在有效范围内
if (new_volume > 100) {
new_volume = 100;
ESP_LOGW(TAG, "Volume reached maximum limit: %d", new_volume);
} else if (new_volume < 0) {
new_volume = 0;
ESP_LOGW(TAG, "Volume reached minimum limit: %d", new_volume);
}
codec->SetOutputVolume(new_volume);
ESP_LOGI(TAG, "Volume changed from %d to %d", current_volume, new_volume);
// 显示通知前检查实际变化
if (new_volume != codec->output_volume()) {
ESP_LOGE(TAG, "Failed to set volume! Expected:%d Actual:%d",
new_volume, codec->output_volume());
}
GetDisplay()->ShowNotification("音量: " + std::to_string(codec->output_volume()));
power_save_timer_->WakeUp();
}
void InitializeKnob() {
knob_ = std::make_unique<Knob>(BSP_KNOB_A_PIN, BSP_KNOB_B_PIN);
knob_->OnRotate([this](bool clockwise) {
ESP_LOGD(TAG, "Knob rotation detected. Clockwise:%s", clockwise ? "true" : "false");
OnKnobRotate(clockwise);
});
ESP_LOGI(TAG, "Knob initialized with pins A:%d B:%d", BSP_KNOB_A_PIN, BSP_KNOB_B_PIN);
}
void InitializeButton() {
button_config_t btn_config = {
.type = BUTTON_TYPE_CUSTOM,
@@ -217,18 +278,13 @@ private:
.emoji_font = font_emoji_64_init(),
});
// 使每次刷新的行数是4的倍数防止花屏
// 使每次刷新的起始列数索引是4的倍数且列数总数是4的倍数以满足SPD2010的要求
lv_display_add_event_cb(lv_display_get_default(), [](lv_event_t *e) {
lv_area_t *area = (lv_area_t *)lv_event_get_param(e);
uint16_t x1 = area->x1;
uint16_t x2 = area->x2;
// round the start of area down to the nearest 4N number
area->x1 = (x1 >> 2) << 2;
// round the start of area down to the nearest 4N number
area->x1 = (x1 >> 2) << 2;
// round the end of area up to the nearest 4M+3 number
area->x2 = ((x2 >> 2) << 2) + 3;
// round the end of area up to the nearest 4M+3 number
area->x2 = ((x2 >> 2) << 2) + 3;
}, LV_EVENT_INVALIDATE_AREA, NULL);
@@ -250,6 +306,7 @@ public:
InitializeSpi();
InitializeExpander();
InitializeButton();
InitializeKnob();
Initializespd2010Display();
InitializeIot();
GetBacklight()->RestoreBrightness();

View File

@@ -0,0 +1,40 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
#define AUDIO_INPUT_SAMPLE_RATE 16000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_I2S_MIC_GPIO_WS GPIO_NUM_4
#define AUDIO_I2S_MIC_GPIO_SCK GPIO_NUM_5
#define AUDIO_I2S_MIC_GPIO_DIN GPIO_NUM_6
#define AUDIO_I2S_SPK_GPIO_DOUT GPIO_NUM_7
#define AUDIO_I2S_SPK_GPIO_BCLK GPIO_NUM_15
#define AUDIO_I2S_SPK_GPIO_LRCK GPIO_NUM_16
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define VOLUME_UP_BUTTON_GPIO GPIO_NUM_NC
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_NC
#define DISPLAY_SDA GPIO_NUM_10
#define DISPLAY_SCL GPIO_NUM_9
#define DISPLAY_DC GPIO_NUM_8
#define DISPLAY_CS GPIO_NUM_14
#define DISPLAY_RES GPIO_NUM_18
#define DISPLAY_WIDTH 128
#define DISPLAY_HEIGHT 128
#define DISPLAY_SWAP_XY false
#define DISPLAY_MIRROR_X false
#define DISPLAY_MIRROR_Y true
#define BACKLIGHT_INVERT false
#define DISPLAY_OFFSET_X 0
#define DISPLAY_OFFSET_Y 0
#define DISPLAY_BACKLIGHT_PIN GPIO_NUM_13
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT false
#define ML307_RX_PIN GPIO_NUM_11
#define ML307_TX_PIN GPIO_NUM_12
#endif // _BOARD_CONFIG_H_

View File

@@ -0,0 +1,9 @@
{
"target": "esp32s3",
"builds": [
{
"name": "xingzhi-cube-0.85tft-ml307",
"sdkconfig_append": []
}
]
}

View File

@@ -0,0 +1,262 @@
#include "ml307_board.h"
#include "audio_codecs/no_audio_codec.h"
#include "display/lcd_display.h"
#include "system_reset.h"
#include "application.h"
#include "button.h"
#include "config.h"
#include "power_save_timer.h"
#include "iot/thing_manager.h"
#include "led/single_led.h"
#include "assets/lang_config.h"
#include "../xingzhi-cube-1.54tft-wifi/power_manager.h"
#include <esp_log.h>
#include <esp_lcd_panel_vendor.h>
#include <wifi_station.h>
#include <driver/rtc_io.h>
#include <esp_sleep.h>
#include <esp_lcd_nv3023.h>
#include "settings.h"
#define TAG "XINGZHI_CUBE_0_85TFT_ML307"
LV_FONT_DECLARE(font_puhui_16_4);
LV_FONT_DECLARE(font_awesome_16_4);
static const nv3023_lcd_init_cmd_t lcd_init_cmds[] = {
{0xff, (uint8_t[]){0xa5}, 1, 0},
{0x3E, (uint8_t[]){0x09}, 1, 0},
{0x3A, (uint8_t[]){0x65}, 1, 0},
{0x82, (uint8_t[]){0x00}, 1, 0},
{0x98, (uint8_t[]){0x00}, 1, 0},
{0x63, (uint8_t[]){0x0f}, 1, 0},
{0x64, (uint8_t[]){0x0f}, 1, 0},
{0xB4, (uint8_t[]){0x34}, 1, 0},
{0xB5, (uint8_t[]){0x30}, 1, 0},
{0x83, (uint8_t[]){0x03}, 1, 0},
{0x86, (uint8_t[]){0x04}, 1, 0},
{0x87, (uint8_t[]){0x16}, 1, 0},
{0x88, (uint8_t[]){0x0A}, 1, 0},
{0x89, (uint8_t[]){0x27}, 1, 0},
{0x93, (uint8_t[]){0x63}, 1, 0},
{0x96, (uint8_t[]){0x81}, 1, 0},
{0xC3, (uint8_t[]){0x10}, 1, 0},
{0xE6, (uint8_t[]){0x00}, 1, 0},
{0x99, (uint8_t[]){0x01}, 1, 0},
{0x70, (uint8_t[]){0x09}, 1, 0},
{0x71, (uint8_t[]){0x1D}, 1, 0},
{0x72, (uint8_t[]){0x14}, 1, 0},
{0x73, (uint8_t[]){0x0a}, 1, 0},
{0x74, (uint8_t[]){0x11}, 1, 0},
{0x75, (uint8_t[]){0x16}, 1, 0},
{0x76, (uint8_t[]){0x38}, 1, 0},
{0x77, (uint8_t[]){0x0B}, 1, 0},
{0x78, (uint8_t[]){0x08}, 1, 0},
{0x79, (uint8_t[]){0x3E}, 1, 0},
{0x7a, (uint8_t[]){0x07}, 1, 0},
{0x7b, (uint8_t[]){0x0D}, 1, 0},
{0x7c, (uint8_t[]){0x16}, 1, 0},
{0x7d, (uint8_t[]){0x0F}, 1, 0},
{0x7e, (uint8_t[]){0x14}, 1, 0},
{0x7f, (uint8_t[]){0x05}, 1, 0},
{0xa0, (uint8_t[]){0x04}, 1, 0},
{0xa1, (uint8_t[]){0x28}, 1, 0},
{0xa2, (uint8_t[]){0x0c}, 1, 0},
{0xa3, (uint8_t[]){0x11}, 1, 0},
{0xa4, (uint8_t[]){0x0b}, 1, 0},
{0xa5, (uint8_t[]){0x23}, 1, 0},
{0xa6, (uint8_t[]){0x45}, 1, 0},
{0xa7, (uint8_t[]){0x07}, 1, 0},
{0xa8, (uint8_t[]){0x0a}, 1, 0},
{0xa9, (uint8_t[]){0x3b}, 1, 0},
{0xaa, (uint8_t[]){0x0d}, 1, 0},
{0xab, (uint8_t[]){0x18}, 1, 0},
{0xac, (uint8_t[]){0x14}, 1, 0},
{0xad, (uint8_t[]){0x0F}, 1, 0},
{0xae, (uint8_t[]){0x19}, 1, 0},
{0xaf, (uint8_t[]){0x08}, 1, 0},
{0xff, (uint8_t[]){0x00}, 1, 0},
{0x11, (uint8_t[]){0x00}, 0, 120},
{0x29, (uint8_t[]){0x00}, 0, 10}
};
class XINGZHI_CUBE_0_85TFT_ML307 : public Ml307Board {
private:
Button boot_button_;
Button volume_up_button_;
Button volume_down_button_;
SpiLcdDisplay* display_;
PowerSaveTimer* power_save_timer_;
PowerManager* power_manager_;
esp_lcd_panel_io_handle_t panel_io_ = nullptr;
esp_lcd_panel_handle_t panel_ = nullptr;
void InitializePowerManager() {
power_manager_ = new PowerManager(GPIO_NUM_38);
power_manager_->OnChargingStatusChanged([this](bool is_charging) {
if (is_charging) {
power_save_timer_->SetEnabled(false);
} else {
power_save_timer_->SetEnabled(true);
}
});
}
void InitializePowerSaveTimer() {
rtc_gpio_init(GPIO_NUM_21);
rtc_gpio_set_direction(GPIO_NUM_21, RTC_GPIO_MODE_OUTPUT_ONLY);
rtc_gpio_set_level(GPIO_NUM_21, 1);
power_save_timer_ = new PowerSaveTimer(-1, 60, 300);
power_save_timer_->OnEnterSleepMode([this]() {
ESP_LOGI(TAG, "Enabling sleep mode");
display_->SetChatMessage("system", "");
display_->SetEmotion("sleepy");
GetBacklight()->SetBrightness(1);
});
power_save_timer_->OnExitSleepMode([this]() {
display_->SetChatMessage("system", "");
display_->SetEmotion("neutral");
GetBacklight()->RestoreBrightness();
});
power_save_timer_->OnShutdownRequest([this]() {
ESP_LOGI(TAG, "Shutting down");
rtc_gpio_set_level(GPIO_NUM_21, 0);
// 启用保持功能,确保睡眠期间电平不变
rtc_gpio_hold_en(GPIO_NUM_21);
esp_lcd_panel_disp_on_off(panel_, false); //关闭显示
esp_deep_sleep_start();
});
power_save_timer_->SetEnabled(true);
}
void InitializeSpi() {
spi_bus_config_t buscfg = {};
buscfg.mosi_io_num = DISPLAY_SDA;
buscfg.miso_io_num = GPIO_NUM_NC;
buscfg.sclk_io_num = DISPLAY_SCL;
buscfg.quadwp_io_num = GPIO_NUM_NC;
buscfg.quadhd_io_num = GPIO_NUM_NC;
buscfg.max_transfer_sz = DISPLAY_HEIGHT * 80 *sizeof(uint16_t);
ESP_ERROR_CHECK(spi_bus_initialize(SPI3_HOST, &buscfg, SPI_DMA_CH_AUTO));
}
void InitializeButtons() {
boot_button_.OnClick([this]() {
auto& app = Application::GetInstance();
app.ToggleChatState();
});
}
void InitializeNv3023Display() {
ESP_LOGD(TAG, "Install panel IO");
esp_lcd_panel_io_spi_config_t io_config = NV3023_PANEL_IO_SPI_CONFIG(DISPLAY_CS, DISPLAY_DC, NULL, NULL);
ESP_ERROR_CHECK(esp_lcd_new_panel_io_spi((esp_lcd_spi_bus_handle_t)SPI3_HOST, &io_config, &panel_io_));
ESP_LOGD(TAG, "Install LCD driver");
esp_lcd_panel_dev_config_t panel_config = {};
nv3023_vendor_config_t vendor_config = { // Uncomment these lines if use custom initialization commands
.init_cmds = lcd_init_cmds,
.init_cmds_size = sizeof(lcd_init_cmds) / sizeof(nv3023_lcd_init_cmd_t),
};
panel_config.reset_gpio_num = DISPLAY_RES;
panel_config.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_BGR;
panel_config.bits_per_pixel = 16;
panel_config.vendor_config = &vendor_config;
ESP_ERROR_CHECK(esp_lcd_new_panel_nv3023(panel_io_, &panel_config, &panel_));
ESP_ERROR_CHECK(esp_lcd_panel_reset(panel_));
ESP_ERROR_CHECK(esp_lcd_panel_init(panel_));
ESP_ERROR_CHECK(esp_lcd_panel_swap_xy(panel_, DISPLAY_SWAP_XY));
ESP_ERROR_CHECK(esp_lcd_panel_mirror(panel_, DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y));
ESP_ERROR_CHECK(esp_lcd_panel_invert_color(panel_, false));
ESP_ERROR_CHECK(esp_lcd_panel_disp_on_off(panel_, true));
display_ = new SpiLcdDisplay(panel_io_, panel_, DISPLAY_WIDTH, DISPLAY_HEIGHT, DISPLAY_OFFSET_X, DISPLAY_OFFSET_Y,
DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y, DISPLAY_SWAP_XY,
{
.text_font = &font_puhui_16_4,
.icon_font = &font_awesome_16_4,
.emoji_font = font_emoji_32_init(),
});
}
void InitializeIot() {
auto& thing_manager = iot::ThingManager::GetInstance();
thing_manager.AddThing(iot::CreateThing("Speaker"));
thing_manager.AddThing(iot::CreateThing("Screen"));
thing_manager.AddThing(iot::CreateThing("Battery"));
}
void Initializegpio21_45() {
rtc_gpio_init(GPIO_NUM_21);
rtc_gpio_set_direction(GPIO_NUM_21, RTC_GPIO_MODE_OUTPUT_ONLY);
rtc_gpio_set_level(GPIO_NUM_21, 1);
//gpio_num_t sp_45 = GPIO_NUM_45;
gpio_config_t io_conf = {};
io_conf.intr_type = GPIO_INTR_DISABLE;
io_conf.mode = GPIO_MODE_OUTPUT;
io_conf.pin_bit_mask = (1ULL << GPIO_NUM_45);
io_conf.pull_up_en = GPIO_PULLUP_DISABLE;
io_conf.pull_down_en = GPIO_PULLDOWN_ENABLE;
gpio_config(&io_conf);
gpio_set_level(GPIO_NUM_45, 0);
}
public:
XINGZHI_CUBE_0_85TFT_ML307(): Ml307Board(ML307_TX_PIN, ML307_RX_PIN, 4096),
boot_button_(BOOT_BUTTON_GPIO),
volume_up_button_(VOLUME_UP_BUTTON_GPIO),
volume_down_button_(VOLUME_DOWN_BUTTON_GPIO) {
Initializegpio21_45(); // 初始时拉高21引脚保证4g模块正常工作
InitializePowerManager();
InitializePowerSaveTimer();
InitializeSpi();
InitializeButtons();
InitializeNv3023Display();
InitializeIot();
GetBacklight()->RestoreBrightness();
}
virtual AudioCodec* GetAudioCodec() override {
static NoAudioCodecSimplex audio_codec(AUDIO_INPUT_SAMPLE_RATE, AUDIO_OUTPUT_SAMPLE_RATE,
AUDIO_I2S_SPK_GPIO_BCLK, AUDIO_I2S_SPK_GPIO_LRCK, AUDIO_I2S_SPK_GPIO_DOUT, AUDIO_I2S_MIC_GPIO_SCK, AUDIO_I2S_MIC_GPIO_WS, AUDIO_I2S_MIC_GPIO_DIN);
return &audio_codec;
}
virtual Display* GetDisplay() override {
return display_;
}
virtual Backlight* GetBacklight() override {
static PwmBacklight backlight(DISPLAY_BACKLIGHT_PIN, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
return &backlight;
}
virtual bool GetBatteryLevel(int& level, bool& charging, bool& discharging) override {
static bool last_discharging = false;
charging = power_manager_->IsCharging();
discharging = power_manager_->IsDischarging();
if (discharging != last_discharging) {
power_save_timer_->SetEnabled(discharging);
last_discharging = discharging;
}
level = power_manager_->GetBatteryLevel();
return true;
}
virtual void SetPowerSaveMode(bool enabled) override {
if (!enabled) {
power_save_timer_->WakeUp();
}
Ml307Board::SetPowerSaveMode(enabled);
}
};
DECLARE_BOARD(XINGZHI_CUBE_0_85TFT_ML307);

View File

@@ -0,0 +1,37 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
#define AUDIO_INPUT_SAMPLE_RATE 16000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
#define AUDIO_I2S_MIC_GPIO_WS GPIO_NUM_4
#define AUDIO_I2S_MIC_GPIO_SCK GPIO_NUM_5
#define AUDIO_I2S_MIC_GPIO_DIN GPIO_NUM_6
#define AUDIO_I2S_SPK_GPIO_DOUT GPIO_NUM_7
#define AUDIO_I2S_SPK_GPIO_BCLK GPIO_NUM_15
#define AUDIO_I2S_SPK_GPIO_LRCK GPIO_NUM_16
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define VOLUME_UP_BUTTON_GPIO GPIO_NUM_NC
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_NC
#define DISPLAY_SDA GPIO_NUM_10
#define DISPLAY_SCL GPIO_NUM_9
#define DISPLAY_DC GPIO_NUM_8
#define DISPLAY_CS GPIO_NUM_14
#define DISPLAY_RES GPIO_NUM_18
#define DISPLAY_WIDTH 128
#define DISPLAY_HEIGHT 128
#define DISPLAY_SWAP_XY false
#define DISPLAY_MIRROR_X false
#define DISPLAY_MIRROR_Y true
#define BACKLIGHT_INVERT false
#define DISPLAY_OFFSET_X 0
#define DISPLAY_OFFSET_Y 0
#define DISPLAY_BACKLIGHT_PIN GPIO_NUM_13
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT false
#endif // _BOARD_CONFIG_H_

View File

@@ -0,0 +1,9 @@
{
"target": "esp32s3",
"builds": [
{
"name": "xingzhi-cube-0.85tft-wifi",
"sdkconfig_append": []
}
]
}

View File

@@ -0,0 +1,266 @@
#include "wifi_board.h"
#include "audio_codecs/no_audio_codec.h"
#include "display/lcd_display.h"
#include "system_reset.h"
#include "application.h"
#include "button.h"
#include "config.h"
#include "power_save_timer.h"
#include "iot/thing_manager.h"
#include "led/single_led.h"
#include "assets/lang_config.h"
#include "../xingzhi-cube-1.54tft-wifi/power_manager.h"
#include <esp_log.h>
#include <esp_lcd_panel_vendor.h>
#include <wifi_station.h>
#include <driver/rtc_io.h>
#include <esp_sleep.h>
#include <esp_lcd_nv3023.h>
#include "settings.h"
#define TAG "XINGZHI_CUBE_0_85TFT_WIFI"
LV_FONT_DECLARE(font_puhui_16_4);
LV_FONT_DECLARE(font_awesome_16_4);
static const nv3023_lcd_init_cmd_t lcd_init_cmds[] = {
{0xff, (uint8_t[]){0xa5}, 1, 0},
{0x3E, (uint8_t[]){0x09}, 1, 0},
{0x3A, (uint8_t[]){0x65}, 1, 0},
{0x82, (uint8_t[]){0x00}, 1, 0},
{0x98, (uint8_t[]){0x00}, 1, 0},
{0x63, (uint8_t[]){0x0f}, 1, 0},
{0x64, (uint8_t[]){0x0f}, 1, 0},
{0xB4, (uint8_t[]){0x34}, 1, 0},
{0xB5, (uint8_t[]){0x30}, 1, 0},
{0x83, (uint8_t[]){0x03}, 1, 0},
{0x86, (uint8_t[]){0x04}, 1, 0},
{0x87, (uint8_t[]){0x16}, 1, 0},
{0x88, (uint8_t[]){0x0A}, 1, 0},
{0x89, (uint8_t[]){0x27}, 1, 0},
{0x93, (uint8_t[]){0x63}, 1, 0},
{0x96, (uint8_t[]){0x81}, 1, 0},
{0xC3, (uint8_t[]){0x10}, 1, 0},
{0xE6, (uint8_t[]){0x00}, 1, 0},
{0x99, (uint8_t[]){0x01}, 1, 0},
{0x70, (uint8_t[]){0x09}, 1, 0},
{0x71, (uint8_t[]){0x1D}, 1, 0},
{0x72, (uint8_t[]){0x14}, 1, 0},
{0x73, (uint8_t[]){0x0a}, 1, 0},
{0x74, (uint8_t[]){0x11}, 1, 0},
{0x75, (uint8_t[]){0x16}, 1, 0},
{0x76, (uint8_t[]){0x38}, 1, 0},
{0x77, (uint8_t[]){0x0B}, 1, 0},
{0x78, (uint8_t[]){0x08}, 1, 0},
{0x79, (uint8_t[]){0x3E}, 1, 0},
{0x7a, (uint8_t[]){0x07}, 1, 0},
{0x7b, (uint8_t[]){0x0D}, 1, 0},
{0x7c, (uint8_t[]){0x16}, 1, 0},
{0x7d, (uint8_t[]){0x0F}, 1, 0},
{0x7e, (uint8_t[]){0x14}, 1, 0},
{0x7f, (uint8_t[]){0x05}, 1, 0},
{0xa0, (uint8_t[]){0x04}, 1, 0},
{0xa1, (uint8_t[]){0x28}, 1, 0},
{0xa2, (uint8_t[]){0x0c}, 1, 0},
{0xa3, (uint8_t[]){0x11}, 1, 0},
{0xa4, (uint8_t[]){0x0b}, 1, 0},
{0xa5, (uint8_t[]){0x23}, 1, 0},
{0xa6, (uint8_t[]){0x45}, 1, 0},
{0xa7, (uint8_t[]){0x07}, 1, 0},
{0xa8, (uint8_t[]){0x0a}, 1, 0},
{0xa9, (uint8_t[]){0x3b}, 1, 0},
{0xaa, (uint8_t[]){0x0d}, 1, 0},
{0xab, (uint8_t[]){0x18}, 1, 0},
{0xac, (uint8_t[]){0x14}, 1, 0},
{0xad, (uint8_t[]){0x0F}, 1, 0},
{0xae, (uint8_t[]){0x19}, 1, 0},
{0xaf, (uint8_t[]){0x08}, 1, 0},
{0xff, (uint8_t[]){0x00}, 1, 0},
{0x11, (uint8_t[]){0x00}, 0, 120},
{0x29, (uint8_t[]){0x00}, 0, 10}
};
class XINGZHI_CUBE_0_85TFT_WIFI : public WifiBoard {
private:
Button boot_button_;
Button volume_up_button_;
Button volume_down_button_;
SpiLcdDisplay* display_;
PowerSaveTimer* power_save_timer_;
PowerManager* power_manager_;
esp_lcd_panel_io_handle_t panel_io_ = nullptr;
esp_lcd_panel_handle_t panel_ = nullptr;
void InitializePowerManager() {
power_manager_ = new PowerManager(GPIO_NUM_38);
power_manager_->OnChargingStatusChanged([this](bool is_charging) {
if (is_charging) {
power_save_timer_->SetEnabled(false);
} else {
power_save_timer_->SetEnabled(true);
}
});
}
void InitializePowerSaveTimer() {
rtc_gpio_init(GPIO_NUM_21);
rtc_gpio_set_direction(GPIO_NUM_21, RTC_GPIO_MODE_OUTPUT_ONLY);
rtc_gpio_set_level(GPIO_NUM_21, 1);
power_save_timer_ = new PowerSaveTimer(-1, 60, 300);
power_save_timer_->OnEnterSleepMode([this]() {
ESP_LOGI(TAG, "Enabling sleep mode");
display_->SetChatMessage("system", "");
display_->SetEmotion("sleepy");
GetBacklight()->SetBrightness(1);
});
power_save_timer_->OnExitSleepMode([this]() {
display_->SetChatMessage("system", "");
display_->SetEmotion("neutral");
GetBacklight()->RestoreBrightness();
});
power_save_timer_->OnShutdownRequest([this]() {
ESP_LOGI(TAG, "Shutting down");
rtc_gpio_set_level(GPIO_NUM_21, 0);
// 启用保持功能,确保睡眠期间电平不变
rtc_gpio_hold_en(GPIO_NUM_21);
esp_lcd_panel_disp_on_off(panel_, false); //关闭显示
esp_deep_sleep_start();
});
power_save_timer_->SetEnabled(true);
}
void InitializeSpi() {
spi_bus_config_t buscfg = {};
buscfg.mosi_io_num = DISPLAY_SDA;
buscfg.miso_io_num = GPIO_NUM_NC;
buscfg.sclk_io_num = DISPLAY_SCL;
buscfg.quadwp_io_num = GPIO_NUM_NC;
buscfg.quadhd_io_num = GPIO_NUM_NC;
buscfg.max_transfer_sz = DISPLAY_HEIGHT * 80 *sizeof(uint16_t);
ESP_ERROR_CHECK(spi_bus_initialize(SPI3_HOST, &buscfg, SPI_DMA_CH_AUTO));
}
void InitializeButtons() {
boot_button_.OnClick([this]() {
power_save_timer_->WakeUp();
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting && !WifiStation::GetInstance().IsConnected()) {
ResetWifiConfiguration();
}
app.ToggleChatState();
});
}
void InitializeNv3023Display() {
ESP_LOGD(TAG, "Install panel IO");
esp_lcd_panel_io_spi_config_t io_config = NV3023_PANEL_IO_SPI_CONFIG(DISPLAY_CS, DISPLAY_DC, NULL, NULL);
ESP_ERROR_CHECK(esp_lcd_new_panel_io_spi((esp_lcd_spi_bus_handle_t)SPI3_HOST, &io_config, &panel_io_));
ESP_LOGD(TAG, "Install LCD driver");
esp_lcd_panel_dev_config_t panel_config = {};
nv3023_vendor_config_t vendor_config = { // Uncomment these lines if use custom initialization commands
.init_cmds = lcd_init_cmds,
.init_cmds_size = sizeof(lcd_init_cmds) / sizeof(nv3023_lcd_init_cmd_t),
};
panel_config.reset_gpio_num = DISPLAY_RES;
panel_config.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_BGR;
panel_config.bits_per_pixel = 16;
panel_config.vendor_config = &vendor_config;
ESP_ERROR_CHECK(esp_lcd_new_panel_nv3023(panel_io_, &panel_config, &panel_));
ESP_ERROR_CHECK(esp_lcd_panel_reset(panel_));
ESP_ERROR_CHECK(esp_lcd_panel_init(panel_));
ESP_ERROR_CHECK(esp_lcd_panel_swap_xy(panel_, DISPLAY_SWAP_XY));
ESP_ERROR_CHECK(esp_lcd_panel_mirror(panel_, DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y));
ESP_ERROR_CHECK(esp_lcd_panel_invert_color(panel_, false));
ESP_ERROR_CHECK(esp_lcd_panel_disp_on_off(panel_, true));
display_ = new SpiLcdDisplay(panel_io_, panel_, DISPLAY_WIDTH, DISPLAY_HEIGHT, DISPLAY_OFFSET_X, DISPLAY_OFFSET_Y,
DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y, DISPLAY_SWAP_XY,
{
.text_font = &font_puhui_16_4,
.icon_font = &font_awesome_16_4,
.emoji_font = font_emoji_32_init(),
});
}
void InitializeIot() {
auto& thing_manager = iot::ThingManager::GetInstance();
thing_manager.AddThing(iot::CreateThing("Speaker"));
thing_manager.AddThing(iot::CreateThing("Screen"));
thing_manager.AddThing(iot::CreateThing("Battery"));
}
void Initializegpio21_45() {
rtc_gpio_init(GPIO_NUM_21);
rtc_gpio_set_direction(GPIO_NUM_21, RTC_GPIO_MODE_OUTPUT_ONLY);
rtc_gpio_set_level(GPIO_NUM_21, 1);
//gpio_num_t sp_45 = GPIO_NUM_45;
gpio_config_t io_conf = {};
io_conf.intr_type = GPIO_INTR_DISABLE;
io_conf.mode = GPIO_MODE_OUTPUT;
io_conf.pin_bit_mask = (1ULL << GPIO_NUM_45);
io_conf.pull_up_en = GPIO_PULLUP_DISABLE;
io_conf.pull_down_en = GPIO_PULLDOWN_ENABLE;
gpio_config(&io_conf);
gpio_set_level(GPIO_NUM_45, 0);
}
public:
XINGZHI_CUBE_0_85TFT_WIFI():
boot_button_(BOOT_BUTTON_GPIO),
volume_up_button_(VOLUME_UP_BUTTON_GPIO),
volume_down_button_(VOLUME_DOWN_BUTTON_GPIO) {
Initializegpio21_45(); // 初始时拉高21引脚保证4g模块正常工作
InitializePowerManager();
InitializePowerSaveTimer();
InitializeSpi();
InitializeButtons();
InitializeNv3023Display();
InitializeIot();
GetBacklight()->RestoreBrightness();
}
virtual AudioCodec* GetAudioCodec() override {
static NoAudioCodecSimplex audio_codec(AUDIO_INPUT_SAMPLE_RATE, AUDIO_OUTPUT_SAMPLE_RATE,
AUDIO_I2S_SPK_GPIO_BCLK, AUDIO_I2S_SPK_GPIO_LRCK, AUDIO_I2S_SPK_GPIO_DOUT, AUDIO_I2S_MIC_GPIO_SCK, AUDIO_I2S_MIC_GPIO_WS, AUDIO_I2S_MIC_GPIO_DIN);
return &audio_codec;
}
virtual Display* GetDisplay() override {
return display_;
}
virtual Backlight* GetBacklight() override {
static PwmBacklight backlight(DISPLAY_BACKLIGHT_PIN, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
return &backlight;
}
virtual bool GetBatteryLevel(int& level, bool& charging, bool& discharging) override {
static bool last_discharging = false;
charging = power_manager_->IsCharging();
discharging = power_manager_->IsDischarging();
if (discharging != last_discharging) {
power_save_timer_->SetEnabled(discharging);
last_discharging = discharging;
}
level = power_manager_->GetBatteryLevel();
return true;
}
virtual void SetPowerSaveMode(bool enabled) override {
if (!enabled) {
power_save_timer_->WakeUp();
}
WifiBoard::SetPowerSaveMode(enabled);
}
};
DECLARE_BOARD(XINGZHI_CUBE_0_85TFT_WIFI);

View File

@@ -103,6 +103,7 @@ SpiLcdDisplay::SpiLcdDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_h
ESP_LOGI(TAG, "Initialize LVGL port");
lvgl_port_cfg_t port_cfg = ESP_LVGL_PORT_INIT_CONFIG();
port_cfg.task_priority = 1;
port_cfg.timer_period_ms = 50;
lvgl_port_init(&port_cfg);
ESP_LOGI(TAG, "Adding LCD screen");
@@ -278,7 +279,7 @@ void LcdDisplay::SetupUI() {
/* Status bar */
status_bar_ = lv_obj_create(container_);
lv_obj_set_size(status_bar_, LV_HOR_RES, fonts_.emoji_font->line_height);
lv_obj_set_size(status_bar_, LV_HOR_RES, LV_SIZE_CONTENT);
lv_obj_set_style_radius(status_bar_, 0, 0);
lv_obj_set_style_bg_color(status_bar_, current_theme.background, 0);
lv_obj_set_style_text_color(status_bar_, current_theme.text, 0);
@@ -288,7 +289,7 @@ void LcdDisplay::SetupUI() {
lv_obj_set_style_radius(content_, 0, 0);
lv_obj_set_width(content_, LV_HOR_RES);
lv_obj_set_flex_grow(content_, 1);
lv_obj_set_style_pad_all(content_, 5, 0);
lv_obj_set_style_pad_all(content_, 10, 0);
lv_obj_set_style_bg_color(content_, current_theme.chat_background, 0); // Background for chat area
lv_obj_set_style_border_color(content_, current_theme.border, 0); // Border color for chat area
@@ -309,8 +310,10 @@ void LcdDisplay::SetupUI() {
lv_obj_set_style_pad_all(status_bar_, 0, 0);
lv_obj_set_style_border_width(status_bar_, 0, 0);
lv_obj_set_style_pad_column(status_bar_, 0, 0);
lv_obj_set_style_pad_left(status_bar_, 2, 0);
lv_obj_set_style_pad_right(status_bar_, 2, 0);
lv_obj_set_style_pad_left(status_bar_, 10, 0);
lv_obj_set_style_pad_right(status_bar_, 10, 0);
lv_obj_set_style_pad_top(status_bar_, 2, 0);
lv_obj_set_style_pad_bottom(status_bar_, 2, 0);
lv_obj_set_scrollbar_mode(status_bar_, LV_SCROLLBAR_MODE_OFF);
// 设置状态栏的内容垂直居中
lv_obj_set_flex_align(status_bar_, LV_FLEX_ALIGN_SPACE_BETWEEN, LV_FLEX_ALIGN_CENTER, LV_FLEX_ALIGN_CENTER);
@@ -366,7 +369,7 @@ void LcdDisplay::SetupUI() {
lv_obj_add_flag(low_battery_popup_, LV_OBJ_FLAG_HIDDEN);
}
#define MAX_MESSAGES 50
#define MAX_MESSAGES 20
void LcdDisplay::SetChatMessage(const char* role, const char* content) {
DisplayLockGuard lock(this);
if (content_ == nullptr) {
@@ -376,6 +379,21 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
//避免出现空的消息框
if(strlen(content) == 0) return;
// 检查消息数量是否超过限制
uint32_t child_count = lv_obj_get_child_cnt(content_);
if (child_count >= MAX_MESSAGES) {
// 删除最早的消息(第一个子对象)
lv_obj_t* first_child = lv_obj_get_child(content_, 0);
lv_obj_t* last_child = lv_obj_get_child(content_, child_count - 1);
if (first_child != nullptr) {
lv_obj_del(first_child);
}
// Scroll to the last message immediately
if (last_child != nullptr) {
lv_obj_scroll_to_view_recursive(last_child, LV_ANIM_OFF);
}
}
// Create a message bubble
lv_obj_t* msg_bubble = lv_obj_create(content_);
lv_obj_set_style_radius(msg_bubble, 8, 0);
@@ -431,9 +449,6 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
lv_obj_set_width(msg_bubble, LV_SIZE_CONTENT);
lv_obj_set_height(msg_bubble, LV_SIZE_CONTENT);
// Add some margin
lv_obj_set_style_margin_right(msg_bubble, 10, 0);
// Don't grow
lv_obj_set_style_flex_grow(msg_bubble, 0, 0);
} else if (strcmp(role, "assistant") == 0) {
@@ -449,9 +464,6 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
lv_obj_set_width(msg_bubble, LV_SIZE_CONTENT);
lv_obj_set_height(msg_bubble, LV_SIZE_CONTENT);
// Add some margin
lv_obj_set_style_margin_left(msg_bubble, -4, 0);
// Don't grow
lv_obj_set_style_flex_grow(msg_bubble, 0, 0);
} else if (strcmp(role, "system") == 0) {
@@ -487,7 +499,7 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
lv_obj_set_parent(msg_bubble, container);
// Right align the bubble in the container
lv_obj_align(msg_bubble, LV_ALIGN_RIGHT_MID, -10, 0);
lv_obj_align(msg_bubble, LV_ALIGN_RIGHT_MID, -25, 0);
// Auto-scroll to this container
lv_obj_scroll_to_view_recursive(container, LV_ANIM_ON);
@@ -521,19 +533,6 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
// Store reference to the latest message label
chat_message_label_ = msg_text;
// 检查消息数量是否超过限制
uint32_t msg_count = lv_obj_get_child_cnt(content_);
while (msg_count >= MAX_MESSAGES) {
// 删除最早的消息(第一个子节点)
lv_obj_t* oldest_msg = lv_obj_get_child(content_, 0);
if (oldest_msg != nullptr) {
lv_obj_del(oldest_msg);
msg_count--;
}else{
break;
}
}
}
#else
void LcdDisplay::SetupUI() {

View File

@@ -22,6 +22,7 @@ OledDisplay::OledDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handl
ESP_LOGI(TAG, "Initialize LVGL");
lvgl_port_cfg_t port_cfg = ESP_LVGL_PORT_INIT_CONFIG();
port_cfg.task_priority = 1;
port_cfg.timer_period_ms = 50;
lvgl_port_init(&port_cfg);
ESP_LOGI(TAG, "Adding LCD screen");

View File

@@ -9,16 +9,21 @@ dependencies:
espressif/esp_lcd_panel_io_additions: "^1.0.1"
78/esp_lcd_nv3023: "~1.0.0"
78/esp-wifi-connect: "~2.3.1"
78/esp-opus-encoder: "~2.1.0"
78/esp-ml307: "~1.7.2"
78/esp-opus-encoder: "~2.3.0"
78/esp-ml307: "~1.7.3"
78/xiaozhi-fonts: "~1.3.2"
espressif/led_strip: "^2.4.1"
espressif/esp_codec_dev: "~1.3.2"
espressif/esp-sr: "^2.0.2"
espressif/esp-sr: "^2.0.3"
espressif/button: "^3.3.1"
espressif/knob: "^1.0.0"
lvgl/lvgl: "~9.2.2"
esp_lvgl_port: "~2.4.4"
espressif/esp_io_expander_tca95xx_16bit: "^2.0.0"
tny-robotics/sh1106-esp-idf:
version: "^1.0.0"
rules:
- if: 'idf_version >= "5.4.0"'
## Required IDF version
idf:
version: ">=5.3"

209
main/iot/README.md Normal file
View File

@@ -0,0 +1,209 @@
# 物联网控制模块
本模块实现了小智AI语音聊天机器人的物联网控制功能使用户可以通过语音指令控制接入到ESP32开发板的各种物联网设备。
## 工作原理
整个物联网控制模块的工作流程如下:
1. **设备注册**:在开发板初始化阶段(如在`compact_wifi_board.cc`中),各种物联网设备通过`ThingManager`注册到系统中
2. **设备描述**系统将设备描述信息包括名称、属性、方法等通过通信协议如MQTT或WebSocket发送给AI服务器
3. **用户交互**用户通过语音与小智AI对话表达控制物联网设备的意图
4. **命令执行**AI服务器解析用户意图生成控制命令通过协议发送回ESP32`ThingManager`分发给对应的设备执行
5. **状态更新**:设备执行命令后,状态变化会通过`ThingManager`收集并发送回AI服务器保持状态同步
## 核心组件
### ThingManager
`ThingManager`是物联网控制模块的核心管理类,采用单例模式实现:
- `AddThing`:注册物联网设备
- `GetDescriptorsJson`获取所有设备的描述信息用于向AI服务器报告设备能力
- `GetStatesJson`:获取所有设备的当前状态,可以选择只返回变化的部分
- `Invoke`根据AI服务器下发的命令调用对应设备的方法
### Thing
`Thing`是所有物联网设备的基类,提供了以下核心功能:
- 属性管理:通过`PropertyList`定义设备的可查询状态
- 方法管理:通过`MethodList`定义设备可执行的操作
- JSON序列化将设备描述和状态转换为JSON格式便于网络传输
- 命令执行解析和执行来自AI服务器的指令
## 设备设计示例
### 灯Lamp
灯是一个简单的物联网设备示例通过GPIO控制LED的开关状态
```cpp
class Lamp : public Thing {
private:
gpio_num_t gpio_num_ = GPIO_NUM_18; // GPIO引脚
bool power_ = false; // 灯的开关状态
public:
Lamp() : Thing("Lamp", "一个测试用的灯") {
// 初始化GPIO
InitializeGpio();
// 定义属性power表示灯的开关状态
properties_.AddBooleanProperty("power", "灯是否打开", [this]() -> bool {
return power_;
});
// 定义方法TurnOn打开灯
methods_.AddMethod("TurnOn", "打开灯", ParameterList(), [this](const ParameterList& parameters) {
power_ = true;
gpio_set_level(gpio_num_, 1);
});
// 定义方法TurnOff关闭灯
methods_.AddMethod("TurnOff", "关闭灯", ParameterList(), [this](const ParameterList& parameters) {
power_ = false;
gpio_set_level(gpio_num_, 0);
});
}
};
```
用户可以通过语音指令如"小智,请打开灯"来控制灯的开关。
### 扬声器Speaker
扬声器控制实现了音量调节功能:
```cpp
class Speaker : public Thing {
public:
Speaker() : Thing("Speaker", "扬声器") {
// 定义属性volume当前音量值
properties_.AddNumberProperty("volume", "当前音量值", [this]() -> int {
auto codec = Board::GetInstance().GetAudioCodec();
return codec->output_volume();
});
// 定义方法SetVolume设置音量
methods_.AddMethod("SetVolume", "设置音量", ParameterList({
Parameter("volume", "0到100之间的整数", kValueTypeNumber, true)
}), [this](const ParameterList& parameters) {
auto codec = Board::GetInstance().GetAudioCodec();
codec->SetOutputVolume(static_cast<uint8_t>(parameters["volume"].number()));
});
}
};
```
用户可以通过语音指令如"小智把音量调到50"来控制扬声器的音量。
## 设计自定义物联网设备
要设计一个新的物联网设备,需要以下步骤:
1. **创建设备类**:继承`Thing`基类
2. **定义属性**:使用`properties_`添加设备的可查询状态
3. **定义方法**:使用`methods_`添加设备可执行的操作
4. **实现硬件控制**:在方法回调中实现对硬件的控制
5. **注册设备**:注册设备有两种方式(见下文),并在板级初始化中添加设备实例
### 两种设备注册方式
1. **使用DECLARE_THING宏**:适合通用设备类型
```cpp
// 在设备实现文件末尾添加
DECLARE_THING(MyDevice);
// 然后在板级初始化中
thing_manager.AddThing(iot::CreateThing("MyDevice"));
```
2. **直接创建设备实例**:适合特定于板级的设备
```cpp
// 在板级初始化中
auto my_device = new iot::MyDevice();
thing_manager.AddThing(my_device);
```
### 设备实现位置建议
您可以根据设备的通用性选择不同的实现位置:
1. **通用设备**:放在`main/iot/things/`目录下,使用`DECLARE_THING`注册
2. **板级特定设备**:直接在板级目录(如`main/boards/your_board/`)中实现,使用直接创建实例的方式注册
这种灵活性允许您为不同的板设计特定的设备实现,同时保持通用设备的可重用性。
### 属性类型
物联网设备支持以下属性类型:
- **布尔值**`kValueTypeBoolean`):开关状态等
- **数值**`kValueTypeNumber`):温度、音量等
- **字符串**`kValueTypeString`):设备名称、状态描述等
### 方法参数
设备方法可以定义参数,支持以下参数类型:
- **布尔值**:开关等
- **数值**:温度、音量等
- **字符串**:命令、模式等
## 使用示例
在板级初始化代码(如`compact_wifi_board.cc`)中注册物联网设备:
```cpp
void InitializeIot() {
auto& thing_manager = iot::ThingManager::GetInstance();
thing_manager.AddThing(iot::CreateThing("Speaker"));
thing_manager.AddThing(iot::CreateThing("Lamp"));
}
```
之后,用户可以通过语音指令控制这些设备,例如:
- "小智,打开灯"
- "我要睡觉了,帮我关灯"
- "音量有点太小了"
- "把音量设置为80%"
AI服务器会将这些语音指令解析为对应的设备控制命令通过协议发送给ESP32执行。
## 注意事项
### 大模型控制的随机性
由于语音控制由大型语言模型(LLM)处理,控制过程可能存在一定的随机性和不确定性。为了增强安全性和可靠性,请考虑以下建议:
1. **关键操作添加警示信息**:对于潜在危险或不可逆的操作,在方法描述中添加警示信息
```cpp
methods_.AddMethod("PowerOff", "关闭系统电源[警告:此操作将导致设备完全关闭,请慎重使用]",
ParameterList(), [this](const ParameterList& parameters) {
// 关闭电源的实现
});
```
2. **二次确认机制**:重要操作应在描述中明确要求二次确认
```cpp
methods_.AddMethod("ResetToFactory", "恢复出厂设置[必须要用户二次确认]",
ParameterList(), [this](const ParameterList& parameters) {
// 恢复出厂设置的实现
});
```
### 通信协议限制
当前IoT协议(1.0版本)存在以下限制:
1. **单向控制流**:大模型只能下发指令,无法立即获取指令执行结果
2. **状态更新延迟**设备状态变更需要等到下一轮对话时通过读取property属性值才能获知
3. **异步反馈**:如果需要操作结果反馈,必须通过设备属性的方式间接实现
### 最佳实践
1. **使用有意义的属性名称**:属性名称应清晰表达其含义,便于大模型理解和使用
2. **不产生歧义的方法描述**:为每个方法提供明确的自然语言描述,帮助大模型更准确地理解和调用

View File

@@ -1,38 +0,0 @@
[
{
"name": "lamp",
"description": "A lamp",
"properties": {
"power": {
"type": "boolean",
"description": "Whether the lamp is on or off"
}
},
"methods": {
"TurnOn": {
"description": "Turns the lamp on"
}
}
},
{
"name": "speaker",
"description": "当前 AI 机器人的扬声器",
"properties": {
"volume": {
"type": "number",
"description": "当前扬声器的音量0-100"
}
},
"methods": {
"SetVolume": {
"description": "设置当前扬声器的音量",
"parameters": {
"volume": {
"type": "number",
"description": "The volume of the speaker (0-100)"
}
}
}
}
}
]

View File

@@ -20,11 +20,19 @@
#define LEDC_LS_MODE LEDC_LOW_SPEED_MODE
#define LEDC_LS_CH0_CHANNEL LEDC_CHANNEL_0
#define LEDC_DUTY (4096)
#define LEDC_DUTY (8191)
#define LEDC_FADE_TIME (1000)
// GPIO_LED
GpioLed::GpioLed(gpio_num_t gpio, int output_invert) {
GpioLed::GpioLed(gpio_num_t gpio)
: GpioLed(gpio, 0, LEDC_LS_TIMER, LEDC_LS_CH0_CHANNEL) {
}
GpioLed::GpioLed(gpio_num_t gpio, int output_invert)
: GpioLed(gpio, output_invert, LEDC_LS_TIMER, LEDC_LS_CH0_CHANNEL) {
}
GpioLed::GpioLed(gpio_num_t gpio, int output_invert, ledc_timer_t timer_num, ledc_channel_t channel) {
// If the gpio is not connected, you should use NoLed class
assert(gpio != GPIO_NUM_NC);
@@ -33,20 +41,20 @@ GpioLed::GpioLed(gpio_num_t gpio, int output_invert) {
* that will be used by LED Controller
*/
ledc_timer_config_t ledc_timer = {};
ledc_timer.duty_resolution = LEDC_TIMER_13_BIT; // resolution of PWM duty
ledc_timer.duty_resolution = LEDC_TIMER_13_BIT; // resolution of PWM duty
ledc_timer.freq_hz = 4000; // frequency of PWM signal
ledc_timer.speed_mode = LEDC_LS_MODE; // timer mode
ledc_timer.timer_num = LEDC_LS_TIMER; // timer index
ledc_timer.timer_num = timer_num; // timer index
ledc_timer.clk_cfg = LEDC_AUTO_CLK; // Auto select the source clock
ESP_ERROR_CHECK(ledc_timer_config(&ledc_timer));
ledc_channel_.channel = LEDC_LS_CH0_CHANNEL,
ledc_channel_.channel = channel,
ledc_channel_.duty = 0,
ledc_channel_.gpio_num = gpio,
ledc_channel_.speed_mode = LEDC_LS_MODE,
ledc_channel_.hpoint = 0,
ledc_channel_.timer_sel = LEDC_LS_TIMER,
ledc_channel_.timer_sel = timer_num,
ledc_channel_.flags.output_invert = output_invert & 0x01,
// Set LED Controller with previously prepared configuration
@@ -86,7 +94,11 @@ GpioLed::~GpioLed() {
void GpioLed::SetBrightness(uint8_t brightness) {
duty_ = brightness * LEDC_DUTY / 100;
if (brightness == 100) {
duty_ = LEDC_DUTY;
} else {
duty_ = brightness * LEDC_DUTY / 100;
}
}
void GpioLed::TurnOn() {

View File

@@ -11,13 +11,18 @@
#include <mutex>
class GpioLed : public Led {
public:
GpioLed(gpio_num_t gpio, int output_invert=0);
public:
GpioLed(gpio_num_t gpio);
GpioLed(gpio_num_t gpio, int output_invert);
GpioLed(gpio_num_t gpio, int output_invert, ledc_timer_t timer_num, ledc_channel_t channel);
virtual ~GpioLed();
void OnStateChanged() override;
void TurnOn();
void TurnOff();
void SetBrightness(uint8_t brightness);
private:
private:
std::mutex mutex_;
TaskHandle_t blink_task_ = nullptr;
ledc_channel_config_t ledc_channel_ = {0};
@@ -34,12 +39,9 @@ private:
void BlinkOnce();
void Blink(int times, int interval_ms);
void StartContinuousBlink(int interval_ms);
void TurnOn();
void TurnOff();
void SetBrightness(uint8_t brightness);
void StartFadeTask();
void OnFadeEnd();
static bool FadeCallback(const ledc_cb_param_t *param, void *user_arg);
};
#endif // _GPIO_LED_H_
#endif // _GPIO_LED_H_

View File

@@ -254,6 +254,10 @@ void MqttProtocol::ParseServerHello(const cJSON* root) {
if (sample_rate != NULL) {
server_sample_rate_ = sample_rate->valueint;
}
auto frame_duration = cJSON_GetObjectItem(audio_params, "frame_duration");
if (frame_duration != NULL) {
server_frame_duration_ = frame_duration->valueint;
}
}
auto udp = cJSON_GetObjectItem(root, "udp");

View File

@@ -49,7 +49,7 @@ void Protocol::SendWakeWordDetected(const std::string& wake_word) {
void Protocol::SendStartListening(ListeningMode mode) {
std::string message = "{\"session_id\":\"" + session_id_ + "\"";
message += ",\"type\":\"listen\",\"state\":\"start\"";
if (mode == kListeningModeAlwaysOn) {
if (mode == kListeningModeRealtime) {
message += ",\"mode\":\"realtime\"";
} else if (mode == kListeningModeAutoStop) {
message += ",\"mode\":\"auto\"";

View File

@@ -21,7 +21,7 @@ enum AbortReason {
enum ListeningMode {
kListeningModeAutoStop,
kListeningModeManualStop,
kListeningModeAlwaysOn // 需要 AEC 支持
kListeningModeRealtime // 需要 AEC 支持
};
class Protocol {
@@ -31,6 +31,9 @@ public:
inline int server_sample_rate() const {
return server_sample_rate_;
}
inline int server_frame_duration() const {
return server_frame_duration_;
}
inline const std::string& session_id() const {
return session_id_;
}
@@ -60,7 +63,8 @@ protected:
std::function<void()> on_audio_channel_closed_;
std::function<void(const std::string& message)> on_network_error_;
int server_sample_rate_ = 16000;
int server_sample_rate_ = 24000;
int server_frame_duration_ = 60;
bool error_occurred_ = false;
std::string session_id_;
std::chrono::time_point<std::chrono::steady_clock> last_incoming_time_;

View File

@@ -146,6 +146,10 @@ void WebsocketProtocol::ParseServerHello(const cJSON* root) {
if (sample_rate != NULL) {
server_sample_rate_ = sample_rate->valueint;
}
auto frame_duration = cJSON_GetObjectItem(audio_params, "frame_duration");
if (frame_duration != NULL) {
server_frame_duration_ = frame_duration->valueint;
}
}
xEventGroupSetBits(event_group_handle_, WEBSOCKET_PROTOCOL_SERVER_HELLO_EVENT);

View File

@@ -4,6 +4,7 @@ nvsfactory, data, nvs, , 200K,
nvs, data, nvs, , 840K,
otadata, data, ota, , 0x2000,
phy_init, data, phy, , 0x1000,
model, data, spiffs, , 1024K,
ota_0, app, ota_0, , 12M,
ota_1, app, ota_1, , 12M,
model, data, spiffs, , 0xF0000,
# According to scripts/versions.py, app partition must be aligned to 1MB
ota_0, app, ota_0, 0x200000, 12M,
ota_1, app, ota_1, , 12M,
1 # ESP-IDF Partition Table
4 nvs, data, nvs, , 840K,
5 otadata, data, ota, , 0x2000,
6 phy_init, data, phy, , 0x1000,
7 model, data, spiffs, , 1024K, model, data, spiffs, , 0xF0000,
8 ota_0, app, ota_0, , 12M, # According to scripts/versions.py, app partition must be aligned to 1MB
9 ota_1, app, ota_1, , 12M, ota_0, app, ota_0, 0x200000, 12M,
10 ota_1, app, ota_1, , 12M,

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,33 @@
# LVGL图片转换工具
这个目录包含两个用于处理和转换图片为LVGL格式的Python脚本
## 1. LVGLImage (LVGLImage.py)
引用自LVGL[官方repo](https://github.com/lvgl/lvgl)的转换脚本[LVGLImage.py](https://github.com/lvgl/lvgl/blob/master/scripts/LVGLImage.py)
## 2. LVGL图片转换工具 (lvgl_tools_gui.py)
调用`LVGLImage.py`将图片批量转换为LVGL图片格式
可用于修改小智的默认表情,具体修改教程[在这里](https://www.bilibili.com/video/BV12FQkYeEJ3/)
### 特性
- 图形化操作,界面更友好
- 支持批量转换图片
- 自动识别图片格式并选择最佳的颜色格式转换
- 多分辨率支持
### 使用方法
安装Pillow
```bash
pip install Pillow # 处理图像需要
```
运行转换工具
```bash
python lvgl_tools_gui.py
```

View File

@@ -0,0 +1,253 @@
import tkinter as tk
from tkinter import ttk, filedialog, messagebox
from PIL import Image
import os
import tempfile
import sys
from LVGLImage import LVGLImage, ColorFormat, CompressMethod
HELP_TEXT = """LVGL图片转换工具使用说明
1. 添加文件:点击“添加文件”按钮选择需要转换的图片,支持批量导入
2. 移除文件:在列表中选中文件前的复选框“[ ]”(选中后会变成“[√]”),点击“移除选中”可删除选定文件
3. 设置分辨率选择需要的分辨率如128x128
建议根据自己的设备的屏幕分辨率来选择。过大和过小都会影响显示效果。
4. 颜色格式:选择“自动识别”会根据图片是否透明自动选择,或手动指定
除非你了解这个选项,否则建议使用自动识别,不然可能会出现一些意想不到的问题……
5. 压缩方式选择NONE或RLE压缩
除非你了解这个选项否则建议保持默认NONE不压缩
6. 输出目录:设置转换后文件的保存路径
默认为程序所在目录下的output文件夹
7. 转换:点击“转换全部”或“转换选中”开始转换
"""
class ImageConverterApp:
def __init__(self, root):
self.root = root
self.root.title("LVGL图片转换工具")
self.root.geometry("750x650")
# 初始化变量
self.output_dir = tk.StringVar(value=os.path.abspath("output"))
self.resolution = tk.StringVar(value="128x128")
self.color_format = tk.StringVar(value="自动识别")
self.compress_method = tk.StringVar(value="NONE")
# 创建UI组件
self.create_widgets()
self.redirect_output()
def create_widgets(self):
# 参数设置框架
settings_frame = ttk.LabelFrame(self.root, text="转换设置")
settings_frame.grid(row=0, column=0, padx=10, pady=5, sticky="ew")
# 分辨率设置
ttk.Label(settings_frame, text="分辨率:").grid(row=0, column=0, padx=2)
ttk.Combobox(settings_frame, textvariable=self.resolution,
values=["128x128", "64x64", "32x32"], width=8).grid(row=0, column=1, padx=2)
# 颜色格式
ttk.Label(settings_frame, text="颜色格式:").grid(row=0, column=2, padx=2)
ttk.Combobox(settings_frame, textvariable=self.color_format,
values=["自动识别", "RGB565", "RGB565A8"], width=10).grid(row=0, column=3, padx=2)
# 压缩方式
ttk.Label(settings_frame, text="压缩方式:").grid(row=0, column=4, padx=2)
ttk.Combobox(settings_frame, textvariable=self.compress_method,
values=["NONE", "RLE"], width=8).grid(row=0, column=5, padx=2)
# 文件操作框架
file_frame = ttk.LabelFrame(self.root, text="输入文件")
file_frame.grid(row=1, column=0, padx=10, pady=5, sticky="nsew")
# 文件操作按钮
btn_frame = ttk.Frame(file_frame)
btn_frame.pack(fill=tk.X, pady=2)
ttk.Button(btn_frame, text="添加文件", command=self.select_files).pack(side=tk.LEFT, padx=2)
ttk.Button(btn_frame, text="移除选中", command=self.remove_selected).pack(side=tk.LEFT, padx=2)
ttk.Button(btn_frame, text="清空列表", command=self.clear_files).pack(side=tk.LEFT, padx=2)
# 文件列表Treeview
self.tree = ttk.Treeview(file_frame, columns=("selected", "filename"),
show="headings", height=10)
self.tree.heading("selected", text="选中", anchor=tk.W)
self.tree.heading("filename", text="文件名", anchor=tk.W)
self.tree.column("selected", width=60, anchor=tk.W)
self.tree.column("filename", width=600, anchor=tk.W)
self.tree.pack(fill=tk.BOTH, expand=True)
self.tree.bind("<ButtonRelease-1>", self.on_tree_click)
# 输出目录
output_frame = ttk.LabelFrame(self.root, text="输出目录")
output_frame.grid(row=2, column=0, padx=10, pady=5, sticky="ew")
ttk.Entry(output_frame, textvariable=self.output_dir, width=60).pack(side=tk.LEFT, padx=5)
ttk.Button(output_frame, text="浏览", command=self.select_output_dir).pack(side=tk.RIGHT, padx=5)
# 转换按钮和帮助按钮
convert_frame = ttk.Frame(self.root)
convert_frame.grid(row=3, column=0, padx=10, pady=10)
ttk.Button(convert_frame, text="转换全部文件", command=lambda: self.start_conversion(True)).pack(side=tk.LEFT, padx=5)
ttk.Button(convert_frame, text="转换选中文件", command=lambda: self.start_conversion(False)).pack(side=tk.LEFT, padx=5)
ttk.Button(convert_frame, text="帮助", command=self.show_help).pack(side=tk.RIGHT, padx=5)
# 日志区域(新增清空按钮部分)
log_frame = ttk.LabelFrame(self.root, text="日志")
log_frame.grid(row=4, column=0, padx=10, pady=5, sticky="nsew")
# 添加按钮框架
log_btn_frame = ttk.Frame(log_frame)
log_btn_frame.pack(fill=tk.X, side=tk.BOTTOM)
# 清空日志按钮
ttk.Button(log_btn_frame, text="清空日志", command=self.clear_log).pack(side=tk.RIGHT, padx=5, pady=2)
self.log_text = tk.Text(log_frame, height=15)
self.log_text.pack(fill=tk.BOTH, expand=True)
# 布局配置
self.root.columnconfigure(0, weight=1)
self.root.rowconfigure(1, weight=1)
self.root.rowconfigure(4, weight=1)
def clear_log(self):
"""清空日志内容"""
self.log_text.delete(1.0, tk.END)
def show_help(self):
messagebox.showinfo("帮助", HELP_TEXT)
def redirect_output(self):
class StdoutRedirector:
def __init__(self, text_widget):
self.text_widget = text_widget
self.original_stdout = sys.stdout
def write(self, message):
self.text_widget.insert(tk.END, message)
self.text_widget.see(tk.END)
self.original_stdout.write(message)
def flush(self):
self.original_stdout.flush()
sys.stdout = StdoutRedirector(self.log_text)
def on_tree_click(self, event):
region = self.tree.identify("region", event.x, event.y)
if region == "cell":
col = self.tree.identify_column(event.x)
item = self.tree.identify_row(event.y)
if col == "#1": # 点击的是选中列
current_val = self.tree.item(item, "values")[0]
new_val = "[√]" if current_val == "[ ]" else "[ ]"
self.tree.item(item, values=(new_val, self.tree.item(item, "values")[1]))
def select_output_dir(self):
path = filedialog.askdirectory()
if path:
self.output_dir.set(path)
def select_files(self):
files = filedialog.askopenfilenames(filetypes=[("图片文件", "*.png;*.jpg;*.jpeg;*.bmp;*.gif")])
for f in files:
self.tree.insert("", tk.END, values=("[ ]", os.path.basename(f)), tags=(f,))
def remove_selected(self):
to_remove = []
for item in self.tree.get_children():
if self.tree.item(item, "values")[0] == "[√]":
to_remove.append(item)
for item in reversed(to_remove):
self.tree.delete(item)
def clear_files(self):
for item in self.tree.get_children():
self.tree.delete(item)
def start_conversion(self, convert_all):
input_files = [
self.tree.item(item, "tags")[0]
for item in self.tree.get_children()
if convert_all or self.tree.item(item, "values")[0] == "[√]"
]
if not input_files:
msg = "没有找到可转换的文件" if convert_all else "没有选中任何文件"
messagebox.showwarning("警告", msg)
return
os.makedirs(self.output_dir.get(), exist_ok=True)
# 解析转换参数
width, height = map(int, self.resolution.get().split('x'))
compress = CompressMethod.RLE if self.compress_method.get() == "RLE" else CompressMethod.NONE
# 执行转换
self.convert_images(input_files, width, height, compress)
def convert_images(self, input_files, width, height, compress):
success_count = 0
total_files = len(input_files)
for idx, file_path in enumerate(input_files):
try:
print(f"正在处理: {os.path.basename(file_path)}")
with Image.open(file_path) as img:
# 调整图片大小
img = img.resize((width, height), Image.Resampling.LANCZOS)
# 处理颜色格式
color_format_str = self.color_format.get()
if color_format_str == "自动识别":
# 检测透明通道
has_alpha = img.mode in ('RGBA', 'LA') or (img.mode == 'P' and 'transparency' in img.info)
if has_alpha:
img = img.convert('RGBA')
cf = ColorFormat.RGB565A8
else:
img = img.convert('RGB')
cf = ColorFormat.RGB565
else:
if color_format_str == "RGB565A8":
img = img.convert('RGBA')
cf = ColorFormat.RGB565A8
else:
img = img.convert('RGB')
cf = ColorFormat.RGB565
# 保存调整后的图片
base_name = os.path.splitext(os.path.basename(file_path))[0]
output_image_path = os.path.join(self.output_dir.get(), f"{base_name}_{width}x{height}.png")
img.save(output_image_path, 'PNG')
# 创建临时文件
with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as tmpfile:
temp_path = tmpfile.name
img.save(temp_path, 'PNG')
# 转换为LVGL C数组
lvgl_img = LVGLImage().from_png(temp_path, cf=cf)
output_c_path = os.path.join(self.output_dir.get(), f"{base_name}.c")
lvgl_img.to_c_array(output_c_path, compress=compress)
success_count += 1
os.unlink(temp_path)
print(f"成功转换: {base_name}.c\n")
except Exception as e:
print(f"转换失败: {str(e)}\n")
print(f"转换完成! 成功 {success_count}/{total_files} 个文件\n")
if __name__ == "__main__":
root = tk.Tk()
app = ImageConverterApp(root)
root.mainloop()

View File

@@ -75,8 +75,14 @@ def get_board_name(folder):
def read_binary(dir_path):
merged_bin_path = os.path.join(dir_path, "merged-binary.bin")
data = open(merged_bin_path, "rb").read()[0x100000:]
if data[0] != 0xE9:
merged_bin_data = open(merged_bin_path, "rb").read()
# find app partition
if merged_bin_data[0x100000] == 0xE9:
data = merged_bin_data[0x100000:]
elif merged_bin_data[0x200000] == 0xE9:
data = merged_bin_data[0x200000:]
else:
print(dir_path, "is not a valid image")
return
# get flash size

View File

@@ -13,7 +13,7 @@ CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions.csv"
CONFIG_PARTITION_TABLE_OFFSET=0x8000
ESP_TASK_WDT_TIMEOUT_S=10
CONFIG_ESP_TASK_WDT_TIMEOUT_S=10
CONFIG_FREERTOS_GENERATE_RUN_TIME_STATS=y
CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS=y

View File

@@ -18,3 +18,5 @@ CONFIG_ESP32S3_DATA_CACHE_64KB=y
CONFIG_ESP32S3_DATA_CACHE_LINE_64B=y
CONFIG_SR_WN_WN9_NIHAOXIAOZHI_TTS=y
CONFIG_ESP_SYSTEM_EVENT_TASK_STACK_SIZE=4096