AI Voice Assistant on an ESP32-S3 with XiaoZhi

You say "Hi ESP, turn on the lamp", an LED on GPIO18 lights up, and a voice tells you it's on. This project builds that with XiaoZhi, an MIT-licensed voice assistant firmware for the ESP32-S3 that supports more than 70 boards, including a breadboard build made from four cheap modules. The chip does the wake word, the microphone and the speaker; speech recognition, the language model and speech synthesis run on a server. At the end you add a tool of your own, so the assistant can answer "is the door open?" from a switch on GPIO17.

What happens when you say "turn on the lamp"

Following one request from start to finish shows what each part of the build is for.

1. Wake word, on the chip. ESP-SR's WakeNet model runs on the ESP32-S3 and listens for "Hi ESP" all the time. Nothing leaves the device until it hears it. This is why the board needs PSRAM: the wake-word engine doesn't fit without it.

2. Handshake. The device opens a WebSocket to its server and introduces itself. "mcp": true tells the server the device has tools it can call:

   {
     "type": "hello",
     "version": 1,
     "features": { "mcp": true },
     "transport": "websocket",
     "audio_params": {
       "format": "opus",
       "sample_rate": 16000,
       "channels": 1,
       "frame_duration": 60
     }
   }

   Copy

   The server answers with its own hello, a session_id, and the format of the audio it will send back: Opus at 24 kHz.

3. Your voice, as Opus frames. The INMP441 microphone is sampled at 16 kHz, compressed with Opus in 60 ms frames (about 17 per second), and each frame goes out as one binary WebSocket message.

4. Speech to text, on the server. The server sends back what it heard, so the OLED can show it: {"type": "stt", "text": "Turn on the lamp"}.

5. The model calls a device tool. The language model has the device's tool list (more on that below) and decides that self.lamp.turn_on does what you asked. The server forwards the call to the device as a JSON-RPC message:

   {
     "session_id": "…",
     "type": "mcp",
     "payload": {
       "jsonrpc": "2.0",
       "method": "tools/call",
       "params": { "name": "self.lamp.turn_on", "arguments": {} },
       "id": 3
     }
   }

   Copy

   The firmware runs the matching C++ function, which sets GPIO18 HIGH and replies with {"content": [{"type": "text", "text": "true"}], "isError": false}.

6. The answer, as speech. The server sends {"type": "tts", "state": "start"}, then Opus audio frames that the device decodes and plays through the MAX98357A, then {"type": "tts", "state": "stop"}. The device starts listening again, so you can follow up without repeating the wake word.

The device needs Wi-Fi, a server address, and permission to use that server. The first boot sets up all three.

Wiring

Everything below comes from the firmware's own pin map for this build (main/boards/bread-compact-wifi/config.h), so the stock firmware works with it unchanged. Power the board from the USB-C port labelled UART.

Microphone (INMP441)

Amplifier (MAX98357A)

The speaker goes to the amplifier's + and − screw terminals. Unconnected, SD leaves the amplifier on and GAIN gives 9 dB.

Display, buttons and lamp

The buttons need no resistors: the firmware treats a button pulled to GND as pressed. The board's own BOOT button (GPIO0) is the fourth button.

The firmware also drives a WS2812 status LED on GPIO48, the RGB LED on v1.0 DevKitC-1 boards. v1.1 boards moved theirs to GPIO38, so on those it stays dark. Nothing else is affected.

Install ESP-IDF 6

XiaoZhi recommends ESP-IDF's release/v6.1 branch, not the v6.1 tag: the project reports an ESP32-C3 Wi-Fi start-up failure with the tag. On Linux or macOS:

mkdir -p ~/esp && cd ~/esp
git clone -b release/v6.1 --recursive https://github.com/espressif/esp-idf.git
cd esp-idf
./install.sh esp32s3
. ./export.sh

Run . ~/esp/esp-idf/export.sh in every new terminal before building. On Windows, use Espressif's installer or the ESP-IDF extension for VS Code and pick the same branch. Linux builds are noticeably faster and have fewer USB driver problems.

Build and flash the firmware

git clone https://github.com/78/xiaozhi-esp32.git
cd xiaozhi-esp32
python scripts/build.py bread-compact-wifi --name bread-compact-wifi
idf.py -p /dev/ttyUSB0 flash monitor

build.py reads the board's config.json, sets the target to esp32s3, applies the board's settings and builds. --name picks the variant: bread-compact-wifi for the 0.91" 128×32 OLED, bread-compact-wifi-128x64 for a 0.96" 128×64 one. The port is /dev/ttyUSB0 or /dev/ttyACM0 on Linux, /dev/cu.usbserial-… on macOS and COM3 or similar on Windows. Leave the monitor running: the first boot is easier to follow with the log.

Language and wake word are build options. English with the "Hi ESP" wake word is the default; python scripts/build.py --list-languages and --list-wake-words show the alternatives, and --language zh-CN switches to Chinese with "你好小智".

First boot: Wi-Fi and activation

1. Wi-Fi. With no saved network, the device starts a hotspot whose name begins with Xiaozhi-, and the OLED shows it along with the web address to open. A voice prompt says the same. Join the hotspot from your phone, open that address, and pick your network. The ESP32-S3 only has 2.4 GHz Wi-Fi, so a 5 GHz-only network won't appear. The device restarts and connects.
2. Server address. The firmware asks its OTA URL (by default https://api.tenclass.net/xiaozhi/ota/) for firmware updates and where to connect. The answer includes the WebSocket address, and, for a new device, an activation code.
3. Activation. The device shows a six-digit code on the OLED and reads it aloud. Sign in at xiaozhi.me, add a device, and enter the code. Personal accounts can use the Qwen real-time model for free. The console is also where you choose the assistant's language model, voice and personality.

The device remembers all of this. To join a different network later, press BOOT while it is starting up.

Talk to it

• Wake word: say "Hi ESP", then your question. After each reply the device listens again, so you can keep talking; click BOOT to end the conversation.
• Push to talk: hold the talk button (GPIO47) while you speak and release it when you're done. After the reply the device goes idle instead of listening again.
• BOOT: a click starts or ends a conversation.
• Volume: a click changes it by 10; a long press on up sets 100, a long press on down mutes.

Now try "turn on the lamp", "is the lamp on?" and "set the volume to 40". All three work because the firmware exposes them as tools.

How the model knows about the lamp: MCP tools

When the device connects, the server sends it tools/list and gets back every tool the firmware registered: a name, a description written for the model, and a JSON Schema of the arguments. The lamp is a 48-line header that ships with the firmware (main/boards/common/lamp_controller.h). This is one of its three tools:

mcp_server.AddTool("self.lamp.turn_on", "Turn on the lamp", PropertyList(),
    [this](const PropertyList& properties) -> ReturnValue {
        power_ = true;
        gpio_set_level(gpio_num_, 1);
        return true;
    });

The board class creates it with static LampController lamp(LAMP_GPIO);, and LAMP_GPIO is 18 in the board's config.h. Tools that take arguments declare them as typed properties. The volume tool, which every board gets, takes an integer from 0 to 100:

AddTool("self.audio_speaker.set_volume",
        "Set the volume of the audio speaker. If the current volume is unknown, you must call "
        "`self.get_device_status` tool first and then call this tool.",
        PropertyList({Property("volume", kPropertyTypeInteger, 0, 100)}),
        [&board](const PropertyList& properties) -> ReturnValue {
            board.GetAudioCodec()->SetOutputVolume(properties["volume"].value<int>());
            return true;
        });

This is what the model receives for it in tools/list. The range comes from Property(…, 0, 100), and volume is required because it has no default:

{
  "name": "self.audio_speaker.set_volume",
  "description": "Set the volume of the audio speaker. If the current volume is unknown, you must call `self.get_device_status` tool first and then call this tool.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "volume": { "type": "integer", "minimum": 0, "maximum": 100 }
    },
    "required": ["volume"]
  }
}

The description is the only thing the model reads to decide when to call a tool, so it is effectively part of the prompt. The volume tool's description tells the model to read the current volume first. That is why "turn it up a bit" works: the model calls self.get_device_status, sees 60, then calls set_volume with 70.

A tool can take booleans, integers (with an optional range and default) and strings. It returns true/false, a number, or a string, and a string can hold JSON, as self.lamp.get_state does with {"power": true}.

Add your own tool: is the door open?

A tool that reads a sensor is as short as one that switches a pin. This one reports a switch on GPIO17: a reed switch on a door, or any switch you can flip by hand.

1. Copy the board and give the copy its own identity:

cp -r main/boards/bread-compact-wifi main/boards/my-breadboard

In main/boards/my-breadboard/config.json, change both names. Lowercase letters, digits, dots and hyphens only:

{
  "type": "my-breadboard",
  "target": "esp32s3",
  "builds": [{ "name": "my-breadboard", "sdkconfig_append": [] }]
}

2. Fix the display height in config.h. The OLED size menu only appears for the stock breadboard boards, so in a copy the #if CONFIG_OLED_… block falls through to #error "OLED display type is not selected". Replace the whole block with the height of your display, and add the switch pin:

#define DISPLAY_HEIGHT  32   // 64 for a 0.96" 128×64 OLED
 
#define DOOR_GPIO GPIO_NUM_17

3. Write the tool in a new file, main/boards/my-breadboard/door_sensor.h:

#ifndef DOOR_SENSOR_H
#define DOOR_SENSOR_H
 
#include <driver/gpio.h>
#include "mcp_server.h"
 
class DoorSensor {
private:
    gpio_num_t gpio_num_;
 
public:
    DoorSensor(gpio_num_t gpio_num) : gpio_num_(gpio_num) {
        gpio_config_t config = {
            .pin_bit_mask = (1ULL << gpio_num_),
            .mode = GPIO_MODE_INPUT,
            .pull_up_en = GPIO_PULLUP_ENABLE,
            .pull_down_en = GPIO_PULLDOWN_DISABLE,
            .intr_type = GPIO_INTR_DISABLE,
        };
        ESP_ERROR_CHECK(gpio_config(&config));
 
        auto& mcp_server = McpServer::GetInstance();
        mcp_server.AddTool("self.door.get_state",
            "Whether the front door is open or closed. "
            "Use this when the user asks about the door.",
            PropertyList(),
            [this](const PropertyList& properties) -> ReturnValue {
                // The switch connects the pin to GND when the door is closed;
                // the pull-up reads HIGH when it opens.
                bool open = gpio_get_level(gpio_num_) == 1;
                return open ? "{\"door\": \"open\"}" : "{\"door\": \"closed\"}";
            });
    }
};
 
#endif  // DOOR_SENSOR_H

It follows the lamp controller: configure the pin in the constructor, then register the tool with a callback that captures this. The internal pull-up means the switch needs no resistor.

4. Create it with the lamp. In main/boards/my-breadboard/compact_wifi_board.cc, add #include "door_sensor.h" next to #include "lamp_controller.h", and one line in InitializeTools():

void InitializeTools() {
    static LampController lamp(LAMP_GPIO);
    static DoorSensor door(DOOR_GPIO);
}

static matters: the tool callback captures this, so the object has to live as long as the firmware runs. A plain local would be destroyed when InitializeTools() returns, and the first "is the door open?" would read freed memory.

5. Register the board with the build. In main/Kconfig.projbuild, inside choice BOARD_TYPE, add an entry next to BOARD_TYPE_BREAD_COMPACT_WIFI:

    config BOARD_TYPE_MY_BREADBOARD
        bool "My breadboard (door sensor)"
        depends on IDF_TARGET_ESP32S3

In main/CMakeLists.txt, add a branch right after the bread-compact-wifi one, with the same fonts:

elseif(CONFIG_BOARD_TYPE_MY_BREADBOARD)
    set(BOARD_DIR "my-breadboard")
    set(BUILTIN_TEXT_FONT font_noto_sans_basic_14_1)
    set(BUILTIN_ICON_FONT font_material_symbols_14_1)

build.py finds a board by looking for the CMakeLists.txt branch whose BOARD_DIR matches the directory name, and from it the Kconfig option to turn on.

6. Build, flash, ask.

python scripts/build.py my-breadboard --name my-breadboard
idf.py -p /dev/ttyUSB0 flash monitor

Wi-Fi settings survive the flash. Ask "is the door open?", flip the switch, and ask again. The same pattern covers any sensor or actuator: the model only needs a tool name, a description that says when to use it, and a callback that returns something it can read.

Running your own server

Everything so far uses the public xiaozhi.me server. The protocol is open, and several community servers implement it. The most complete is xiaozhi-esp32-server (Python, with a web console), which lets you choose your own speech recognition, language model and voice providers. To point the device at it, change Xiaozhi Assistant → Default OTA URL in idf.py menuconfig to your server's OTA address, then build and flash again. The device gets the WebSocket address from that URL, as in step 2 of the first boot.

Troubleshooting

• "Hi ESP" does nothing, but the buttons work. Wake-word detection is off. Rebuild with scripts/build.py, which enables it (see the warning in "Build and flash"). It also needs PSRAM, so check that the module marking says N16R8.
• The board resets in a loop with PSRAM or flash-size errors in the log. The firmware expects 16 MB of flash and 8 MB of octal PSRAM. N8R2 and other variants have less flash or quad PSRAM and can't run this build unchanged.
• The display and speaker work, but it never understands you. The INMP441 microphone isn't delivering audio. Check that its L/R pin is tied to GND and that WS and SCK aren't swapped (GPIO4 and GPIO5).
• It understands you, but you hear nothing. Check the amplifier's VIN (5V, not 3V3, for full volume), then DIN, BCLK and LRC on GPIO7, 15 and 16, then the volume: a long press on volume-down mutes it.
• The OLED stays blank. SDA and SCL are swapped, or the display is a 128×64 module on the 128×32 build (use --name bread-compact-wifi-128x64).
• "Turn on the lamp" gets a reply but no light. The LED is backwards (the long leg goes toward GPIO18) or on the wrong pin. The answer itself shows that the tool call reached the device.

Next steps

• Add a tool with an argument: a servo angle from 0 to 180 or a fan speed from 0 to 100, declared with Property("angle", kPropertyTypeInteger, 0, 180).
• Put sensors from other projects behind tools. The BME280 weather station becomes "what's the temperature in here?" with one tool that returns the reading as JSON.
• Read docs/websocket.md and docs/mcp-protocol.md in the XiaoZhi repository for the full message set, including abort to interrupt a reply and the MQTT + UDP transport.