Building an AI voice assistant on a bare ESP32: integrating Whisper, GPT, and TTS

Alex: Building an AI Voice Assistant on a Bare ESP32

Alex is a fully DIY AI voice assistant built on an ordinary ESP32 development board—no cloud speaker relay service, no dedicated hardware, and no Raspberry Pi. It captures sound through an I2S microphone, transcribes speech using OpenAI's Whisper API, generates real answers with GPT, and plays the replies out loud through a wired amplifier and speaker. The whole process happens in real time, paired with a small OLED display that dynamically shows lively "puppy eyes" that react as the assistant listens, thinks, and speaks.

This article walks through the project's build process from scratch in detail, covering the implementation details of every stage, including the actual debugging experiences—because the difficulty of the project matters just as much as the final result.

Hardware used:

- ESP32 development board (ESP32-D0WD-V3, standard dev module, no PSRAM)

- INMP441 — I2S MEMS microphone

- MAX98357A — I2S 3W Class D amplifier breakout board

- 3W speaker

- SSD1306 OLED display — 0.96 inch, 128x64 resolution, I2C interface

Wiring configuration

Note: supplying dedicated 5V power to the amplifier (rather than the ESP32's own 5V supply) is a deliberate choice—sharing a power supply with WiFi's instantaneous current peaks causes audio anomalies at the start of recording.

Part 1: The Core Voice Processing Pipeline

The foundation of the project is a five-step loop:

• Listen — The ESP32 continuously samples the I2S microphone, detecting whether there is a volume spike that persists above a threshold. Several consecutive loud sound segments are required to trigger it, so brief noise spikes (such as a door closing or a chair creaking) do not falsely trigger recording.

• Record — Once triggered, recording begins and continues until natural silence is detected, then stops.

• Transcribe — The captured audio segment is packaged into a WAV file (built manually, including the complete header information) and sent to OpenAI's Whisper API.

• Think — The transcribed text is sent to GPT via the Chat Completions API to obtain a real, conversational answer.

• Voice output — The answer is generated by OpenAI's TTS API, and the raw PCM audio is streamed directly over I2S to the MAX98357A amplifier, without waiting for the full response to download first.

Some known Whisper "hallucination" phrases (such as "thanks for watching") have been filtered out, because nearly silent or noise-only segments tend to cause these to appear persistently instead of genuinely outputting no text at all.

Part 2: Adding an OLED "Face"

To make the assistant look more alive rather than just a black box reacting to audio, we added an SSD1306 OLED display that expresses the assistant's behavior state through dynamic dog-eye animations:

Key design decision: The OLED runs independently in its own FreeRTOS task, pinned to the ESP32's second core. This is crucial because the Whisper, GPT, and TTS calls are all blocking HTTP requests, and each request can take 1 to 3 seconds. If screen updates happened in the same task as these calls, the animation would freeze during the calls and could not play smoothly. Running the main pipeline on core 1 while running the subroutine independently on core 0 ensures the face stays active throughout the entire cycle, communicating through a shared state protected by a small-scope mutex.

Problems encountered and how they were solved

1. Compile error: setBufferSizes is not a member of WiFiClientSecure

An early attempt to fix a memory problem (see below) used the client.setBufferSizes(1024, 512) method—this method exists in older versions of the Arduino-ESP32 core, but has been removed in the 3.x version of the core. The 3.x version replaced the original BearSSL-based WiFiClientSecure with a new mbedtls-based implementation (NetworkClientSecure), which no longer provides runtime TLS buffer control. Fix: the relevant calls were removed entirely, and the underlying memory problem was solved in another way (see below).

2. "Direct connection failed"—TLS handshake occasionally fails

The first HTTPS request of the day (a standalone TTS test at startup) would fail, even though the Wi-Fi itself was connected properly. Diagnostics showed that the ESP32 had plenty of total available heap, but not enough contiguous available heap—the TLS handshake requires a large, contiguous block of memory, and fragmentation caused by other allocations (such as microphone sampling buffers, JSON parsing, and Wi-Fi stack buffers) left less than 40KB of contiguous memory available, right at the edge of what mbedtls requires. Solution: reducing MAX_RECORD_SECONDS from 3 seconds to 2 seconds shrank the audio upload buffer and freed about 32KB of contiguous heap space. This alone was enough to make the TLS connection reliable.

3. OLED completely blank

This was the longest debugging process in the project, and it took several misdiagnoses before the real cause was found:

• First, the wiring was suspected—but after heartbeat prints confirmed that the display task was running normally and that display.begin() returned success (i.e., a real I2C handshake), this possibility was ruled out.

• Next, the wrong OLED controller was suspected (SH1106 confused with SSD1306, a common mismatch in cheap modules)—but after confirming that the module was a genuine 0.96-inch SSD1306, the problem still persisted.

• Then a damaged GPIO pin was suspected—a physical continuity check confirmed that the wiring itself was properly connected end to end.

• Finally, the real cause was found: the OLED module needed to use the SSD1306_EXTERNALVCC power mode, rather than the SSD1306_SWITCHCAPVCC power mode passed by default in the display.begin() function. Because the mode was set incorrectly, the I2C handshake and all initialization commands executed successfully, but the display never received the correct internal drive voltage to light up. Solution: after changing the power mode constant to SSD1306_EXTERNALVCC, the problem was completely resolved.

4. When testing the OLED wiring, I tried using GPIO 22 as an I2C pin, but that pin was already physically soldered to the amplifier's DIN line on the assembled PCB, which would cause the two peripherals to contend for the same line. Solution: I moved the OLED's I2C clock line to a truly free pin (GPIO 19).

5. TTS playback sounded too fast

Once the pipeline was fully working end to end, the AI's voice replies sounded as if they were being played too fast at normal pitch—this ruled out a sample rate mismatch and instead pointed to the default speaking rate of the TTS voice. Solution: I added a "speed" parameter of 0.85 to the TTS API request, slowing the voice down to a more natural speaking rate.

6. I had to speak very close to the microphone

The INMP441 microphone has high sensitivity, but the raw sample values were not amplified during processing, while the speaker path already had gain applied in advance. Fix: before trigger detection and saving/transcribing audio, I added software gain (MIC_GAIN) to each microphone sample, thereby increasing sensitivity and ensuring that sound at a normal speaking distance could be captured reliably.

What this project demonstrates

Beyond the final result, this build also involves several fundamental embedded systems principles that are easily overlooked in advanced tutorials:

- Manually configuring the I2S interface for recording (microphone) and streaming playback (speaker), including construction of the raw WAV header

- Streaming HTTP responses (chunked transfer encoding) and playing audio in real time, rather than buffering the entire file

- Distinguishing heap fragmentation from total available heap space, making clear that they are two different problems

- Dual-core FreeRTOS task design, so that the user interface remains responsive when the main task is blocked by slow network calls

- A systematic hardware debugging approach—isolating variables one by one (wiring, software, power, or component defects) rather than guessing

Note: Before uploading the code, go to "Tools" -> "Partition Scheme" and update it to the option shown below.

This article was compiled from hackster.io