AI-Powered Hardware Control via Natural Language

Abstract

This project presents a system for controlling physical hardware components using natural language commands, interpreted by a Large Language Model (LLM). The goal is to create an intuitive interface for hardware interaction that abstracts away the need for traditional programming. By interfacing Claude AI with an ESP32-based microcontroller, users can manage devices like LCDs and sensors simply by typing commands in plain English. This project showcases a practical application of AI in the realm of electronics and human-computer interaction.

System Overview

The system is comprised of three main components: a cloud-based AI, a host PC, and a custom USB hardware device.

The workflow is as follows:

1. The user issues a command to the **Claude AI desktop application**.

2. The AI interprets the command's intent and sends a corresponding instruction to the **MCP (Machine Control Protocol) server** running on the host PC.

3. The MCP server translates this instruction and forwards it to the **USB hardware device** via a Python-based USB host interface.

4. The **ESP32 firmware** receives the command and executes the appropriate action, such as writing to the LCD via SPI or reading data from the BME280 sensor via I2C.

A video demonstration of this core functionality is available below:

Implementation Details

1. Firmware (ESP32 USB Device)

The device firmware is developed using the Arduino framework. It leverages the **"TinyUSB" library** to implement a generic vendor-specific USB device, enabling robust communication with the host PC.

-  **Source Code:**

https://github.com/wichayen/AI_experimental/blob/main/arduino/vendor_generic_inout_ili9488_ai.ino

2. Host Software (PC)

The PC runs the server-side logic that bridges the AI with the hardware.

2.1. USB Interface

A Python script using the **"pyusb" library** manages low-level USB communication. This script is responsible for finding the correct USB device and transmitting data packets to it.

-  **Source Code:**

https://github.com/wichayen/AI_experimental/blob/main/mcp_server/simpleio.py

-  **Driver Note:** The **Zadig utility** ([https://zadig.akeo.ie/](https://zadig.akeo.ie/)) is required to install the correct WinUSB driver for the device, allowing `pyusb` to access it.

2.2. MCP Server

The "FastMCP" library** is used to create a lightweight server that exposes the hardware control functions to the AI.

-  **Source Code:**

https://github.com/wichayen/AI_experimental/blob/main/mcp_server/server.py

3. AI Configuration (Claude)

The Claude AI desktop application must be configured to recognize the MCP server as an available tool. This is done by editing the `claude_desktop_config.json` file to define the server endpoint.

-  **Configuration File:**

https://github.com/wichayen/AI_experimental/blob/main/claude_desktop_config.json

**Example Configuration:**

{
    "mcpServers": {
        "usb_device_bot": {
            "command": "uv",
            "args": [
                "--directory",
                "D:/mcp_server/usb_device_bot",
                "run",
                "server.py"
            ]
        }
    }
}

Advanced Demonstration: Flowchart-Based Control

Beyond simple commands, the system can interpret and execute complex, multi-step logic described in a **Mermaid flowchart**. This demonstrates the AI's ability to understand sequential and conditional tasks.

**Example Prompt:**

> Please execute the following flowchart.

The AI will parse this flowchart and generate the corresponding commands to blink an LED three times.

A video of this advanced control is available here: