HushMap/backend/README.md

AI Voice Services Backend

This directory contains the FastAPI backend for the AI Voice Agent, facilitating communication between the M5GO device, Terp AI, and ElevenLabs.

Setup Instructions

Prerequisites

1. Python 3.9+ is recommended.
2. FFmpeg must be installed on the system to handle audio format conversions (MP3 to 16-bit 16kHz PCM).
   • On Ubuntu/Debian: sudo apt install ffmpeg
   • On macOS: brew install ffmpeg
   • On Windows: Download from the FFmpeg website and add to PATH.

Installation

1. Navigate to the ai_services directory.
2. (Optional but recommended) Create a virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   

3. Install the required Python packages:

   pip install -r requirements.txt
   

Configuration

Update the .env file in this directory with your credentials:

ELEVENLABS_API_KEY=sk_...
ELEVENLABS_VOICE_ID=JBFqnCBsd6RMkjVDRZzb
TERP_AI_BEARER_TOKEN=eyJhbGciOiJSUz...
TERP_AI_CONVERSATION_ID=37fa27cc-542a-c8a8-9c31-9d1954fdc1d2
MONGODB_URI=mongodb+srv://...

Running the Server

Start the FastAPI application using Uvicorn:

uvicorn server:app --host 0.0.0.0 --port 8000

This will start the server and make it accessible on your local network on port 8000.

WebSocket Endpoints

/ws/voice

This is the primary WebSocket endpoint used by the M5GO device for real-time voice communication.

Protocol Flow:

1. Connection: The client establishes a WebSocket connection to ws://<server_ip>:8000/ws/voice.
2. Streaming Audio (Client -> Server): While the user holds the record button, the client continuously sends binary frames containing raw audio data.
   • Expected Format: 16-bit signed integer, 16 kHz, Mono PCM.
3. End of Audio Signal (Client -> Server): When the user releases the button, the client sends a JSON text frame to signal the end of the recording:

   {
       "event": "stop_listening"
   }
   

4. Processing (Server): Upon receiving the stop_listening event, the server executes the AI pipeline:
   • Transcribes the accumulated Int16 PCM audio organically using faster-whisper.
   • Injects a MongoDB aggregate map of the latest 24hr Campus Location noise levels seamlessly into the LLM system prompt.
   • Sends the transcribed text & location context to the Terp AI conversational endpoint and waits for the full response.
   • Streams the Terp AI response text directly to ElevenLabs TTS and demands pcm_16000 via URL flags natively!
5. TTS Endpoint Notification: The server saves the TTS audio buffer and pushes a JSON:

   {
       "event": "tts_ready",
       "size": 105000
   }
   

6. Audio Callback: Client queries GET /api/tts-audio to play the binary wav response.

REST Endpoints

/api/vision/room-status (POST)

This endpoint uses a YOLO object detection model to detect people and chairs in a room image, determining if the room is full and pairing the closest chairs to people.

Request:

• file: (Required) The image file to analyze (e.g., JPEG, PNG) sent as multipart form-data.

Response: Returns a JSON object detailing the room status, counts, and pairings.

{
  "room_status": "full",
  "counts": {
    "people": 2,
    "chairs": 2
  },
  "pairs": [
    {
      "person_index": 0,
      "chair_index": 1,
      "distance": 150.5
    }
  ],
  "details": {
    "people": [ ... ],
    "chairs": [ ... ]
  }
}

/api/study-rooms (GET)

Returns a list of all recorded study room data.

/api/study-rooms/history (GET)

Returns a list of all recorded study room data from the last 24 hours, sorted by most recent first.

Response:

{
  "data": [
    {
      "location": {
        "type": "Point",
        "coordinates": [-77.3079, 38.8315]
      },
      "db": 65.2,
      "date": "2026-04-12T14:30:00.000Z"
    }
  ]
}

Client Integration Notes

For the ESP32/M5GO hardware client (m5go/main.py), ensure you update the WS_URL variable to point to the correct internal server IP.

For the Web Frontend (VoiceButton.svelte), it uses standard Web Audio API's ScriptProcessorNode to bridge the Float32 arrays strictly into 16-Bit Mono over a dynamic WebSocket tunnel automatically.

# In m5go/main.py
WS_URL = "ws://192.168.1.100:8000/ws/voice"