Code comments Removed Update
This commit is contained in:
+47
-95
@@ -1,93 +1,74 @@
|
||||
# AI Voice Services Backend
|
||||
<div align="center">
|
||||
<h1>HushMap: AI Services API</h1>
|
||||
<p>
|
||||
<a href="https://fastapi.tiangolo.com/"><img src="https://img.shields.io/badge/FastAPI-009688?style=for-the-badge&logo=fastapi&logoColor=white" alt="FastAPI"></a>
|
||||
<a href="https://python.org"><img src="https://img.shields.io/badge/Python_3.9+-3776AB?style=for-the-badge&logo=python&logoColor=white" alt="Python"></a>
|
||||
<img src="https://img.shields.io/badge/Ultralytics-YOLOv8-FF0000?style=for-the-badge" alt="YOLOv8 Vision">
|
||||
<img src="https://img.shields.io/badge/Whisper-STT-4A90E2?style=for-the-badge" alt="Whisper">
|
||||
</p>
|
||||
<p><i>The central nervous system linking physical M5GO devices, external Computer Vision tensors, and Conversational NLP APIs synchronously.</i></p>
|
||||
</div>
|
||||
|
||||
This directory contains the FastAPI backend for the AI Voice Agent, facilitating communication between the M5GO device, Terp AI, and ElevenLabs.
|
||||
---
|
||||
|
||||
## Setup Instructions
|
||||
|
||||
### Prerequisites
|
||||
1. **Python 3.9+** is recommended.
|
||||
2. **FFmpeg** must be installed on the system to handle audio format conversions (MP3 to 16-bit 16kHz PCM).
|
||||
- On Ubuntu/Debian: `sudo apt install ffmpeg`
|
||||
- On macOS: `brew install ffmpeg`
|
||||
- On Windows: Download from the [FFmpeg website](https://ffmpeg.org/download.html) and add to PATH.
|
||||
1. **Python 3.9+** is strictly recommended to support asynchronous typing paradigms.
|
||||
2. **FFmpeg** must be successfully registered onto your OS PATH environments. This engine handles the core conversions decoding MP3 output arrays into 16-bit, 16kHz Mono arrays natively required for browser contexts:
|
||||
- **Ubuntu/Debian**: `sudo apt install ffmpeg`
|
||||
- **macOS**: `brew install ffmpeg`
|
||||
- **Windows**: Install globally via the [FFmpeg website](https://ffmpeg.org/download.html).
|
||||
|
||||
### Installation
|
||||
### Environment Initialization
|
||||
|
||||
1. Navigate to the `ai_services` directory.
|
||||
2. (Optional but recommended) Create a virtual environment:
|
||||
```bash
|
||||
python -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
```
|
||||
3. Install the required Python packages:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
Bootstrap the virtual environment and initialize project dependencies:
|
||||
|
||||
### Configuration
|
||||
```bash
|
||||
cd backend
|
||||
python -m venv venv
|
||||
source venv/bin/activate # Windows: .\venv\Scripts\activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Update the `.env` file in this directory with your credentials:
|
||||
### Configuration Tokens
|
||||
|
||||
Provide runtime keys securely targeting TerpAI context queues and ElevenLabs synthesized avatars within a `.env` dotfile:
|
||||
|
||||
```ini
|
||||
ELEVENLABS_API_KEY=sk_...
|
||||
ELEVENLABS_VOICE_ID=JBFqnCBsd6RMkjVDRZzb
|
||||
TERP_AI_BEARER_TOKEN=eyJhbGciOiJSUz...
|
||||
TERP_AI_CONVERSATION_ID=37fa27cc-542a-c8a8-9c31-9d1954fdc1d2
|
||||
TERP_AI_CONVERSATION_ID=37fa27cc-...
|
||||
MONGODB_URI=mongodb+srv://...
|
||||
```
|
||||
|
||||
## Running the Server
|
||||
|
||||
Start the FastAPI application using Uvicorn:
|
||||
To invoke the engine, simply execute Uvicorn across your `0.0.0.0` loopback:
|
||||
|
||||
```bash
|
||||
uvicorn server:app --host 0.0.0.0 --port 8000
|
||||
```
|
||||
This will start the server and make it accessible on your local network on port 8000.
|
||||
|
||||
## WebSocket Endpoints
|
||||
---
|
||||
|
||||
### `/ws/voice`
|
||||
## Gateway Pipelines
|
||||
|
||||
This is the primary WebSocket endpoint used by the M5GO device for real-time voice communication.
|
||||
### Full-Duplex Subroutines (`/ws/voice`)
|
||||
|
||||
**Protocol Flow:**
|
||||
This WebSocket proxy establishes a fully integrated multi-turn communication bridge seamlessly interacting between Edge node Hardware APIs (ESP32/M5GO/Browsers) and NLP architectures.
|
||||
|
||||
1. **Connection:** The client establishes a WebSocket connection to `ws://<server_ip>:8000/ws/voice`.
|
||||
2. **Streaming Audio (Client -> Server):** While the user holds the record button, the client continuously sends binary frames containing raw audio data.
|
||||
- **Expected Format:** 16-bit signed integer, 16 kHz, Mono PCM.
|
||||
3. **End of Audio Signal (Client -> Server):** When the user releases the button, the client sends a JSON text frame to signal the end of the recording:
|
||||
```json
|
||||
{
|
||||
"event": "stop_listening"
|
||||
}
|
||||
```
|
||||
4. **Processing (Server):** Upon receiving the `stop_listening` event, the server executes the AI pipeline:
|
||||
- Transcribes the accumulated Int16 PCM audio organically using `faster-whisper`.
|
||||
- Injects a MongoDB aggregate map of the latest 24hr Campus Location noise levels seamlessly into the LLM system prompt.
|
||||
- Sends the transcribed text & location context to the Terp AI conversational endpoint and waits for the full response.
|
||||
- Streams the Terp AI response text directly to ElevenLabs TTS and demands `pcm_16000` via URL flags natively!
|
||||
5. **TTS Endpoint Notification**: The server saves the TTS audio buffer and pushes a JSON:
|
||||
```json
|
||||
{
|
||||
"event": "tts_ready",
|
||||
"size": 105000
|
||||
}
|
||||
```
|
||||
6. **Audio Callback**: Client queries `GET /api/tts-audio` to play the binary wav response.
|
||||
1. **Int16 Byte Array Exchange**: Devices connect to `ws://<server_ip>:8000/ws/voice` and push raw binary frames asynchronously over the socket.
|
||||
2. **Contextual Augmentation**: The server waits for the `"stop_listening"` payload event to signify a completed audio snippet. That float array is cast through `faster-whisper` and combined seamlessly with real-time `MongoDB` decibel tracking telemetry parameters natively attached into the `TerpAI` user conversation chunk.
|
||||
3. **TTS Pipeline Rendering**: Output predictions are caught instantly, forwarded natively into the `ElevenLabs` TTS interface rendering `pcm_16000` wav codecs, and alerted back down to clients using a `tts_ready` dispatcher.
|
||||
|
||||
## REST Endpoints
|
||||
### Tensor Vision Endpoints (`/api/vision/room-status`)
|
||||
|
||||
### `/api/vision/room-status` (POST)
|
||||
Leveraging OpenCV bindings layered beneath a YOLOv8-driven bounding box topology detector, this `POST` API analyzes raw camera image buffers returning capacity logic natively.
|
||||
|
||||
This endpoint uses a YOLO object detection model to detect people and chairs in a room image, determining if the room is full and pairing the closest chairs to people.
|
||||
|
||||
**Request:**
|
||||
- `file`: (Required) The image file to analyze (e.g., JPEG, PNG) sent as multipart form-data.
|
||||
|
||||
**Response:**
|
||||
Returns a JSON object detailing the room status, counts, and pairings.
|
||||
> [!NOTE]
|
||||
> This API calculates euclidean distances algorithmically detecting adjacent proximities between "person" classifiers and untaken "chair" bounding frames to accurately diagnose available seats inside crowded architectures!
|
||||
|
||||
**Response Output Protocol:**
|
||||
```json
|
||||
{
|
||||
"room_status": "full",
|
||||
@@ -101,45 +82,16 @@ Returns a JSON object detailing the room status, counts, and pairings.
|
||||
"chair_index": 1,
|
||||
"distance": 150.5
|
||||
}
|
||||
],
|
||||
"details": {
|
||||
"people": [ ... ],
|
||||
"chairs": [ ... ]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `/api/study-rooms` (GET)
|
||||
|
||||
Returns a list of all recorded study room data.
|
||||
|
||||
### `/api/study-rooms/history` (GET)
|
||||
|
||||
Returns a list of all recorded study room data from the last 24 hours, sorted by most recent first.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"data": [
|
||||
{
|
||||
"location": {
|
||||
"type": "Point",
|
||||
"coordinates": [-77.3079, 38.8315]
|
||||
},
|
||||
"db": 65.2,
|
||||
"date": "2026-04-12T14:30:00.000Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Client Integration Notes
|
||||
---
|
||||
|
||||
For the ESP32/M5GO hardware client (`m5go/main.py`), ensure you update the `WS_URL` variable to point to the correct internal server IP.
|
||||
## Database Registries
|
||||
|
||||
For the Web Frontend (`VoiceButton.svelte`), it uses standard Web Audio API's `ScriptProcessorNode` to bridge the Float32 arrays strictly into 16-Bit Mono over a dynamic WebSocket tunnel automatically.
|
||||
* `GET /api/study-rooms/history`: Pulls the active global repository of logged architectural noise measurements captured universally within the preceding 24 hours. Data payloads correspond geographically mapping `GeoJSON` nodes to front-end Mapbox topologies.
|
||||
* `GET /api/study-rooms`: Pulls generic unstructured noise lists directly unfiltered from Cosmos bounds.
|
||||
|
||||
```python
|
||||
# In m5go/main.py
|
||||
WS_URL = "ws://192.168.1.100:8000/ws/voice"
|
||||
```
|
||||
> [!IMPORTANT]
|
||||
> The browser frontend strictly configures standard Web Audio API's `ScriptProcessorNode` interfaces routing data synchronously to this backend! Wait to close down pipelines until *after* all WS queues have successfully been delivered.
|
||||
|
||||
Reference in New Issue
Block a user