Hackathons/HushMap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Code comments Removed Update

Browse Source
This commit is contained in:
SirBlob
2026-04-12 08:49:36 -04:00
parent e6c7ed230b
commit 96272aedb7
22 changed files with 677 additions and 433 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/README.md
+47 -95
View File
@@ -1,93 +1,74 @@
# AI Voice Services Backend
<div align="center">
<h1>HushMap: AI Services API</h1>
<p>
<a href="https://fastapi.tiangolo.com/"><img src="https://img.shields.io/badge/FastAPI-009688?style=for-the-badge&logo=fastapi&logoColor=white" alt="FastAPI"></a>
<a href="https://python.org"><img src="https://img.shields.io/badge/Python_3.9+-3776AB?style=for-the-badge&logo=python&logoColor=white" alt="Python"></a>
<img src="https://img.shields.io/badge/Ultralytics-YOLOv8-FF0000?style=for-the-badge" alt="YOLOv8 Vision">
<img src="https://img.shields.io/badge/Whisper-STT-4A90E2?style=for-the-badge" alt="Whisper">
</p>
<p><i>The central nervous system linking physical M5GO devices, external Computer Vision tensors, and Conversational NLP APIs synchronously.</i></p>
</div>
This directory contains the FastAPI backend for the AI Voice Agent, facilitating communication between the M5GO device, Terp AI, and ElevenLabs.
---
## Setup Instructions
### Prerequisites
1. **Python 3.9+** is recommended.
2. **FFmpeg** must be installed on the system to handle audio format conversions (MP3 to 16-bit 16kHz PCM).
- On Ubuntu/Debian: `sudo apt install ffmpeg`
- On macOS: `brew install ffmpeg`
- On Windows: Download from the [FFmpeg website](https://ffmpeg.org/download.html) and add to PATH.
1. **Python 3.9+** is strictly recommended to support asynchronous typing paradigms.
2. **FFmpeg** must be successfully registered onto your OS PATH environments. This engine handles the core conversions decoding MP3 output arrays into 16-bit, 16kHz Mono arrays natively required for browser contexts:
- **Ubuntu/Debian**: `sudo apt install ffmpeg`
- **macOS**: `brew install ffmpeg`
- **Windows**: Install globally via the [FFmpeg website](https://ffmpeg.org/download.html).
### Installation
### Environment Initialization
1. Navigate to the `ai_services` directory.
2. (Optional but recommended) Create a virtual environment:
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
3. Install the required Python packages:
```bash
pip install -r requirements.txt
```
Bootstrap the virtual environment and initialize project dependencies:
### Configuration
```bash
cd backend
python -m venv venv
source venv/bin/activate # Windows: .\venv\Scripts\activate
pip install -r requirements.txt
```
Update the `.env` file in this directory with your credentials:
### Configuration Tokens
Provide runtime keys securely targeting TerpAI context queues and ElevenLabs synthesized avatars within a `.env` dotfile:
```ini
ELEVENLABS_API_KEY=sk_...
ELEVENLABS_VOICE_ID=JBFqnCBsd6RMkjVDRZzb
TERP_AI_BEARER_TOKEN=eyJhbGciOiJSUz...
TERP_AI_CONVERSATION_ID=37fa27cc-542a-c8a8-9c31-9d1954fdc1d2
TERP_AI_CONVERSATION_ID=37fa27cc-...
MONGODB_URI=mongodb+srv://...
```
## Running the Server
Start the FastAPI application using Uvicorn:
To invoke the engine, simply execute Uvicorn across your `0.0.0.0` loopback:
```bash
uvicorn server:app --host 0.0.0.0 --port 8000
```
This will start the server and make it accessible on your local network on port 8000.
## WebSocket Endpoints
---
### `/ws/voice`
## Gateway Pipelines
This is the primary WebSocket endpoint used by the M5GO device for real-time voice communication.
### Full-Duplex Subroutines (`/ws/voice`)
**Protocol Flow:**
This WebSocket proxy establishes a fully integrated multi-turn communication bridge seamlessly interacting between Edge node Hardware APIs (ESP32/M5GO/Browsers) and NLP architectures.
1. **Connection:** The client establishes a WebSocket connection to `ws://<server_ip>:8000/ws/voice`.
2. **Streaming Audio (Client -> Server):** While the user holds the record button, the client continuously sends binary frames containing raw audio data.
- **Expected Format:** 16-bit signed integer, 16 kHz, Mono PCM.
3. **End of Audio Signal (Client -> Server):** When the user releases the button, the client sends a JSON text frame to signal the end of the recording:
```json
{
"event": "stop_listening"
}
```
4. **Processing (Server):** Upon receiving the `stop_listening` event, the server executes the AI pipeline:
- Transcribes the accumulated Int16 PCM audio organically using `faster-whisper`.
- Injects a MongoDB aggregate map of the latest 24hr Campus Location noise levels seamlessly into the LLM system prompt.
- Sends the transcribed text & location context to the Terp AI conversational endpoint and waits for the full response.
- Streams the Terp AI response text directly to ElevenLabs TTS and demands `pcm_16000` via URL flags natively!
5. **TTS Endpoint Notification**: The server saves the TTS audio buffer and pushes a JSON:
```json
{
"event": "tts_ready",
"size": 105000
}
```
6. **Audio Callback**: Client queries `GET /api/tts-audio` to play the binary wav response.
1. **Int16 Byte Array Exchange**: Devices connect to `ws://<server_ip>:8000/ws/voice` and push raw binary frames asynchronously over the socket.
2. **Contextual Augmentation**: The server waits for the `"stop_listening"` payload event to signify a completed audio snippet. That float array is cast through `faster-whisper` and combined seamlessly with real-time `MongoDB` decibel tracking telemetry parameters natively attached into the `TerpAI` user conversation chunk.
3. **TTS Pipeline Rendering**: Output predictions are caught instantly, forwarded natively into the `ElevenLabs` TTS interface rendering `pcm_16000` wav codecs, and alerted back down to clients using a `tts_ready` dispatcher.
## REST Endpoints
### Tensor Vision Endpoints (`/api/vision/room-status`)
### `/api/vision/room-status` (POST)
Leveraging OpenCV bindings layered beneath a YOLOv8-driven bounding box topology detector, this `POST` API analyzes raw camera image buffers returning capacity logic natively.
This endpoint uses a YOLO object detection model to detect people and chairs in a room image, determining if the room is full and pairing the closest chairs to people.
**Request:**
- `file`: (Required) The image file to analyze (e.g., JPEG, PNG) sent as multipart form-data.
**Response:**
Returns a JSON object detailing the room status, counts, and pairings.
> [!NOTE]
> This API calculates euclidean distances algorithmically detecting adjacent proximities between "person" classifiers and untaken "chair" bounding frames to accurately diagnose available seats inside crowded architectures!
**Response Output Protocol:**
```json
{
"room_status": "full",
@@ -101,45 +82,16 @@ Returns a JSON object detailing the room status, counts, and pairings.
"chair_index": 1,
"distance": 150.5
}
],
"details": {
"people": [ ... ],
"chairs": [ ... ]
}
}
```
### `/api/study-rooms` (GET)
Returns a list of all recorded study room data.
### `/api/study-rooms/history` (GET)
Returns a list of all recorded study room data from the last 24 hours, sorted by most recent first.
**Response:**
```json
{
"data": [
{
"location": {
"type": "Point",
"coordinates": [-77.3079, 38.8315]
},
"db": 65.2,
"date": "2026-04-12T14:30:00.000Z"
}
]
}
```
## Client Integration Notes
---
For the ESP32/M5GO hardware client (`m5go/main.py`), ensure you update the `WS_URL` variable to point to the correct internal server IP.
## Database Registries
For the Web Frontend (`VoiceButton.svelte`), it uses standard Web Audio API's `ScriptProcessorNode` to bridge the Float32 arrays strictly into 16-Bit Mono over a dynamic WebSocket tunnel automatically.
* `GET /api/study-rooms/history`: Pulls the active global repository of logged architectural noise measurements captured universally within the preceding 24 hours. Data payloads correspond geographically mapping `GeoJSON` nodes to front-end Mapbox topologies.
* `GET /api/study-rooms`: Pulls generic unstructured noise lists directly unfiltered from Cosmos bounds.
```python
# In m5go/main.py
WS_URL = "ws://192.168.1.100:8000/ws/voice"
```
> [!IMPORTANT]
> The browser frontend strictly configures standard Web Audio API's `ScriptProcessorNode` interfaces routing data synchronously to this backend! Wait to close down pipelines until *after* all WS queues have successfully been delivered.
backend/emoji_data.py
+276
View File
@@ -0,0 +1,276 @@
"""Data containing all current emoji
Extracted from https://unicode.org/Public/emoji/latest/emoji-test.txt
and https://www.unicode.org/Public/UCD/latest/ucd/emoji/emoji-variation-sequences.txt
See utils/generate_emoji.py
+----------------+-------------+------------------+-------------------+
| Emoji Version | Date | Unicode Version | Data File Comment |
+----------------+-------------+------------------+-------------------+
| N/A | 2010-10-11 | Unicode 6.0 | E0.6 |
| N/A | 2014-06-16 | Unicode 7.0 | E0.7 |
| Emoji 1.0 | 2015-06-09 | Unicode 8.0 | E1.0 |
| Emoji 2.0 | 2015-11-12 | Unicode 8.0 | E2.0 |
| Emoji 3.0 | 2016-06-03 | Unicode 9.0 | E3.0 |
| Emoji 4.0 | 2016-11-22 | Unicode 9.0 | E4.0 |
| Emoji 5.0 | 2017-06-20 | Unicode 10.0 | E5.0 |
| Emoji 11.0 | 2018-05-21 | Unicode 11.0 | E11.0 |
| Emoji 12.0 | 2019-03-05 | Unicode 12.0 | E12.0 |
| Emoji 12.1 | 2019-10-21 | Unicode 12.1 | E12.1 |
| Emoji 13.0 | 2020-03-10 | Unicode 13.0 | E13.0 |
| Emoji 13.1 | 2020-09-15 | Unicode 13.0 | E13.1 |
| Emoji 14.0 | 2021-09-14 | Unicode 14.0 | E14.0 |
| Emoji 15.0 | 2022-09-13 | Unicode 15.0 | E15.0 |
| Emoji 15.1 | 2023-09-12 | Unicode 15.1 | E15.1 |
| Emoji 16.0 | 2024-09-10 | Unicode 16.0 | E16.0 |
http://www.unicode.org/reports/tr51/
"""
__all__ = ['STATUS', 'LANGUAGES']
from typing import Any, Dict, List
component = 1
fully_qualified = 2
minimally_qualified = 3
unqualified = 4
STATUS: Dict[str, int] = {
'component': component,
'fully_qualified': fully_qualified,
'minimally_qualified': minimally_qualified,
'unqualified': unqualified,
}
LANGUAGES: List[str] = [
'en',
'es',
'ja',
'ko',
'pt',
'it',
'fr',
'de',
'fa',
'id',
'zh',
'ru',
'tr',
'ar',
]
EMOJI_DATA: Dict[str, Dict[str, Any]] = {
'\U0001f947': {
'en': ':1st_place_medal:',
'status': fully_qualified,
'E': 3,
'de': ':goldmedaille:',
'es': ':medalla_de_oro:',
'fr': ':médaille_d’or:',
'ja': ':金メダル:',
'ko': ':금메달:',
'pt': ':medalha_de_ouro:',
'it': ':medaglia_d’oro:',
'fa': ':مدال_طلا:',
'id': ':medali_emas:',
'zh': ':金牌:',
'ru': ':золотая_медаль:',
'tr': ':birincilik_madalyası:',
'ar': ':ميدالية_مركز_أول:',
},
'\U0001f948': {
'en': ':2nd_place_medal:',
'status': fully_qualified,
'E': 3,
'de': ':silbermedaille:',
'es': ':medalla_de_plata:',
'fr': ':médaille_d’argent:',
'ja': ':銀メダル:',
'ko': ':은메달:',
'pt': ':medalha_de_prata:',
'it': ':medaglia_d’argento:',
'fa': ':مدال_نقره:',
'id': ':medali_perak:',
'zh': ':银牌:',
'ru': ':серебряная_медаль:',
'tr': ':ikincilik_madalyası:',
'ar': ':ميدالية_مركز_ثان:',
},
'\U0001f949': {
'en': ':3rd_place_medal:',
'status': fully_qualified,
'E': 3,
'de': ':bronzemedaille:',
'es': ':medalla_de_bronce:',
'fr': ':médaille_de_bronze:',
'ja': ':銅メダル:',
'ko': ':동메달:',
'pt': ':medalha_de_bronze:',
'it': ':medaglia_di_bronzo:',
'fa': ':مدال_برنز:',
'id': ':medali_perunggu:',
'zh': ':铜牌:',
'ru': ':бронзовая_медаль:',
'tr': ':üçüncülük_madalyası:',
'ar': ':ميدالية_مركز_ثالث:',
},
'\U0001f18e': {
'en': ':AB_button_(blood_type):',
'status': fully_qualified,
'E': 0.6,
'alias': [':ab:', ':ab_button_blood_type:'],
'de': ':großbuchstaben_ab_in_rotem_quadrat:',
'es': ':grupo_sanguíneo_ab:',
'fr': ':groupe_sanguin_ab:',
'ja': ':血液型ab型:',
'ko': ':에이비형:',
'pt': ':botão_ab_(tipo_sanguíneo):',
'it': ':gruppo_sanguigno_ab:',
'fa': ':دکمه_آ_ب_(گروه_خونی):',
'id': ':tombol_ab_(golongan_darah):',
'zh': ':AB型血:',
'ru': ':IV_группа_крови:',
'tr': ':ab_düğmesi_(kan_grubu):',
'ar': ':زر_ab_(فئة_الدم):',
},
'\U0001f3e7': {
'en': ':ATM_sign:',
'status': fully_qualified,
'E': 0.6,
'alias': [':atm:', ':atm_sign:'],
'de': ':symbol_geldautomat:',
'es': ':señal_de_cajero_automático:',
'fr': ':distributeur_de_billets:',
'ja': ':atm:',
'ko': ':에이티엠:',
'pt': ':símbolo_de_caixa_automático:',
'it': ':simbolo_dello_sportello_bancomat:',
'fa': ':نشان_عابربانک:',
'id': ':tanda_atm:',
'zh': ':取款机:',
'ru': ':значок_банкомата:',
'tr': ':atm_işareti:',
'ar': ':علامة_ماكينة_صرف_آلي:',
},
'\U0001f170\U0000fe0f': {
'en': ':A_button_(blood_type):',
'status': fully_qualified,
'E': 0.6,
'alias': [':a:', ':a_button_blood_type:'],
'variant': True,
'de': ':großbuchstabe_a_in_rotem_quadrat:',
'es': ':grupo_sanguíneo_a:',
'fr': ':groupe_sanguin_a:',
'ja': ':血液型a型:',
'ko': ':에이형:',
'pt': ':botão_a_(tipo_sanguíneo):',
'it': ':gruppo_sanguigno_a:',
'fa': ':دکمه_آ_(گروه_خونی):',
'id': ':tombol_a_(golongan_darah):',
'zh': ':A型血:',
'ru': ':ii_группа_крови:',
'tr': ':a_düğmesi_(kan_grubu):',
'ar': ':زر_a:',
},
'\U0001f170': {
'en': ':A_button_(blood_type):',
'status': unqualified,
'E': 0.6,
'alias': [':a:', ':a_button_blood_type:'],
'variant': True,
'de': ':großbuchstabe_a_in_rotem_quadrat:',
'es': ':grupo_sanguíneo_a:',
'fr': ':groupe_sanguin_a:',
'ja': ':血液型a型:',
'ko': ':에이형:',
'pt': ':botão_a_(tipo_sanguíneo):',
'it': ':gruppo_sanguigno_a:',
'fa': ':دکمه_آ_(گروه_خونی):',
'id': ':tombol_a_(golongan_darah):',
'zh': ':A型血:',
'ru': ':II_группа_крови:',
'tr': ':a_düğmesi_(kan_grubu):',
'ar': ':زر_a:',
},
'\U0001f1e6\U0001f1eb': {
'en': ':Afghanistan:',
'status': fully_qualified,
'E': 2,
'alias': [':flag_for_Afghanistan:', ':afghanistan:'],
'de': ':flagge_afghanistan:',
'es': ':bandera_afganistán:',
'fr': ':drapeau_afghanistan:',
'ja': ':旗_アフガニスタン:',
'ko': ':깃발_아프가니스탄:',
'pt': ':bandeira_afeganistão:',
'it': ':bandiera_afghanistan:',
'fa': ':پرچم_افغانستان:',
'id': ':bendera_afganistan:',
'zh': ':阿富汗:',
'ru': ':флаг_Афганистан:',
'tr': ':bayrak_afganistan:',
'ar': ':علم_أفغانستان:',
},
'\U0001f1e6\U0001f1f1': {
'en': ':Albania:',
'status': fully_qualified,
'E': 2,
'alias': [':flag_for_Albania:', ':albania:'],
'de': ':flagge_albanien:',
'es': ':bandera_albania:',
'fr': ':drapeau_albanie:',
'ja': ':旗_アルバニア:',
'ko': ':깃발_알바니아:',
'pt': ':bandeira_albânia:',
'it': ':bandiera_albania:',
'fa': ':پرچم_آلبانی:',
'id': ':bendera_albania:',
'zh': ':阿尔巴尼亚:',
'ru': ':флаг_Албания:',
'tr': ':bayrak_arnavutluk:',
'ar': ':علم_ألبانيا:',
},
'\U0001f1e9\U0001f1ff': {
'en': ':Algeria:',
'status': fully_qualified,
'E': 2,
'alias': [':flag_for_Algeria:', ':algeria:'],
'de': ':flagge_algerien:',
'es': ':bandera_argelia:',
'fr': ':drapeau_algérie:',
'ja': ':旗_アルジェリア:',
'ko': ':깃발_알제리:',
'pt': ':bandeira_argélia:',
'it': ':bandiera_algeria:',
'fa': ':پرچم_الجزایر:',
'id': ':bendera_aljazair:',
'zh': ':阿尔及利亚:',
'ru': ':флаг_Алжир:',
'tr': ':bayrak_cezayir:',
'ar': ':علم_الجزائر:',
},
'\U0001f1e6\U0001f1f8': {
'en': ':American_Samoa:',
'status': fully_qualified,
'E': 2,
'alias': [':flag_for_American_Samoa:', ':american_samoa:'],
'de': ':flagge_amerikanisch-samoa:',
'es': ':bandera_samoa_americana:',
'fr': ':drapeau_samoa_américaines:',
'ja': ':旗_米領サモア:',
'ko': ':깃발_아메리칸_사모아:',
'pt': ':bandeira_samoa_americana:',
'it': ':bandiera_samoa_americane:',
'fa': ':پرچم_ساموآی_امریکا:',
'id': ':bendera_samoa_amerika:',
'zh': ':美属萨摩亚:',
'ru': ':флаг_Американское_Самоа:',
'tr': ':bayrak_amerikan_samoası:',
'ar': ':علم_ساموا_الأمريكية:',
},
}
backend/server.py
+54 -53
View File
@@ -8,6 +8,7 @@ import requests
import json
import base64
import urllib.request
# import ssl
# ssl._create_default_https_context = ssl._create_unverified_context
@@ -35,14 +36,14 @@ app.add_middleware(
allow_headers=["*"],
)
# MongoDB Setup
import certifi
MONGO_URI = os.getenv("MONGODB_URI", "mongodb://localhost:27017/")
mongo_client = MongoClient(MONGO_URI, tlsCAFile=certifi.where())
db = mongo_client.study_buddy_db
study_rooms_collection = db.study_rooms
# Pydantic models for Study Room Data
class GeoJSONPoint(BaseModel):
type: str = "Point"
coordinates: List[float]
@@ -75,12 +76,12 @@ HEADERS = {
"sec-fetch-site": "same-origin",
"sentry-trace": "250c82a03041415b99422d838ccc7003-9a9ec11d7fd0293b-0",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Edg/146.0.0.0",
"x-cosmos-session-281286": "0:-1#10931",
"x-cosmos-session-295334": "0:-1#749854",
"x-cosmos-session-317755": "0:-1#191601",
"x-cosmos-session-382299": "0:-1#265024",
"x-cosmos-session-418988": "0:-1#4058856",
"x-cosmos-session-793952": "0:-1#14004",
"x-cosmos-session-281286": "0:-1",
"x-cosmos-session-295334": "0:-1",
"x-cosmos-session-317755": "0:-1",
"x-cosmos-session-382299": "0:-1",
"x-cosmos-session-418988": "0:-1",
"x-cosmos-session-793952": "0:-1",
"x-request-id": "6a128b8a-7f63-4f97-a40b-bfd31b4a376e",
"x-timezone": "America/New_York",
}
@@ -97,7 +98,7 @@ def _write_wav_to_buffer(pcm_data: bytes, sample_rate: int = SAMPLE_RATE) -> byt
buf.write(b"WAVE")
buf.write(b"fmt ")
buf.write(struct.pack("<I", 16))
buf.write(struct.pack("<H", 1)) # PCM
buf.write(struct.pack("<H", 1))
buf.write(struct.pack("<H", NUM_CHANNELS))
buf.write(struct.pack("<I", sample_rate))
buf.write(struct.pack("<I", byte_rate))
@@ -112,8 +113,8 @@ def _transcribe_pcm(pcm_data: bytes, sample_rate: int = SAMPLE_RATE) -> str:
"""Transcribe raw PCM audio using faster-whisper via a temp WAV file."""
from faster_whisper import WhisperModel
print(f" Using sample rate: {sample_rate} Hz")
# Debug: analyze PCM audio quality
num_samples = len(pcm_data) // 2
if num_samples > 0:
samples = list(struct.unpack(f"<{num_samples}h", pcm_data[:num_samples * 2]))
@@ -121,30 +122,30 @@ def _transcribe_pcm(pcm_data: bytes, sample_rate: int = SAMPLE_RATE) -> str:
mean_s = sum(samples) / num_samples
rms = (sum(s * s for s in samples) / num_samples) ** 0.5
print(f" PCM stats (raw): {num_samples} samples, min={min_s}, max={max_s}, mean={mean_s:.1f}, RMS={rms:.1f}")
# Remove DC offset (center audio at 0)
dc_offset = int(round(mean_s))
samples = [max(-32768, min(32767, s - dc_offset)) for s in samples]
pcm_data = struct.pack(f"<{num_samples}h", *samples)
# Stats after correction
rms_fixed = (sum(s * s for s in samples) / num_samples) ** 0.5
print(f" PCM stats (fixed): DC offset removed={dc_offset}, RMS={rms_fixed:.1f}")
wav_data = _write_wav_to_buffer(pcm_data, sample_rate=sample_rate)
tmp_fd, tmp_path = tempfile.mkstemp(suffix=".wav")
try:
with os.fdopen(tmp_fd, "wb") as f:
f.write(wav_data)
# Save a debug copy so we can listen
debug_path = os.path.join(os.path.dirname(__file__), "debug_audio.wav")
with open(debug_path, "wb") as df:
df.write(wav_data)
print(f" Debug WAV saved to: {debug_path}")
# Initialize the model (using base model for speed)
model = WhisperModel("base", device="cpu", compute_type="int8")
segments, info = model.transcribe(tmp_path, beam_size=5)
seg_list = list(segments)
@@ -190,7 +191,7 @@ def get_terp_ai_response(message: str) -> str:
except Exception as e:
print(f"Terp AI error: {e}")
return "I am sorry, there was an error connecting to Terp AI."
return full_response
def _convert_to_pcm(audio_data: bytes, input_format: str = "mp3") -> bytes | None:
@@ -232,7 +233,7 @@ def _generate_tts(text: str) -> bytes | None:
print("ElevenLabs API key not configured")
return None
# Request PCM directly — no ffmpeg needed
url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}?output_format=pcm_16000"
headers = {
@@ -267,7 +268,7 @@ def _generate_tts(text: str) -> bytes | None:
print(f"ElevenLabs TTS error: {e}")
return None
# Latest TTS WAV stored in memory for HTTP download by M5GO
_latest_tts_wav = None
@app.get("/api/tts-audio")
@@ -282,40 +283,40 @@ async def websocket_voice(websocket: WebSocket):
await websocket.accept()
print("Device connected to WebSocket.")
audio_buffer = bytearray()
try:
while True:
data = await websocket.receive()
if "bytes" in data:
audio_buffer.extend(data["bytes"])
elif "text" in data:
try:
msg = json.loads(data["text"])
if msg.get("event") == "stop_listening":
pcm_data = bytes(audio_buffer)
audio_buffer = bytearray() # Reset for next time
audio_buffer = bytearray()
device_sample_rate = msg.get("sample_rate", SAMPLE_RATE)
print(f"Received stop_listening event. Buffer size: {len(pcm_data)} bytes, sample_rate: {device_sample_rate} Hz")
if len(pcm_data) < 3200:
print("Audio too short, ignoring.")
await websocket.send_bytes(b"")
continue
# Step 1: Speech to Text
print("Transcribing...")
user_text = _transcribe_pcm(pcm_data, sample_rate=device_sample_rate)
if not user_text:
print("Transcription failed or empty.")
await websocket.send_text(json.dumps({"event": "error", "msg": "No speech detected"}))
continue
print(f"User said: {user_text}")
# Step 2: Terp AI
print("Sending to Terp AI...")
context_str = get_latest_locations_context()
augmented_prompt = f"USER ASKS: {user_text}\n\n[SYSTEM CONTEXT - LATEST UMD ROOM STATS TO HELP YOU ANSWER IF ASKED]:\n{context_str}"
@@ -324,15 +325,15 @@ async def websocket_voice(websocket: WebSocket):
print("No response from Terp AI.")
await websocket.send_text(json.dumps({"event": "error", "msg": "No AI response"}))
continue
print(f"Terp AI response: {ai_response_text}")
# Step 3: Text to Speech
print("Generating TTS...")
tts_pcm = _generate_tts(ai_response_text)
if tts_pcm:
# Save as WAV for HTTP download by M5GO
global _latest_tts_wav
_latest_tts_wav = _write_wav_to_buffer(tts_pcm)
print(f"TTS WAV ready: {len(_latest_tts_wav)} bytes, serving via /api/tts-audio")
@@ -343,7 +344,7 @@ async def websocket_voice(websocket: WebSocket):
else:
print("TTS failed.")
await websocket.send_text(json.dumps({"event": "error", "msg": "TTS failed"}))
except json.JSONDecodeError:
pass
except Exception as e:
@@ -383,41 +384,41 @@ def get_latest_locations_context() -> str:
}}
]
latest_stats = list(study_rooms_collection.aggregate(pipeline))
if not latest_stats:
return "No recent location noise stats available today."
room_dict = {loc["id"]: loc["name"] for loc in UMD_LOCATIONS}
lines = ["Latest Study Room Stats:"]
for stat in latest_stats:
room_id = stat.get("_id")
name = room_dict.get(room_id, room_id)
db = stat.get("latest_db", 0.0)
status = "Quiet"
if isinstance(db, (int, float)):
if db >= 65: status = "Loud"
elif db >= 55: status = "Moderate"
lines.append(f"- {name}: Noise Level {db:.1f} dB ({status})")
return "\n".join(lines)
@app.post("/api/study-rooms")
async def create_study_room_data(data: StudyRoomData):
# Check if the coordinates match one of the known locations (with small tolerance)
is_valid_location = False
req_lng, req_lat = data.location.coordinates[0], data.location.coordinates[1]
for loc in UMD_LOCATIONS:
if abs(loc["lng"] - req_lng) < 0.0001 and abs(loc["lat"] - req_lat) < 0.0001:
is_valid_location = True
# Override coordinates to exactly match known location for consistency
data.location.coordinates = [loc["lng"], loc["lat"]]
data.room_id = loc["id"]
break
if not is_valid_location:
from fastapi import HTTPException
raise HTTPException(status_code=400, detail="Invalid location. Coordinates must correspond to a known UMD location.")
@@ -438,7 +439,7 @@ async def get_study_room_history():
"""Get all study room data from the last 24 hours."""
twenty_four_hours_ago = datetime.utcnow() - timedelta(hours=24)
rooms = list(study_rooms_collection.find(
{"date": {"$gte": twenty_four_hours_ago}},
{"date": {"$gte": twenty_four_hours_ago}},
{"_id": 0}
).sort("date", -1))
return {"data": rooms}
@@ -448,13 +449,13 @@ async def serve_spa(full_path: str):
static_dir = "static"
if not os.path.exists(static_dir):
return {"error": "Static directory not found. Please build the frontend."}
static_path = os.path.join(static_dir, full_path)
if os.path.isfile(static_path):
return FileResponse(static_path)
index_path = os.path.join(static_dir, "index.html")
if os.path.exists(index_path):
return FileResponse(index_path)
return {"error": "index.html not found in static directory"}
backend/vision.py
+23 -23
View File
@@ -4,15 +4,15 @@ from ultralytics import YOLO
from scipy.spatial.distance import cdist
from scipy.optimize import linear_sum_assignment
# Load YOLOv8 nano model (downloads automatically if not found)
# 'yolov8n.pt' is lightweight and fast for this purpose
try:
model = YOLO("yolov8n.pt")
except Exception as e:
print(f"Error loading YOLO model: {e}")
model = None
# COCO Class IDs
PERSON_CLASS_ID = 0
CHAIR_CLASS_ID = 56
@@ -20,16 +20,16 @@ def analyze_room_image(image_bytes: bytes):
if not model:
return {"error": "Vision model is not loaded"}
# Convert bytes to numpy array then to cv2 image
nparr = np.frombuffer(image_bytes, np.uint8)
img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
if img is None:
return {"error": "Invalid image format"}
# Run inference
results = model(img)
people = []
chairs = []
@@ -38,25 +38,25 @@ def analyze_room_image(image_bytes: bytes):
for box in boxes:
cls_id = int(box.cls[0])
conf = float(box.conf[0])
# Extract center of bounding box
x1, y1, x2, y2 = box.xyxy[0]
cx = (x1 + x2) / 2.0
cy = (y1 + y2) / 2.0
centroid = [float(cx), float(cy)]
# Only consider detections with confidence > 0.3
if conf > 0.3:
if cls_id == PERSON_CLASS_ID:
people.append({
"centroid": centroid,
"box": [float(x1), float(y1), float(x2), float(y2)],
"centroid": centroid,
"box": [float(x1), float(y1), float(x2), float(y2)],
"conf": conf
})
elif cls_id == CHAIR_CLASS_ID:
chairs.append({
"centroid": centroid,
"box": [float(x1), float(y1), float(x2), float(y2)],
"centroid": centroid,
"box": [float(x1), float(y1), float(x2), float(y2)],
"conf": conf
})
@@ -64,18 +64,18 @@ def analyze_room_image(image_bytes: bytes):
num_chairs = len(chairs)
pairs = []
# Bipartite matching if both people and chairs exist
if num_people > 0 and num_chairs > 0:
people_coords = [p["centroid"] for p in people]
chairs_coords = [c["centroid"] for c in chairs]
# Distance matrix (Euclidean distances)
dist_matrix = cdist(people_coords, chairs_coords, metric='euclidean')
# Hungarian algorithm to minimize total distance for pairings
row_ind, col_ind = linear_sum_assignment(dist_matrix)
for person_idx, chair_idx in zip(row_ind, col_ind):
distance = float(dist_matrix[person_idx, chair_idx])
pairs.append({
@@ -84,8 +84,8 @@ def analyze_room_image(image_bytes: bytes):
"distance": distance
})
# Basic logic: room is full if there are at least as many people as chairs.
# Can be adjusted based on specific room definitions
is_full = num_people >= num_chairs if num_chairs > 0 else False
return {