Test & measure accuracy of voice/speech detection

[MeenakshiArunsankar]

Description

Establish ground-truth accuracy metrics for the voice/speech detection system used in proctoring.

The frontend currently uses MediaPipe's AudioClassifier with the YAMNet model to classify audio and flag isSpeaking when the top category is "Speech" with score > 0.5. However, no accuracy evaluation exists — the confidence scores are discarded after thresholding, and no test set validates behavior across exam-room acoustic conditions (typing, background noise, etc.).

Current Implementation

• Detection: frontend/src/components/ai/SpeechDetector.tsx uses @mediapipe/tasks-audio AudioClassifier with YAMNet model
• Thresholding: Lines 84-88 — flags isSpeaking when top class is "Speech" with score > 0.5 (generic YAMNet default, not tuned for proctoring)
• Backend logging: backend/src/modules/anomalies/controllers/AnomalyController.ts /record/audio endpoint stores evidence files but no confidence scores

Work Required

1. Build a labeled audio test set (~100–200 clips, 500ms windows) covering:

   • Silence (empty room)
   • One person speaking (normal exam scenario)
   • Background chatter/TV/video
   • Typing/keyboard noise
   • Music or other sound sources
   • Mixed scenarios (person talking + typing)

2. Persist confidence scores — modify voice anomaly endpoints to include YAMNet's top-class confidence score and full class distribution

3. Run offline evaluation against the test set:

   • Compute precision/recall/F1 for the "Speech" class at threshold 0.5
   • Sweep the threshold (0.3–0.9) and build ROC/PR curves
   • Break down performance by condition (clean speech, background noise, etc.)
   • Identify optimal threshold that minimizes false positives (typing flagged as speech) while keeping false negatives low

4. Document results in a test report with:

   • Accuracy metrics at current (0.5) and recommended thresholds
   • False positive/negative rates by scenario
   • ROC/PR curves
   • Recommendation: should threshold be tuned? Is accuracy acceptable?

Acceptance Criteria

• Labeled ground-truth audio test set (CSV: clip_id, ground_truth_label, condition_tags, duration_ms)
• Confidence scores and class distribution logged in anomaly records
• Offline eval script (Python/Node) computing precision/recall/F1/ROC/PR curves at multiple thresholds
• Test report with results broken down by condition and threshold
• Documented decision on optimal threshold + accuracy adequacy (with evidence)

Related

• Frontend: frontend/src/components/ai/SpeechDetector.tsx
• Backend: backend/src/modules/anomalies/controllers/AnomalyController.ts
• Model docs: https://github.com/google-ai-edge/mediapipe/blob/master/mediapipe/tasks/python/audio/audio_classifier.md