Merge pull request #85 from AnkushMalaker/mic-button

Mic button

Author: AnkushMalaker

README.md

@@ -12,6 +12,14 @@
 2. Configure your API keys and service settings manually
 3. Start with `docker compose up --build -d`
 
+**Mobile App + Phone Audio (Latest Feature):**
+1. **Setup Backend**: Follow Advanced Backend setup above
+2. **Install Mobile App**: Go to `app/` directory and run `npm install && npm start`
+3. **Configure Mobile App**: Point to your backend IP in app settings
+4. **Enable Phone Audio**: Tap "Stream Phone Audio" in app for direct microphone streaming
+5. **Grant Permissions**: Allow microphone access when prompted
+6. **Start Streaming**: Speak into phone for real-time processing with live audio visualization
+
 **Documentation:** See `CLAUDE.md` and `Docs/init-system.md` for detailed setup guide
 
 ## Overview

app/README.md

@@ -5,10 +5,12 @@ React Native mobile application for connecting OMI devices and streaming audio t
 ## Features
 
 - **OMI Device Integration**: Connect via Bluetooth and stream audio
+- **Phone Audio Streaming**: Use phone's microphone directly (NEW)
 - **Cross-Platform**: iOS and Android support using React Native
 - **Real-time Audio Streaming**: OPUS audio transmission to backend services
 - **WebSocket Communication**: Efficient real-time data transfer
 - **Backend Selection**: Configure connection to any compatible backend
+- **Live Audio Visualization**: Real-time audio level meters
 
 ## Quick Start
 
@@ -115,7 +117,7 @@ The app connects to any backend that accepts OPUS audio streams:
    - Good for testing and development
    - WebSocket endpoint: `/ws`
 
-2. **Advanced Backend** (`backends/advanced-backend/`)
+2. **Advanced Backend** (`backends/advanced/`)
    - Full transcription and memory features
    - Real-time processing with speaker recognition
    - WebSocket endpoint: `/ws_pcm`
@@ -149,6 +151,60 @@ Backend URL: wss://[ngrok-subdomain].ngrok.io/ws_pcm
    - Public: `wss://[your-domain]/ws_pcm`
 5. **Save configuration**
 
+## Phone Audio Streaming (NEW)
+
+### Overview
+Stream audio directly from your phone's microphone to Friend-Lite backend, bypassing Bluetooth devices. This feature provides a direct audio input method for users who want to use their phone as the audio source.
+
+### Features
+- **Direct Microphone Access**: Use phone's built-in microphone
+- **Real-time Audio Streaming**: Live audio processing with visualization
+- **Seamless Integration**: Switch between Bluetooth and phone audio modes
+- **Cross-Platform**: Works on both iOS and Android
+- **Live Audio Meters**: Visual feedback showing audio levels in real-time
+
+### Setup & Usage
+
+#### Enable Phone Audio Streaming
+1. **Open Friend-Lite app**
+2. **Configure Backend Connection** (see Backend Configuration section)
+3. **Grant Microphone Permissions** when prompted
+4. **Tap "Stream Phone Audio" button** in main interface
+5. **Start speaking** - audio streams in real-time to backend
+
+#### Requirements
+- **iOS**: iOS 13+ with microphone permissions
+- **Android**: Android API 21+ with microphone permissions  
+- **Network**: Stable connection to Friend-Lite backend
+- **Backend**: Advanced backend running with `/ws_pcm` endpoint
+
+#### Switching Audio Sources
+- **Mutual Exclusion**: Cannot use Bluetooth and phone audio simultaneously
+- **Automatic Detection**: App disables conflicting options when one is active
+- **Visual Feedback**: Clear indicators show active audio source
+
+### Troubleshooting Phone Audio
+
+#### Audio Not Streaming
+- **Check Permissions**: Ensure microphone access granted
+- **Verify Backend URL**: Confirm `ws://[ip]:8000/ws_pcm` format
+- **Network Connection**: Test backend connectivity
+- **Authentication**: Verify JWT token is valid
+
+#### Poor Audio Quality  
+- **Check Signal Strength**: Ensure stable network connection
+- **Reduce Background Noise**: Use in quiet environment
+- **Restart Recording**: Stop and restart phone audio streaming
+
+#### Permission Issues
+- **iOS**: Settings > Privacy & Security > Microphone > Friend-Lite
+- **Android**: Settings > Apps > Friend-Lite > Permissions > Microphone
+
+#### No Audio Level Visualization
+- **Restart App**: Close and reopen the application
+- **Check Audio Input**: Ensure microphone is working in other apps
+- **Backend Logs**: Verify backend is receiving audio data
+
 ## User Workflow
 
 ### Device Connection
@@ -163,6 +219,7 @@ Backend URL: wss://[ngrok-subdomain].ngrok.io/ws_pcm
 
 ### Audio Streaming
 
+#### Option 1: Bluetooth Audio (OMI Device)
 1. **Configure backend connection** (see Configuration Steps above)
 2. **Test connection**:
    - Tap "Test Connection" in settings
@@ -172,6 +229,16 @@ Backend URL: wss://[ngrok-subdomain].ngrok.io/ws_pcm
    - Speak into OMI device
    - Audio streams to backend in real-time
 
+#### Option 2: Phone Audio Streaming
+1. **Configure backend connection** (see Configuration Steps above)
+2. **Enable phone audio**:
+   - Tap "Stream Phone Audio" button
+   - Grant microphone permissions when prompted
+3. **Start speaking**:
+   - Speak directly into phone microphone
+   - Watch real-time audio level visualization
+   - Audio streams to backend automatically
+
 ### Monitoring
 
 1. **Check connection status** in app header
@@ -197,6 +264,13 @@ Backend URL: wss://[ngrok-subdomain].ngrok.io/ws_pcm
 - Test with simple backend first
 - Monitor backend logs for connection attempts
 
+**Phone Audio Streaming Issues:**
+- Grant microphone permissions in device settings
+- Ensure stable network connection to backend
+- Restart phone audio streaming if no data flowing
+- Check backend logs for audio data reception
+- Verify JWT authentication token is valid
+
 **Build Errors:**
 - Clear Expo cache: `npx expo start --clear`
 - Clean prebuild: `npx expo prebuild --clean`
@@ -290,5 +364,5 @@ BluetoothService.onAudioData = (audioBuffer) => {
 
 - **[Backend Setup](../backends/)**: Choose and configure backend services
 - **[Quick Start Guide](../quickstart.md)**: Complete system setup
-- **[Advanced Backend](../backends/advanced-backend/)**: Full-featured backend option
+- **[Advanced Backend](../backends/advanced/)**: Full-featured backend option
 - **[Simple Backend](../backends/simple/)**: Basic backend for testing

app/app.json

@@ -17,7 +17,10 @@
     ],
     "ios": {
       "supportsTablet": true,
-      "bundleIdentifier": "com.cupbearer5517.friendlite"
+      "bundleIdentifier": "com.cupbearer5517.friendlite",
+      "infoPlist": {
+        "NSMicrophoneUsageDescription": "Friend-Lite needs access to your microphone to stream audio to the backend for processing."
+      }
     },
     "android": {
       "adaptiveIcon": {
@@ -29,12 +32,29 @@
         "android.permission.BLUETOOTH",
         "android.permission.BLUETOOTH_ADMIN",
         "android.permission.BLUETOOTH_CONNECT",
-        "android.permission.ACCESS_NETWORK_STATE"
+        "android.permission.ACCESS_NETWORK_STATE",
+        "android.permission.FOREGROUND_SERVICE",
+        "android.permission.FOREGROUND_SERVICE_DATA_SYNC",
+        "android.permission.POST_NOTIFICATIONS",
+        "android.permission.RECORD_AUDIO"
       ],
       "usesCleartextTraffic": true
     },
     "newArchEnabled": true,
     "plugins": [
+      [
+        "@siteed/expo-audio-studio",
+        {
+          "enablePhoneStateHandling": true,
+          "enableNotifications": true,
+          "enableBackgroundAudio": true,
+          "enableDeviceDetection": true,
+          "iosBackgroundModes": { "useProcessing": true },
+          "iosConfig": {
+            "microphoneUsageDescription": "We use the mic for live audio streaming"
+          }
+        }
+      ],
       [
         "react-native-ble-plx",
         {
@@ -45,11 +65,30 @@
           "bluetoothAlwaysPermission": "This app uses Bluetooth to connect to and interact with nearby BLE devices."
         }
       ],
+      [
+        "./plugins/with-ws-fgs",
+        {
+          "microphone": true,
+          "android": {
+            "permissions": [
+              "FOREGROUND_SERVICE",
+              "FOREGROUND_SERVICE_DATA_SYNC",
+              "FOREGROUND_SERVICE_MICROPHONE",
+              "POST_NOTIFICATIONS"
+            ]
+          }
+        }
+      ],
       [
         "expo-build-properties",
         {
           "android": {
-            "usesCleartextTraffic": true
+            "usesCleartextTraffic": true,
+            "compileSdkVersion": 35,
+            "targetSdkVersion": 35,
+            "extraMavenRepos": [
+              "../../node_modules/@notifee/react-native/android/libs"
+            ]
           }
         }
       ]