Measure replay performance overhead and document it

[romtsn]

Description

We should measure Session Replay overhead and document it similar to how we do it for other platforms: https://docs.sentry.io/platforms/react-native/session-replay/performance-overhead/

If we already expose the quality option, we could also add a Reducing Overhead section.

I don't know if there are other means to measure perf for Flutter apps, but we could try using https://flashlight.dev/ for that (iOS probably needs to run from this branch).

[43jay]

setup

Install

• Install/Run Open Foof Facts (OFF) app
• Install flashlight CLI
• Install Maestro

Test setup

• Grab DSN from sentry-flutter proj to confirm Sentry running as expected (by seeing events show up in that proj's dash) and update the DSN used in app

• Build app for release as per OFF gh action

Based on impl of launchSmoothApp we use the release variant (the flutter profile variant won't have Sentry enabled):

1. #[CLAUDE] generate dummy release signing key.properties

2. cd packages/smooth_app && fvm flutter build apk --release -t lib/entrypoints/android/main_google_play.dart --target-platform android-arm,android-arm64

3. adb install build/app/outputs/flutter-apk/app-release.apk

4. launch Maestro Studio to automate user through onboarding as off-onboarding-perf.yaml:

   maestro studio

test runs

OFF repo code modifications:

0001-sentry-dart-3035-Modify-OFF-App-for-measuring-replay.patch

Control test run

$ flashlight test --bundleId org.openfoodfacts.scanner \
 --testCommand "maestro test off-onboarding-perf.yaml" \
 --duration 10000 \
 --resultsFilePath release_without_replay_results.json

$ flashlight report release_with_replay_results.json
> ✅  Opening report: /var/folders/gw/qw7qx4193y3b0bpwct967bpm0000gn/T/report.html

Total CPU usage (max=106%):

Ram usage (max=400MB):

With Sentry Replay test run

$ flashlight test --bundleId org.openfoodfacts.scanner \
 --testCommand "maestro test off-onboarding-perf.yaml" \
 --duration 10000 \
 --resultsFilePath release_with_replay_results.json

$ flashlight report release_with_replay_results.json
> ✅  Opening report: /var/folders/gw/qw7qx4193y3b0bpwct967bpm0000gn/T/report.html

Total CPU usage (max=127%):

RAM usage (max=445MB):

[buenaflor]

@denrase as discussed I'll hand over the iOS part to you

We should find out if we can use the mentioned flashlight branch to run on a physical iOS device and then re-create the maestro flows manually. afaik we cannot use maestro on physical ios devices so we should aim to replicate it (e.g manually doing the flows) as closely as possible as described here:

[denrase]

@buenaflor Took the patch file and applied it to the repo. Also had to update Xcode settings for signing to work with my dev account.

As for flashlight, I was able to run for iOS on the branch you provided, sorta.

Checked out the repo and switched to the branch. Needed to install idb by facebook to manage run destinations.

But the problem is that idb relies on DeveloperDiskImages, which are no longer supported as far as I could tell. The most recent one is iOS 16.6, but the oldest device I have already has iOS 16.7.

idb launch org.openfoodfacts.scanner.denrase                                
The best match /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/DeviceSupport/15.0/DeveloperDiskImage.dmg: 15.0 is not suitable for 0.0

smooth-app % PLATFORM=ios node ~/Repos/other/flashlight/packages/commands/test/dist/bin.js test --bundleId org.openfoodfacts.scanner.denrase \
  --testCommand "maestro test off-onboarding-perf.yaml" \
  --duration 10000 \
  --resultsFilePath results.json
[18:22:45] ℹ️  Running iteration 1/10
[18:22:48] 🚨  Iteration 1/10 failed: No app running
[18:22:48] 🚨  Ignoring measure and retrying... (retry 1/3)
[18:22:48] ℹ️  Running iteration 1/10
[18:22:49] 🚨  Iteration 1/10 failed: No app running
[18:22:49] 🚨  Ignoring measure and retrying... (retry 2/3)
[18:22:49] ℹ️  Running iteration 1/10
[18:22:50] 🚨  Iteration 1/10 failed: No app running
[18:22:50] 🚨  Ignoring measure and retrying... (retry 3/3)
[18:22:50] ℹ️  Running iteration 1/10
[18:22:51] 🚨  Iteration 1/10 failed: No app running
[18:22:51] ✅  Results written to results.json.
To open the web report, run:

flashlight report results.json
[18:22:51] 🚨  Flashlight test FAILED ❌: Max number of retries reached.
      You can still open a degraded view of the report

We need to find a differnt approach to test this on iOS.

---

Tried renaming 16.6 to 16.7 like suggested here, but the issue remains. Seems like idb is incorreclty parsing the 16.7.10 version and cannot match any image to it.

[denrase]

Okay, I have a minimal setup and cocluded a test.

Setup

• App was setup like in the patch file above with the current main of the smooth app (039d731942715cd9dcb4c99c0cabc19a567a3233).

• App build with Xcode Version 16.4 (16F6) in release mode. Verified through logs that Sentry setup was done correctly.

• Installed to an iPhone X with iOS 16.7.10

• Created a performance monitoring script that is sampling CPU/Memory with the help of py-ios-device.

• While the script was sampling, iI went manually through onboarding, clicking the same things in both cases and taking roughly the same time for it, confirmed by the number of sample in both cases.

Results

Without Replay

============================================================
PERFORMANCE MONITORING RESULTS
============================================================
Bundle ID: org.openfoodfacts.scanner.denrase
Sample Count: 46
Duration: 0.0 seconds

CPU USAGE:
  Average: 29.24%
  Minimum: 7.57%
  Maximum: 67.75%
  Median:  25.59%

MEMORY USAGE:
  Average: 190.69 MiB
  Minimum: 150.64 MiB
  Maximum: 233.53 MiB
  Median:  186.36 MiB

INFO: Results saved to: performance_results_1756805784.json
Detailed results saved to: performance_results_1756805784.json
Complete monitoring output saved to: performance_logs/monitoring_output_org_openfoodfacts_scanner_denrase_20250902_113535.txt
Output file size: 8007 bytes
INFO: === PERFORMANCE SUMMARY ===
INFO: CPU - Avg: 29.24%, Min: 7.57%, Max: 67.75%
INFO: Memory - Avg: 190.69 MiB, Min: 150.64 MiB, Max: 233.53 MiB
INFO: Killing app...
Killing app...
INFO: Attempting to kill app: org.openfoodfacts.scanner.denrase
INFO: App killed successfully
INFO: Performance monitoring session completed

With Replay

============================================================
PERFORMANCE MONITORING RESULTS
============================================================
Bundle ID: org.openfoodfacts.scanner.denrase
Sample Count: 45
Duration: 0.0 seconds

CPU USAGE:
  Average: 31.33%
  Minimum: 6.88%
  Maximum: 74.03%
  Median:  34.87%

MEMORY USAGE:
  Average: 191.66 MiB
  Minimum: 137.03 MiB
  Maximum: 235.24 MiB
  Median:  186.97 MiB

INFO: Results saved to: performance_results_1756806068.json
Detailed results saved to: performance_results_1756806068.json
Complete monitoring output saved to: performance_logs/monitoring_output_org_openfoodfacts_scanner_denrase_20250902_114019.txt
Output file size: 7829 bytes
INFO: === PERFORMANCE SUMMARY ===
INFO: CPU - Avg: 31.33%, Min: 6.88%, Max: 74.03%
INFO: Memory - Avg: 191.66 MiB, Min: 137.03 MiB, Max: 235.24 MiB
INFO: Killing app...
Killing app...
INFO: Attempting to kill app: org.openfoodfacts.scanner.denrase
INFO: App killed successfully
INFO: Performance monitoring session completed

REPLAY Link

Files

replay-perf.patch

performance_results_1756805784.json

performance_results_1756806068.json

Summary

	Sentry	Sentry With Replay	Diff
CPU (AVG)	29.24%	31.33%	+7.15%
Memory (AVG)	190.69 MiB	191.66 MiB	+0.51%

Take these with a grain of salt, as for both I recorded only one manual run. But I think they are valuable, as it's a rather old device and there was not much slowdown I subjectively noticed, which is also confirmed by the recorded samples.

[romtsn]

@denrase is it a big lift to run a maestro flow for 10 times while this script is sampling? While one run is valuable, I don't think it's deterministic enough to put it on the docs