A React Native (Expo) app that acts as a digital companion for physical Yahtzee games. Features a digital scorecard and an AR-powered dice scanner using a custom YOLOv8 machine learning model to automatically count your dice rolls.
This project is built to achieve real-time (30-60 FPS) dice detection on mobile devices without lagging the user interface. It accomplishes this by completely separating the UI thread from the camera and machine learning processing pipeline.
Core Tech Stack:
- UI Framework: React Native with Expo (TypeScript)
- Camera:
react-native-vision-camera(Allows raw frame access natively) - Threading:
react-native-worklets-core(Executes ML logic on a background C++ thread) - Machine Learning:
react-native-fast-tflite(Runs TensorFlow Lite models inside the Vision Camera frame processor) - Image Processing:
vision-camera-resize-plugin(Handles dynamic image scaling to model requirements)
- Model: A custom trained YOLOv8 model (
yolov8-dice.tflite) exported from Roboflow and placed in theassets/directory. - Frame Processor: The camera captures a raw frame and hands it to a C++ worklet background thread (
runAtTargetFps). - Resizing: The raw frame is synchronously resized (e.g. to 640x640) and converted to a
Float32Array. - Inference: The model executes
runSyncon the C++ thread, dropping frames if it falls behind to ensure zero camera preview lag. - Parsing: The raw 3D tensor (
[1, 10, 8400]) is parsed manually, applying probability checks and Non-Maximum Suppression (NMS) to detect distinct physical dice (1 through 6). - Bridge to UI: Detections are safely bridged back to the React UI using
Worklets.createRunOnJS, drawing real-time Augmented Reality bounding boxes (<AROverlay>).
Because this project relies on custom C++ native code (Vision Camera, Fast TFLite, Worklets), it cannot be run inside the standard "Expo Go" app. You must create a local native build using expo run:android or expo run:ios.
- Node.js (LTS recommended)
- Java 17 (Strictly required for Android builds)
- Android Studio / Android SDK (for Android builds)
- Xcode (for iOS builds, Mac only)
⚠️ Important Note on Java Version for Android: React Native's Android C++ build tools currently fail on Java 22+ with the errorWARNING: A restricted method in java.lang.System has been called.Ensure your
JAVA_HOMEis pointing to Java 17 before runningnpm run android. If you use SDKMAN!, you can set this up easily by runningsdk env initin the project root to create a.sdkmanrcfile, and thensdk envto switch automatically.
- Clone the repository
- Install dependencies:
npm install
This will compile the native Android project, install the APK on your emulator or connected physical device, and start the Expo Metro Bundler.
npm run androidThis will install CocoaPods, compile the native iOS project, install the app on your simulator or connected physical device, and start the Expo Metro Bundler.
npm run iosThe app is fully compatible with the Web, though standard web browsers lack the C++ Native Modules to access raw camera frames and hardware-accelerated TFLite.
When running on the web, the app dynamically mocks the <DiceScanner> with a "Simulate Roll" interface so you can still play the game and test UI logic!
npm run webApp.tsx- The main entry point, rendering the scorecard, sections, and bottom control deck.src/components/- UI elements (die.tsx,ar-overlay.tsx) and the camera integration (dice-scanner.tsx).dice-scanner.web.tsx- The fallback mock for the web platform.
src/constants/- Yahtzee category rules.src/utils/scoring.ts- Algorithms for validating and scoring different Yahtzee combinations.src/domain/- Typescript interfaces for the ML pipeline (DiceDetection).
- Roll Dice Behaviour without Scanner: Introduce persistence for setting whether the Button rolls random dice, or opens the camera.
- Multiplayer: Setup Option to add at least one Player
- YOLOv8 Output Parser Tuning: The C++ worklet extracts bounding boxes depending on the exact
.tflitemodel used. Currently it is still a bit erronous and can be improved on Roboflow.