Skip to content

Commit 0abb584

Browse files
mirkokurtBeanRepodsammarugaTaddyHCstefanotorneo
authored
Add examples for App Lab v0.3 release (#28)
* Replace SineGenerator with WaveGenerator Brick in Theremin (#8) * Mascot Jump Game Example Addition (#7) * Add Bedtime story teller example (#15) * Mascot Jump Game Update UI (#17) * Add Object Hunting example documentation (#18) * Update Theremin Simulator example documentation (#16) * Add led-matrix-painter (#5) * Docs: update bedtime story teller (#20) * Fan vibration monitoring UI (#19) * Vibration Anomaly Detection Example Documentation (#22) * Led matrix UI improvement (#24) * Docs led matrix painter (#26) --------- Co-authored-by: Dario Sammaruga <[email protected]> Co-authored-by: Dario Sammaruga <[email protected]> Co-authored-by: TaddyHC <[email protected]> Co-authored-by: Stefano Torneo <[email protected]> Co-authored-by: Matteo Marsala <[email protected]> Co-authored-by: Ernesto Voltaggio <[email protected]> Co-authored-by: Christopher Méndez <[email protected]> Co-authored-by: Stefano Torneo <[email protected]>
1 parent db45846 commit 0abb584

File tree

190 files changed

+13078
-33
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

190 files changed

+13078
-33
lines changed

examples/audio-classification/python/main.py

Lines changed: 3 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -13,41 +13,13 @@
1313

1414
# Global state
1515
AUDIO_DIR = "/app/assets/audio"
16-
audio_classifier = None
17-
18-
def get_audio_classifier():
19-
"""Lazy initialization of audio classifier"""
20-
global audio_classifier
21-
if audio_classifier is None:
22-
try:
23-
from arduino.app_peripherals.microphone import Microphone
24-
try:
25-
audio_classifier = AudioClassification(mic=None)
26-
except:
27-
class MockMicrophone:
28-
def __init__(self):
29-
self.sample_rate = 16000
30-
self.channels = 1
31-
def start_recording(self): pass
32-
def stop_recording(self): pass
33-
def read(self): return b''
34-
mock_mic = MockMicrophone()
35-
audio_classifier = AudioClassification(mic=mock_mic)
36-
except Exception as e:
37-
raise e
38-
return audio_classifier
3916

4017
def parse_data(data):
41-
"""Parse incoming data - handle both string and dict"""
4218
if isinstance(data, str):
43-
try:
44-
return json.loads(data)
45-
except:
46-
return {}
19+
return json.loads(data)
4720
return data if isinstance(data, dict) else {}
4821

4922
def on_run_classification(sid, data):
50-
"""Run classification"""
5123
try:
5224
parsed_data = parse_data(data)
5325
confidence = parsed_data.get('confidence', 0.5)
@@ -65,19 +37,17 @@ def on_run_classification(sid, data):
6537
return
6638
with open(file_path, "rb") as f:
6739
input_audio = io.BytesIO(f.read())
68-
6940
if input_audio:
70-
classifier = get_audio_classifier()
7141
start_time = time.time() * 1000
72-
results = classifier.classify_from_file(input_audio, confidence)
42+
results = AudioClassification.classify_from_file(input_audio, confidence)
7343
diff = time.time() * 1000 - start_time
7444

7545
response_data = { 'results': results, 'processing_time': diff }
7646
if results:
7747
response_data['classification'] = { 'class_name': results["class_name"], 'confidence': results["confidence"] }
7848
else:
7949
response_data['error'] = "No objects detected in the audio. Try to lower the confidence threshold."
80-
50+
8151
ui.send_message('classification_complete', response_data, sid)
8252
else:
8353
ui.send_message('classification_error', {'message': "No audio available for classification"}, sid)
Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# Bedtime Story Teller
2+
3+
The **Bedtime Story Teller** example demonstrates how to build a generative AI application using the Arduino UNO Q. It uses a Large Language Model (LLM) to create personalized bedtime stories based on user-selected parameters like age, theme, and characters, streaming the result in real-time to a web interface.
4+
5+
![Bedtime Story Teller Example](assets/docs_assets/thumbnail.png)
6+
7+
## Description
8+
9+
This App transforms the UNO Q into an AI storytelling assistant. It uses the `cloud_llm` Brick to connect to a cloud-based AI model and the `web_ui` Brick to provide a rich configuration interface.
10+
11+
The workflow allows you to craft a story by selecting specific parameters—such as the child's age, story theme, tone, and specific characters—or to let the App **generate a story randomly** for a quick surprise. The backend constructs a detailed prompt, sends it to the AI model, and streams the generated story back to the browser text-token by text-token.
12+
13+
## Bricks Used
14+
15+
The bedtime story teller example uses the following Bricks:
16+
17+
- `cloud_llm`: Brick to interact with cloud-based Large Language Models (LLMs) like Google Gemini, OpenAI GPT, or Anthropic Claude.
18+
- `web_ui`: Brick to create the web interface for parameter input and story display.
19+
20+
## Hardware and Software Requirements
21+
22+
### Hardware
23+
24+
- Arduino UNO Q (x1)
25+
- USB-C® cable (for power and programming) (x1)
26+
27+
### Software
28+
29+
- Arduino App Lab
30+
31+
**Note:** This example requires an active internet connection to reach the AI provider's API. You will also need a valid **API Key** for the service used (e.g., Google AI Studio API Key).
32+
33+
## How to Use the Example
34+
35+
This example requires a valid API Key from an LLM provider (Google Gemini, OpenAI GPT, or Anthropic Claude) and an internet connection.
36+
37+
### Configure & Launch App
38+
39+
1. **Duplicate the Example**
40+
Since built-in examples are read-only, you must duplicate this App to edit the configuration. Click the arrow next to the App name and select **Duplicate** or click the **Copy and edit app** button on the top right corner of the App page.
41+
![Duplicate example](assets/docs_assets/duplicate-app.png)
42+
43+
2. **Open Brick Configuration**
44+
On the App page, locate the **Bricks** section on the left. Click on the **Cloud LLM** Brick, then click the **Brick Configuration** button on the right side of the screen.
45+
![Open Brick Configuration](assets/docs_assets/brick-config.png)
46+
47+
3. **Add API Key**
48+
In the configuration panel, enter your API Key into the corresponding field. This securely saves your credentials for the App to use. You can generate an API key from your preferred provider:
49+
* **Google Gemini:** [Get API Key](https://aistudio.google.com/app/apikey)
50+
* **OpenAI GPT:** [Get API Key](https://platform.openai.com/api-keys)
51+
* **Anthropic Claude:** [Get API Key](https://console.anthropic.com/settings/keys)
52+
53+
![Enter your API KEY](assets/docs_assets/brick-credentials.png)
54+
55+
4. **Run the App**
56+
Launch the App by clicking the **Run** button in the top right corner. Wait for the App to start.
57+
![Launch the App](assets/docs_assets/launch-app.png)
58+
59+
5. **Access the Web Interface**
60+
Open the App in your browser at `<UNO-Q-IP-ADDRESS>:7000`.
61+
62+
### Interacting with the App
63+
64+
1. **Choose Your Path**
65+
You have two options to create a story:
66+
* **Option A: Manual Configuration** (Follow step 2)
67+
* **Option B: Random Generation** (Skip to step 3)
68+
69+
2. **Set Parameters (Manual)**
70+
Use the interactive interface to configure the story details. The interface unlocks sections sequentially:
71+
- **Age:** Select the target audience (3-5, 6-8, 9-12, 13-16 years, or Adult).
72+
- **Theme:** Choose a genre (Fantasy/Adventure, Fairy Tale, Mystery/Horror, Science/Universe, Animals, or Comedy).
73+
- **Story Type (Optional):** Fine-tune the narrative:
74+
- *Tone:* e.g., Calm and sweet, Epic and adventurous, Tense and grotesque.
75+
- *Ending:* e.g., Happy, With a moral, Open and mysterious.
76+
- *Structure:* Classic, Chapter-based, or Episodic.
77+
- *Duration:* Short (5 min), Medium (10-15 min), or Long (20+ min).
78+
- **Characters:** You must add **at least one character** (max 5). Define their Name, Description, and Role (Protagonist, Antagonist, Positive/Negative Helper, or Other).
79+
- **Generate:** Once ready, click the **Generate story** button.
80+
81+
3. **Generate Randomly**
82+
If you prefer a surprise, click the **Generate Randomly** button on the right side of the screen. The App will automatically select random options for age, theme, tone, and structure to create a unique story instantly.
83+
84+
4. **Interact**
85+
The story streams in real-time. Once complete, you can:
86+
- **Copy** the text to your clipboard.
87+
- Click **New story** to reset the interface and start over.
88+
89+
## How it Works
90+
91+
Once the App is running, it performs the following operations:
92+
93+
- **User Input Collection**: The `web_ui` Brick serves an HTML page where users select story attributes via interactive "chips" and forms.
94+
- **Prompt Engineering**: When the user requests a story, the Python backend receives a JSON object containing all parameters. It dynamically constructs a natural language prompt optimized for the LLM (e.g., "As a parent... I need a story about [Theme]...").
95+
- **AI Inference**: The `cloud_llm` Brick sends this prompt to the configured cloud provider using the API Key set in the Brick Configuration.
96+
- **Stream Processing**: Instead of waiting for the full text, the backend receives the response in chunks (tokens) and forwards them immediately to the frontend via WebSockets, ensuring the user sees progress instantly.
97+
98+
## Understanding the Code
99+
100+
### 🔧 Backend (`main.py`)
101+
102+
The Python script handles the logic of connecting to the AI and managing the data flow. Note that the API Key is not hardcoded; it is retrieved automatically from the Brick configuration.
103+
104+
- **Initialization**: The `CloudLLM` is set up with a system prompt that enforces HTML formatting for the output. The `CloudModel` constants map to specific efficient model versions:
105+
* `CloudModel.GOOGLE_GEMINI``gemini-2.5-flash`
106+
* `CloudModel.OPENAI_GPT``gpt-4o-mini`
107+
* `CloudModel.ANTHROPIC_CLAUDE``claude-3-7-sonnet-latest`
108+
109+
```python
110+
# The API Key is loaded automatically from the Brick Configuration
111+
llm = CloudLLM(
112+
model=CloudModel.GOOGLE_GEMINI,
113+
system_prompt="You are a bedtime story teller. Your response must be the story itself, formatted directly in HTML..."
114+
)
115+
llm.with_memory()
116+
```
117+
118+
- **Prompt Construction**: The `generate_story` function translates the structured data from the UI into a descriptive text prompt for the AI.
119+
120+
```python
121+
def generate_story(_, data):
122+
# Extract parameters
123+
age = data.get('age', 'any')
124+
theme = data.get('theme', 'any')
125+
126+
# Build natural language prompt
127+
prompt_for_display = f"As a parent who loves to read bedtime stories to my <strong>{age}</strong> year old child..."
128+
129+
# ... logic to append characters and settings ...
130+
131+
# Stream response back to UI
132+
prompt_for_llm = re.sub('<[^>]*>', '', prompt_for_display) # Clean tags for LLM
133+
for resp in llm.chat_stream(prompt_for_llm):
134+
ui.send_message("response", resp)
135+
136+
ui.send_message("stream_end", {})
137+
```
138+
139+
### 🔧 Frontend (`app.js`)
140+
141+
The JavaScript manages the complex UI interactions, random generation logic, and WebSocket communication.
142+
143+
- **Random Generation**: If the user chooses "Generate Randomly", the frontend programmatically selects random chips from the available options and submits the request.
144+
145+
```javascript
146+
document.getElementById('generate-randomly-button').addEventListener('click', () => {
147+
// Select random elements from the UI lists
148+
const ageChips = document.querySelectorAll('.parameter-container:nth-child(1) .chip');
149+
const randomAgeChip = getRandomElement(ageChips);
150+
// ... repeat for theme, tone, etc ...
151+
152+
const storyData = {
153+
age: randomAgeChip ? randomAgeChip.textContent.trim() : 'any',
154+
// ...
155+
characters: [], // Random stories use generic characters
156+
};
157+
158+
generateStory(storyData);
159+
});
160+
```
161+
162+
- **Socket Listeners**: The frontend listens for chunks of text and appends them to the display buffer, creating the streaming effect.
163+
164+
```javascript
165+
socket.on('response', (data) => {
166+
document.getElementById('story-container').style.display = 'flex';
167+
storyBuffer += data; // Accumulate text
168+
});
169+
170+
socket.on('stream_end', () => {
171+
const storyResponse = document.getElementById('story-response');
172+
storyResponse.innerHTML = storyBuffer; // Final render
173+
document.getElementById('loading-spinner').style.display = 'none';
174+
});
175+
```
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
name: Bedtime story teller
2+
icon: 🏰
3+
description:
4+
This example shows how to create a bedtime story teller using Arduino.
5+
It uses a cloud-based language model to generate a story based on user input and shows the story on a web interface.
6+
7+
bricks:
8+
- arduino:cloud_llm
9+
- arduino:web_ui

0 commit comments

Comments
 (0)