-
Notifications
You must be signed in to change notification settings - Fork 141
Add two skills to the genui package #800
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
redbrogdon
wants to merge
6
commits into
flutter:main
Choose a base branch
from
redbrogdon:skills-baby-skills
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
ed42a3e
Adding two skills for genui
redbrogdon 980e58f
Adding README blurb.
redbrogdon 274dd75
Accepting Gemini review comments.
redbrogdon e08c804
Accepting further review from Gemini.
redbrogdon 98b1edb
Fixing skill URL.
redbrogdon 8f1c4f4
Fixing typo
redbrogdon File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| --- | ||
| name: create-catalog-item | ||
| description: Use this skill when the user asks to create a new CatalogItem, data class, and/or widget class based on a JSON Schema definition in an application that uses Flutter's `genui` package. | ||
| --- | ||
|
|
||
| # Create CatalogItem | ||
|
|
||
| ## Goal | ||
| To correctly implement a GenUI CatalogItem based on a provided json_schema_builder Schema, including its corresponding data class, CatalogItem instance, and Widget class. This ensures the AI model can properly generate and interact with the UI component. | ||
|
|
||
| ## Instructions | ||
| When tasked with creating a CatalogItem from a `Schema`, follow these steps: | ||
|
|
||
| 1. **Create the Data Class**: | ||
| - Name it `_<SchemaName>Data` (e.g., if schema is `myCardSchema`, data class is `_MyCardData`). | ||
| - Add final fields for each property defined in the schema. | ||
| - Create a `factory _<SchemaName>Data.fromJson(Map<String, Object?> json)` method. | ||
| - Use a `try-catch` block to parse the properties and return a new instance. | ||
| - Cast each property from the `json` map to its expected type, e.g., `title: json['title'] as String,` or `action: json['action'] as JsonMap?,`. | ||
| - Throw an `Exception('Invalid JSON for _<SchemaName>Data')` in the `catch` block if an error occurs. | ||
|
|
||
| 2. **Create the CatalogItem Instance**: | ||
| - Name it identical to the schema name but without the "Schema" suffix (e.g., `myCard` for `myCardSchema`). | ||
| - Declare as a `final CatalogItem`. | ||
| - Set `name` to the capitalized version of the name (e.g., `'MyCard'`). | ||
| - Set `dataSchema` to the provided schema. | ||
| - Implement the `widgetBuilder: (itemContext)`: | ||
| - Cast `itemContext.data` to `Map<String, Object?>`. | ||
| - Parse the data using the data class `fromJson` method: `_<SchemaName>Data.fromJson(json)`. | ||
| - Return the corresponding Widget class and pass the required data. | ||
| - If the schema includes an action callback (like `onCompleted`), implement it here. You must parse the action context using `resolveContext` and dispatch an event using `itemContext.dispatchEvent(...)`. | ||
|
|
||
| 3. **Create the Widget Class**: | ||
| - Name it `_<CapitalizedSchemaName>` (e.g., `_MyCard`). | ||
| - Inherit from `StatelessWidget` or `StatefulWidget` depending on state requirements. | ||
| - Add the Data Class as a required property (e.g., `final _<SchemaName>Data data;`). | ||
| - Add any required callback properties (e.g., `final void Function(int) onCompleted;`). | ||
| - Implement the `build` method using Flutter Material components (e.g., Card, Column, Text). Make sure each data field in the data class is displayed, and that actions are represented by buttons or other interactive elements. | ||
|
|
||
| ## Examples | ||
| ### Input Schema | ||
| ```dart | ||
| final basicCardSchema = S.object( | ||
| properties: { | ||
| 'component': S.string(enumValues: ['BasicCard']), | ||
| 'title': S.string(), | ||
| 'action': A2uiSchemas.action(), | ||
| }, | ||
| required: ['title'], | ||
| ); | ||
| ``` | ||
|
|
||
| ### Expected Output | ||
| ```dart | ||
| class _BasicCardData { | ||
| final String title; | ||
| final JsonMap? action; | ||
|
|
||
| _BasicCardData({required this.title, this.action}); | ||
|
|
||
| factory _BasicCardData.fromJson(Map<String, Object?> json) { | ||
| try { | ||
| return _BasicCardData( | ||
| title: json['title'] as String, | ||
| action: json['action'] as JsonMap?, | ||
| ); | ||
| } catch (e) { | ||
| throw Exception('Invalid JSON for _BasicCardData: $e'); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| final basicCard = CatalogItem( | ||
| name: 'BasicCard', | ||
| dataSchema: basicCardSchema, | ||
| widgetBuilder: (itemContext) { | ||
| final json = itemContext.data as Map<String, Object?>; | ||
| final data = _BasicCardData.fromJson(json); | ||
|
|
||
| return _BasicCard( | ||
| data: data, | ||
| onTap: () async { | ||
| final action = data.action; | ||
| if (action == null) return; | ||
| final event = action['event'] as JsonMap?; | ||
| final name = (event?['name'] as String?) ?? ''; | ||
| final JsonMap contextDefinition = | ||
| (event?['context'] as JsonMap?) ?? <String, Object?>{}; | ||
| final JsonMap resolvedContext = await resolveContext( | ||
| itemContext.dataContext, | ||
| contextDefinition, | ||
| ); | ||
| itemContext.dispatchEvent( | ||
| UserActionEvent( | ||
| name: name, | ||
| sourceComponentId: itemContext.id, | ||
| context: resolvedContext, | ||
| ), | ||
| ); | ||
| } | ||
| ); | ||
| }, | ||
| ); | ||
|
|
||
| class _BasicCard extends StatelessWidget { | ||
| final _BasicCardData data; | ||
| final VoidCallback onTap; | ||
|
|
||
| const _BasicCard({super.key, required this.data, required this.onTap}); | ||
|
|
||
| @override | ||
| Widget build(BuildContext context) { | ||
| return Card( | ||
| child: ListTile( | ||
| title: Text(data.title), | ||
| onTap: onTap, | ||
| ), | ||
| ); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Constraints | ||
| - Ensure proper use of `try-catch` blocks and type casting when parsing JSON in `fromJson`. | ||
| - Make sure action resolution accurately fetches variables via `resolveContext` and uses `itemContext.dispatchEvent` when actions are present in the Schema. |
151 changes: 151 additions & 0 deletions
151
packages/genui/skills/integrate-genui-firebase/SKILL.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| --- | ||
| name: integrate-genui-firebase | ||
| description: Use this skill when the user asks to integrate the genui package and get a simple conversation going with Firebase AI Logic. | ||
| --- | ||
|
|
||
| # Integrate GenUI with Firebase AI Logic | ||
|
|
||
| ## Goal | ||
| To successfully integrate the `genui` package into a Flutter app and set up a basic conversational agent using Firebase AI Logic. This skill assumes Firebase AI Logic is already set up and working in the project. | ||
|
|
||
| ## Instructions | ||
| When tasked with integrating `genui` and starting a simple conversation, follow these steps: | ||
|
|
||
| 1. **Verify Firebase Setup:** | ||
| Ensure `firebase_core` and `firebase_ai` are available in `pubspec.yaml`. | ||
| Verify that `Firebase.initializeApp` is called in the `main()` function: | ||
| ```dart | ||
| WidgetsFlutterBinding.ensureInitialized(); | ||
| await Firebase.initializeApp(options: DefaultFirebaseOptions.currentPlatform); | ||
| ``` | ||
|
|
||
| 2. **Add GenUI Package:** | ||
| Add `genui` to the `pubspec.yaml` dependencies. | ||
|
|
||
| 3. **Import Required Libraries:** | ||
| Import `genui` and hide `TextPart` so it doesn't conflict with other packages, then import it again with an alias: | ||
| ```dart | ||
| import 'package:genui/genui.dart' hide TextPart; | ||
| import 'package:genui/genui.dart' as genui; | ||
| ``` | ||
|
|
||
| 4. **Configure Basic Logging:** | ||
| At the beginning of the `main()` function, configure GenUI logging: | ||
| ```dart | ||
| configureLogging( | ||
| logCallback: (level, msg) => debugPrint('GenUI $level: $msg'), | ||
| ); | ||
| ``` | ||
|
|
||
| 5. **Create Model and Chat Session:** | ||
| Initialize the generative model and start a chat session. | ||
| ```dart | ||
| final model = FirebaseAI.googleAI().generativeModel( | ||
| model: 'gemini-3-flash-preview', | ||
| ); | ||
| final _chatSession = model.startChat(); | ||
| ``` | ||
|
|
||
| 6. **Identify Target StatefulWidget:** | ||
| **STOP AND ASK THE USER IF UNCLEAR:** This integration requires a `StatefulWidget` to hold the references to GenUI controllers (`SurfaceController`, `A2uiTransportAdapter`, and `Conversation`). Identify which `StatefulWidget` to use in the application. If you are unsure which widget should hold this state, ask the user before proceeding. | ||
|
|
||
| 7. **Wire up GenUI Controllers inside State:** | ||
| Inside your identified `State` class, instantiate `SurfaceController`, `A2uiTransportAdapter`, and `Conversation`: | ||
| ```dart | ||
| final catalog = BasicCatalogItems.asCatalog(); // Optionally inject custom CatalogItems | ||
| final _controller = SurfaceController(catalogs: [catalog]); | ||
| final _transport = A2uiTransportAdapter(onSend: _sendAndReceive); | ||
| final _conversation = Conversation( | ||
| controller: _controller, | ||
| transport: _transport, | ||
| ); | ||
| ``` | ||
|
|
||
| 8. **Implement the `_sendAndReceive` Method:** | ||
| Create a method to take messages from the transport adapter, send them to Firebase, and feed the AI's response back to the transport. | ||
| ```dart | ||
| Future<void> _sendAndReceive(ChatMessage msg) async { | ||
| final buffer = StringBuffer(); | ||
|
|
||
| for (final part in msg.parts) { | ||
| if (part.isUiInteractionPart) { | ||
| buffer.write(part.asUiInteractionPart!.interaction); | ||
| } else if (part is genui.TextPart) { | ||
| buffer.write(part.text); | ||
| } | ||
| } | ||
|
|
||
| if (buffer.isEmpty) return; | ||
|
|
||
| final text = buffer.toString(); | ||
| final response = await _chatSession.sendMessage(Content.text(text)); | ||
|
|
||
| if (response.text?.isNotEmpty ?? false) { | ||
| _transport.addChunk(response.text!); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| 9. **Listen to Conversation Events:** | ||
| Create stubbed-out methods in your State class for each event type, including DartDoc comments explaining their required behavior. | ||
| ```dart | ||
| /// Updates state to include the new [surfaceId] so a new `Surface` widget can be built. | ||
| void _onSurfaceAdded(String surfaceId) { | ||
| // TODO: Implement state update to add surfaceId | ||
| } | ||
|
|
||
| /// Updates state to remove the [surfaceId] so its `Surface` widget is no longer built. | ||
| void _onSurfaceRemoved(String surfaceId) { | ||
| // TODO: Implement state update to remove surfaceId | ||
| } | ||
|
|
||
| /// Handles displaying raw text content received from the AI to the user. | ||
| void _onContentReceived(String text) { | ||
| // TODO: Implement displaying the received text | ||
| } | ||
|
|
||
| /// Handles errors that occur during the conversation appropriately. | ||
| void _onError(Object error) { | ||
| // TODO: Implement error handling | ||
| } | ||
| ``` | ||
|
|
||
| Subscribe to the `_conversation.events` to track when UI surfaces or chat messages arrive, dispatching them to the appropriate stubbed out methods: | ||
| ```dart | ||
| _conversation.events.listen((event) { | ||
| switch (event) { | ||
| case ConversationSurfaceAdded added: | ||
| _onSurfaceAdded(added.surfaceId); | ||
| case ConversationSurfaceRemoved removed: | ||
| _onSurfaceRemoved(removed.surfaceId); | ||
| case ConversationContentReceived content: | ||
| _onContentReceived(content.text); | ||
| case ConversationError error: | ||
| _onError(error.error); | ||
| default: | ||
| } | ||
| }); | ||
| ``` | ||
|
|
||
| 10. **Initialize System Prompt:** | ||
| Use `PromptBuilder` to give the AI basic instructions, then send it as a system message. | ||
| ```dart | ||
| final promptBuilder = PromptBuilder.chat( | ||
| catalog: catalog, | ||
| instructions: 'You are a helpful assistant. Respond to messages in a chatty way.', | ||
| ); | ||
| _conversation.sendRequest(ChatMessage.system(promptBuilder.systemPrompt)); | ||
| ``` | ||
|
|
||
| 11. **Display Surfaces:** | ||
| In your Flutter `build()` method, use the `Surface` widget wherever you need to render GenUI widgets using the `surfaceIds` you collected in step 9. | ||
| ```dart | ||
| Surface(surfaceContext: _controller.contextFor(surfaceId)); | ||
| ``` | ||
|
|
||
| 12. **Ask User for Input Preferences:** | ||
| **STOP AND ASK THE USER:** Ask the user for clarification on what UI elements should be used for user input. Explain that a `TextField` and `ElevatedButton` are good defaults, but you should not assume they want those exact widgets unless they clarify. | ||
|
|
||
| ## Constraints | ||
| - Do not make assumptions about user input UI elements; see step 12. | ||
| - Make sure to properly clean up GenUI controllers (`_transport.dispose()`, `_controller.dispose()`) inside the widget's `dispose()` method. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.