feat(genai): Add local tokenizer samples for Count and Compute

[msampathkumar]

Description

Fixes b_443755237

Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.

Checklist

• Tests pass: nox -s py-3.9 (see Test Environment Setup)
• Lint pass: nox -s lint (see Test Environment Setup)
• Please merge this PR for me once it is approved

[gemini-code-assist]

Summary of Changes

Hello @msampathkumar, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the genai module by integrating new local tokenizer samples for both 'Count' and 'Compute' functionalities. It includes the addition of the sentencepiece library as a new dependency to support these features and extends the test suite with corresponding validation tests. The primary goal is to provide more robust and flexible token counting examples, particularly for scenarios requiring local tokenization.

Highlights

• New Local Tokenizer Samples: Introduced new code samples demonstrating local tokenization capabilities for 'Count' and 'Compute' operations within the genai module.
• New Dependency Added: The sentencepiece library (version 0.2.1) has been added to the requirements.txt file, indicating its necessity for the new local tokenizer functionalities.
• Expanded Test Coverage: New test cases have been added to test_count_tokens_examples.py to validate the functionality of the newly introduced local tokenizer samples.
• Example Output Correction: A minor correction was made to an example output comment in counttoken_with_txt.py, updating the total_tokens value from 10 to 9.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces new samples for local tokenization with GenAI, including adding the sentencepiece dependency and new test cases. The changes are generally good, but I've identified a couple of areas for improvement in the new test code. Specifically, the function names in the new sample modules are identical to the module names, which is unconventional and can be confusing. I've suggested renaming them to be more descriptive and consistent with the existing codebase. This will improve code readability and maintainability.

[gemini-code-assist]

The function name counttoken_localtokenizer_with_txt is the same as its module name, which can be confusing. For better readability and consistency with other examples in this file (e.g., counttoken_with_txt.count_tokens()), consider renaming the function within the counttoken_localtokenizer_with_txt module to something more descriptive, like count_tokens().

[gemini-code-assist]

Similar to the other new test, the function name counttoken_localtokenizer_compute_with_txt is identical to its module name. This is unconventional and reduces clarity. To maintain consistency with other examples like counttoken_compute_with_txt.compute_tokens_example(), please consider renaming the function inside the counttoken_localtokenizer_compute_with_txt module to something like compute_tokens_example() or compute_tokens_locally().

[snippet-bot]

Here is the summary of changes.

You are about to add 2 region tags.

• genai/count_tokens/counttoken_localtokenizer_compute_with_txt.py:17, tag googlegenaisdk_counttoken_localtokenizer_compute_with_txt
• genai/count_tokens/counttoken_localtokenizer_with_txt.py:17, tag googlegenaisdk_counttoken_localtokenizer_with_txt

---

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

• Refresh this comment