Skip to content

Add Gemma3-270M model configuration support #500 #501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add Gemma3-270M model configuration support #500 #501

wants to merge 1 commit into from
+26 −0

Conversation

chethanuk
Copy link

Add ModelConfig.gemma3_270m() classmethod with architecture specs:

  • 18 layers, 640 embed_dim, 2048 hidden_dim
  • 4 heads (256 head_dim), 1 KV head (GQA)
  • 512 sliding window, 262144 vocab size
  • Add checkpoint path constants (GEMMA3_270M_PT, GEMMA3_270M_IT)

Resolves #500

image

It's a good idea to open an issue first for discussion.

Reference

Colab Notebook

Checklist

  • I have added all the necessary unit tests for my change.
  • I have verified that my change does not break existing code and all unit tests pass.
  • I have added all appropriate doc-strings/documentation.
  • My PR is based on the latest changes of the main branch (if unsure, rebase the code).
  • I have signed the Contributor License Agreement.
  • I have followed Contribution Guidelines.

@chethanuk
Add Gemma3-270M model configuration support
b6528ab 
- Add ModelConfig.gemma3_270m() classmethod with architecture specs:
  * 18 layers, 640 embed_dim, 2048 hidden_dim
  * 4 heads (256 head_dim), 1 KV head (GQA)
  * 512 sliding window, 262144 vocab size
- Add checkpoint path constants (GEMMA3_270M_PT, GEMMA3_270M_IT)
- Add test coverage for gemma3-270m model routing
- Verified against official HuggingFace config

Resolves google#500
@tianshub tianshub self-requested a review October 5, 2025 03:58
tianshub
tianshub approved these changes Oct 5, 2025
View reviewed changes
Copy link
Collaborator

@tianshub tianshub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the support! Btw, you don't need to manually merge from github, it will be automatically merged once the internal test passed.

abheesht17
abheesht17 reviewed Oct 5, 2025
View reviewed changes
Copy link
Contributor

@abheesht17 abheesht17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chethanuk - do you have a notebook comparing outputs with the reference model?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tianshub tianshub tianshub approved these changes

+1 more reviewer

@abheesht17 abheesht17 abheesht17 left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Can you add gemma3_model_lib.ModelConfig for Gemma3-270m models?
3 participants
@chethanuk @tianshub @abheesht17