Fix index misalignment in char_wb ngram padding

[adityaanikam]

Fixes #8416

Root cause

get_char_ngrams() pads each token with self.delimiter on both sides before generating character ngrams:

padding = Series(self.delimiter).repeat(len(tokens))

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• Bug Fixes
  • Fixed character n-gram extraction to preserve correct positional alignment when constructing padded tokens.
  • Improved consistency of generated character features for inputs with non-sequential indexes.

Walkthrough

get_char_ngrams now resets token and padding indexes before concatenation. This preserves positional alignment when constructing padded character tokens.

Changes

Character n-gram generation

Layer / File(s)	Summary
Reset indexes before token concatenation python/cuml/cuml/feature_extraction/_vectorizers.py	get_char_ngrams resets token and padding indexes before concatenating padded character tokens.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: jcrist, csadorf

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the index misalignment fix in character word-boundary n-gram padding.
Description check	✅ Passed	The description explains the root cause and directly relates to the character word-boundary n-gram padding fix.
Linked Issues check	✅ Passed	The index reset changes address issue #8416 by preventing misalignment and duplicate labels during char_wb n-gram generation.
Out of Scope Changes check	✅ Passed	The changes are limited to character n-gram padding index alignment and match the linked issue objectives.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Fix failing CI checks

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

python/cuml/cuml/feature_extraction/_vectorizers.py (1)

216-224: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add an enabled char_wb Tfidf regression test.

In python/cuml/tests/test_text_feature_extraction.py, test TfidfVectorizer.fit_transform(analyzer="char_wb", ngram_range=(2, 6)). Compare the sparse output and feature names with scikit-learn using get_feature_names_out().

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/cuml/cuml/feature_extraction/_vectorizers.py` around lines 216 - 224,
Add an enabled regression test in test_text_feature_extraction.py covering
TfidfVectorizer with analyzer="char_wb" and ngram_range=(2, 6). Fit and
transform the same input with cuML and scikit-learn, then compare sparse outputs
and feature names using get_feature_names_out().

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@python/cuml/cuml/feature_extraction/_vectorizers.py`:
- Around line 216-224: Add an enabled regression test in
test_text_feature_extraction.py covering TfidfVectorizer with analyzer="char_wb"
and ngram_range=(2, 6). Fit and transform the same input with cuML and
scikit-learn, then compare sparse outputs and feature names using
get_feature_names_out().

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 80ca255e-7d3d-41f0-ab51-594df4f04c33

📥 Commits

Reviewing files that changed from the base of the PR and between 74f5ffa and 3edfa37.

📒 Files selected for processing (1)

• python/cuml/cuml/feature_extraction/_vectorizers.py