Consider an optional google-re2 path for default scrubbing

[dsfaccini]

This issue was posted by Grok Build using grok-4.6 on behalf of David.

Follow-up to #2190. The start-char prefilter is the right stdlib change. A DFA engine is still much faster on large attribute values.

Measurements

Median Pattern.search on Python 3.13, ~8k synthetic JSON attribute:

engine	benign	matching
stdlib re, no prefilter	361µs	362µs
#2190 1-char prefilter	74µs	75µs
2-char / 3-char prefilter	222–240µs	slower than 1-char
PyPI regex	123µs	123µs
google-re2 (case-insensitive)	5.7µs	10.8µs

google-re2 was about 13× faster than the prefiltered re on the benign 8k haystack, and about 15× on ~40k.

Why this is not a hard dependency

• auth(?!ors?\b) does not compile. RE2 has no lookaround. See RE2 syntax and google-re2.
• Match objects are re2._Match, not re.Match. ScrubMatch.pattern_match is part of the public callback API.
• Unicode case folding is not CPython re.IGNORECASE. credentıal matches in re and misses in RE2. See re.IGNORECASE and re2#262.
• Latest google-re2 wheels cover CPython 3.10–3.14 on macOS, manylinux, and Windows. There are no musllinux wheels. Alpine users would need a source build.
• User extra_patterns are documented as ordinary re strings and may use lookaround.

PyPI regex is the wrong follow-up. It is still a backtracker, slower than the #2190 prefilter, and it also misses credentıal.

Suggested shape if we do this

Keep stdlib re as the default. Do not add google-re2 as a required dependency.

A later change could try google-re2 only for the default patterns, keep re for extra_patterns and for auth(?!ors?\b) (or a post-filter that preserves group(0) == 'auth'), and pick the leftmost match. Preserve ScrubMatch.pattern_match.group(0) / .span() for callbacks.

Sentry does not regex-scan values; it denylists keys. See Sentry scrubbing.

[dmontagu]

Investigation notes for a future revisit — I don't think we need to take this over the line now, but this seems worth recording so the constraints and likely implementation don't have to be rediscovered.

The main conclusion is that a conservative version should be fairly non-invasive: one focused, small-to-medium PR rather than a larger scrubbing redesign. However, simply trying to compile the current combined expression with RE2 and falling back on an exception would not be safe:

• the start-character prefilter uses positive lookahead;
• auth(?!ors?\b) uses negative lookahead;
• patterns can compile under both engines but still differ semantically around Unicode case folding, \b, \d, and some anchors;
• callbacks currently receive a public re.Match via ScrubMatch.pattern_match.

A safe shape would probably be:

1. Keep the current stdlib expression as the canonical matcher.
2. If google-re2 is installed, use it only as a candidate finder for sufficiently large ASCII strings and the fixed RE2-compatible default patterns.
3. Keep auth(?!ors?\b) and every user extra_pattern on stdlib re.
4. Search the RE2 defaults and the stdlib fallback separately, pick the leftmost position (defaults win ties), then call the canonical stdlib pattern with .match(value, position). That preserves an actual re.Match, including the documented group(0) and span().
5. Use the current stdlib-only path when RE2 is unavailable or broken, the value is non-ASCII, or the value is below a size threshold.

Keeping custom patterns on re seems important for an initial version. “RE2 accepts this syntax” isn't enough to establish Python-compatible behavior, so attempting to accelerate arbitrary compatible-looking patterns would add considerably more risk and complexity.

I also did a quick local probe with the official google-re2 package. Passing ASCII bytes to RE2 is important because it avoids the wrapper's UTF-8 character-offset conversion. Short easy misses can be slower through the binding (roughly 1 µs versus 0.2–0.5 µs in my synthetic cases), so a threshold around 256–512 characters is probably sensible. From about 512 characters onward the split design was already faster in those probes; on 8K strings it ranged from roughly 1.5× for inputs the existing prefix handles especially well to roughly 15–25× when many characters are plausible default-pattern starts. That is directionally consistent with the measurements above, though a representative benchmark should choose the threshold.

Implementation-wise, I'd expect roughly 60–100 production lines isolated behind a small .search() wrapper, plus semantic and precedence tests, an optional extra such as logfire[re2], a lockfile update, and a short docs note. No public configuration change should be needed.

So: probably worth retaining as a possible future optimization, but not compelling enough to prioritize now. If revisited, I would scope RE2 to the known defaults and retain stdlib re as both the semantic authority and full fallback.