Restore tokenizer special-id patches in attribution tests - #96
Conversation
|
CI note: the failing The failure is in the I reproduced locally that latest pyright (
Likely cause is the unpinned dev dependency |
|
Yes, that's the conclusion I reached as well - the issue seems to have appeared in a pyright past 1.1.406. The torch issues appear to be noise, and can been suppressed, but some of the issues are legitimate. I've kept pyright unpinned, but suppressed the torch issues, while fixing the other ones, in #99 . I think it should be possible to merge this now! |
a5daa00 to
367dd3c
Compare
hannamw
left a comment
There was a problem hiding this comment.
This looks good, and seems like a sensible change. Thanks for contributing!
Summary
This PR prevents TransformerLens attribution tests from leaking tokenizer class state across tests.
The Gemma and Llama dummy-model tests temporarily patch tokenizer.all_special_ids so attribution treats token 0 as special. Previously, the tests assigned directly to the tokenizer class property without restoring the original value.
This change:
patch_tokenizer_special_ids(...)context managerall_special_idsmutation from dummy model loadersWhy
This avoids leaking global tokenizer class state across tests and follows the safer restore pattern already used by the NNSight attribution tests.
Tests
Run:
python -m ruff check tests/test_attributions_gemma.py tests/test_attributions_llama.pypython -m ruff format --check tests/test_attributions_gemma.py tests/test_attributions_llama.pypython -m pytest tests/test_attributions_gemma.py::test_small_gemma_model tests/test_attributions_llama.py::test_small_llama_modelResults:
ruff check: passed
ruff format --check: passed
pytest: 2 passed
Not run: