Enforce a recursion-depth ceiling in json.dumps serialization#438
Enforce a recursion-depth ceiling in json.dumps serialization#438dsp-ant wants to merge 1 commit into
Conversation
The serialize_value/serialize_sequence/serialize_dict helpers are mutually recursive and run entirely within one bytecode instruction, so the VM's per-instruction recursion check never fires. The cycle detector catches reference loops but not deep acyclic chains, which can overflow the host's native stack. Add a depth ceiling matching the JSON_RECURSION_LIMIT already used by json.loads, raising a catchable RecursionError once exceeded.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Merging this PR will not alter performance
Comparing 
Footnotes
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
| /// Maximum nesting depth accepted by `json.dumps()`. | ||
| /// | ||
| /// Mirrors `JSON_RECURSION_LIMIT` on the `json.loads()` side: serialization is | ||
| /// mutually recursive across `serialize_value`/`serialize_sequence`/`serialize_dict` | ||
| /// and would otherwise overflow the host's native stack on a deep (acyclic) | ||
| /// structure that the cycle detector cannot catch. CPython raises a catchable | ||
| /// `RecursionError` here; we map the depth-limit `ResourceError` to the same. | ||
| const JSON_DUMP_RECURSION_LIMIT: usize = 200; |
There was a problem hiding this comment.
Why does this use a constant rather than respect the ResourceLimit recursion limit?
Separately, it seems like CPython does not, a 20000-deep nested tuple will serialize fine even if sys.setrecursionlimit is set very low.
Seems like CPython uses stack overflow protections for these cases, if I write a million element tuple I eventually get
RecursionError: Stack overflow (used 8144 kB) while encoding a JSON object
... maybe #440?
|
I decided to solve this case with the general recursion limit mechanism for now, see #454 Thanks again for the report & patch, will close this one. |
2 participants
The serialize_value/serialize_sequence/serialize_dict helpers are mutually recursive and run entirely within one bytecode instruction, so the VM's per-instruction recursion check never fires. The cycle detector catches reference loops but not deep acyclic chains, which can overflow the host's native stack.
Add a depth ceiling matching the JSON_RECURSION_LIMIT already used by json.loads, raising a catchable RecursionError once exceeded.
Summary by cubic
Adds a recursion-depth ceiling to
json.dumpsto prevent native stack overflows on deeply nested acyclic structures. Matches thejson.loadslimit and raises a catchableRecursionErrorwhen exceeded.JSON_DUMP_RECURSION_LIMIT(200) in the serializer and map overflow toRecursionError.Written for commit 0582012. Summary will update on new commits.