Output nested tables last#221
Open
JoelColledge wants to merge 1 commit into
Open
Conversation
The TOML format has a restriction that if a table itself contains
tables, all keys with non-table values must be emitted first. This led
to the following error:
$ echo '{"a": {}, "b": 1}' | rq -T
[ERROR] [rq] Encountered: TOML serialize error
[ERROR] [rq] Caused by: values must be emitted before tables
Fix this by using tables_last to output the tables after other keys.
This affects all output formats that serialize using serde. The ordering
is unimportant in the other formats, so that is harmless. The
performance impact should be small.
2785d9a to
84ce256
Compare
Author
|
I did a basic performance test with this and measured a 4% performance degradation relative to master. The test used release builds, converting json to json. Here is the bash hackery: { echo -n '{' ; for i in {0..99} ; do echo -n "\"a$i\": 0, \"b$i\": {}, " ; done ; echo '"a100": {}}' ; } > /tmp/test1
last=/tmp/test1 ; for a in {1..16} ; do next=/tmp/test1-$a ; cat $last $last > $next ; last=$next ; done
for i in {1..50} ; do for version in rq-master rq-tables_last ; do echo "$version $i" >&2 ; time { cat /tmp/test1-16 | ./$version > /dev/null ; } ; done ; done 2> out1-50 |
alerque
approved these changes
May 10, 2022
Contributor
|
Hm, this has the unfortunate side effect of removing length hints, which may make CBOR and similar formats longer in some cases (e.g. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The TOML format has a restriction that if a table itself contains
tables, all keys with non-table values must be emitted first. This led
to the following error:
$ echo '{"a": {}, "b": 1}' | rq -T
[ERROR] [rq] Encountered: TOML serialize error
[ERROR] [rq] Caused by: values must be emitted before tables
Fix this by using tables_last to output the tables after other keys.
This affects all output formats that serialize using serde. The ordering
is unimportant in the other formats, so that is harmless. The
performance impact should be small.
Fixes #194
I encountered this problem and came up with this solution. I can understand that you may not wish to merge it due to the effect on other formats. If not, any ideas how to solve it more cleanly?