Follow up to discussion #1399, opened as requested. I split this into two issues because they are two distinct defects. If you would rather keep one thread, feel free to close the other one as a duplicate.
Test file, already attached to the discussion: tj-vs-TJ.pdf (878 bytes, three lines: the same string drawn with a single TJ, then with one Tj per glyph positioned with Td, then with two TJ next to each other with no space between them).
What changed
In 0.1.13 TextSequence followed TJ only. Since 0.1.14 (f732718, #1241) it also increments on Tj, and on 0.1.15 the number of distinct TextSequence values on a page is exactly the number of Tj plus TJ operations. I measured that on four real world PDFs and on the attached file. To be clear, I am not reporting the 0.1.14 double increment on TJ: fee5ce1 (#1298) fixed that, and 0.1.15 is the correct and final form of that counter.
Effect
I group letters by TextSequence and then run NearestNeighbourWordExtractor.Instance inside each group. On the attached file that gives 6 words on 0.1.13 and 14 words on 0.1.15, because the line drawn with one Tj per glyph is split into ten single letter groups. Without the grouping, both versions give 5 words. Downstream, in a CAD import pipeline, that one line goes from 2 text entities to 10.
Correction on my own example
You are right about the third line of that file: PartNumber is more correct than Part Number, since the two runs are adjacent with no space and a human reader reads them joined. That example was badly chosen on my side. The case I am reporting is the second line, where a single word becomes ten separate letters and there is no reading under which that is correct.
Why I grouped in the first place
I tried dropping the pre-grouping entirely and passing the whole page to the extractor. Across my own corpus, two documents that were correct become wrong and one gets better. So the geometric extractor on its own is not a drop in replacement for me, but I do not have a minimal reproducible example that shows this, and I would rather say so than hand you a claim you cannot check.
What I am asking for
A supported way to know where a show run begins and ends, whatever shape that takes. You already mentioned that ShowText is not virtual at the moment (BaseStreamProcessor.cs:240 on v0.1.15), which is what blocks doing this from outside the library.
Separately, the doc comment on the counter (BaseStreamProcessor.cs:84 to :88) still describes it as exposing "internal grouping of letters used by the PDF creator which may correspond to the intended grouping of letters into words". As you said in the discussion, it is now closer to the rendering order of ShowText. That comment is probably worth aligning either way.
My reading of the code, not measured on the attached file
Two internal consumers read the same counter, so the change is not limited to whatever a caller does with it. UnsupervisedReadingOrderDetector.Instance defaults to useRenderingOrder true (UnsupervisedReadingOrderDetector.cs:43 and :74), and GetBeforeInRendering compares the average TextSequence of two blocks (:176 to :181). When every letter shares one sequence value, that comparison is always false and contributes nothing; once every glyph has its own value it contributes an edge for almost every pair. On one of my own CAD pages (662 blocks, not attached) the resulting block order differs from 0.1.13 at every position, and constructing the detector with useRenderingOrder false reproduces the 0.1.13 order exactly. RenderingReadingOrderDetector orders purely by that average (RenderingReadingOrderDetector.cs:25 and :34), and ContentOrderTextExtractor.cs:73 also reads it.
Follow up to discussion #1399, opened as requested. I split this into two issues because they are two distinct defects. If you would rather keep one thread, feel free to close the other one as a duplicate.
Test file, already attached to the discussion: tj-vs-TJ.pdf (878 bytes, three lines: the same string drawn with a single TJ, then with one Tj per glyph positioned with Td, then with two TJ next to each other with no space between them).
What changed
In 0.1.13 TextSequence followed TJ only. Since 0.1.14 (f732718, #1241) it also increments on Tj, and on 0.1.15 the number of distinct TextSequence values on a page is exactly the number of Tj plus TJ operations. I measured that on four real world PDFs and on the attached file. To be clear, I am not reporting the 0.1.14 double increment on TJ: fee5ce1 (#1298) fixed that, and 0.1.15 is the correct and final form of that counter.
Effect
I group letters by TextSequence and then run NearestNeighbourWordExtractor.Instance inside each group. On the attached file that gives 6 words on 0.1.13 and 14 words on 0.1.15, because the line drawn with one Tj per glyph is split into ten single letter groups. Without the grouping, both versions give 5 words. Downstream, in a CAD import pipeline, that one line goes from 2 text entities to 10.
Correction on my own example
You are right about the third line of that file: PartNumber is more correct than Part Number, since the two runs are adjacent with no space and a human reader reads them joined. That example was badly chosen on my side. The case I am reporting is the second line, where a single word becomes ten separate letters and there is no reading under which that is correct.
Why I grouped in the first place
I tried dropping the pre-grouping entirely and passing the whole page to the extractor. Across my own corpus, two documents that were correct become wrong and one gets better. So the geometric extractor on its own is not a drop in replacement for me, but I do not have a minimal reproducible example that shows this, and I would rather say so than hand you a claim you cannot check.
What I am asking for
A supported way to know where a show run begins and ends, whatever shape that takes. You already mentioned that ShowText is not virtual at the moment (BaseStreamProcessor.cs:240 on v0.1.15), which is what blocks doing this from outside the library.
Separately, the doc comment on the counter (BaseStreamProcessor.cs:84 to :88) still describes it as exposing "internal grouping of letters used by the PDF creator which may correspond to the intended grouping of letters into words". As you said in the discussion, it is now closer to the rendering order of ShowText. That comment is probably worth aligning either way.
My reading of the code, not measured on the attached file
Two internal consumers read the same counter, so the change is not limited to whatever a caller does with it. UnsupervisedReadingOrderDetector.Instance defaults to useRenderingOrder true (UnsupervisedReadingOrderDetector.cs:43 and :74), and GetBeforeInRendering compares the average TextSequence of two blocks (:176 to :181). When every letter shares one sequence value, that comparison is always false and contributes nothing; once every glyph has its own value it contributes an edge for almost every pair. On one of my own CAD pages (662 blocks, not attached) the resulting block order differs from 0.1.13 at every position, and constructing the detector with useRenderingOrder false reproduces the 0.1.13 order exactly. RenderingReadingOrderDetector orders purely by that average (RenderingReadingOrderDetector.cs:25 and :34), and ContentOrderTextExtractor.cs:73 also reads it.