Currently, the program is bottlenecked by the fact that we are only reading one line at a time.
I might lead to serious performance gains if we limited the number of I/O operations (especially writes) by reading a variable number of lines. This number could be
- specified by the user,
- inferred from a given memory target and a line length heuristic, or
- inferred from a memory limit and actual line lengths and therefore dynamically change per batch.