Skip to content

Handle chunked / partial data from server#70

Closed
cellerich wants to merge 1 commit intolanrat:mainfrom
cellerich:trmnl-selfhosted
Closed

Handle chunked / partial data from server#70
cellerich wants to merge 1 commit intolanrat:mainfrom
cellerich:trmnl-selfhosted

Conversation

@cellerich
Copy link
Copy Markdown

Updated two files to correctly process incoming data that may arrive in multiple fragments instead of a single complete payload.

Updated two files to correctly process incoming data that may arrive in
multiple fragments instead of a single complete payload.
@lanrat
Copy link
Copy Markdown
Owner

lanrat commented Mar 11, 2026

Thanks for looking into this! Several of your improvements are great — checking http.begin() return value, adding waitForWiFi(), better error logging, and explicit timeouts. I'd love to see those in a follow-up.

However, after some research, I think the chunk-parsing approach needs to be reworked.

Use the library's built-in chunked handling

The ESP32 HTTPClient already handles chunked transfer encoding transparently — see the source (search for HTTPC_TE_CHUNKED). The reason we see raw chunk data is that httpGet() reads from http.getStreamPtr() (raw WiFiClient*), bypassing the library's decoding. The fix should be using http.writeToStream() in httpGet() instead.

Concerns with manual chunk parsing in trmnl.cpp

  • The detection heuristic (buff[0] >= '0' && buff[0] <= 'f') can false-positive on valid JSON starting with a hex character, and the trailing removal (jsonLen -= 4) could corrupt valid data.
  • Parsing HTTP transport encoding in each consumer is fragile — if we fix it in the network layer, no caller needs to worry about it.

Suggested approach

  1. Replace the manual stream->readBytes() loop in httpGet() with http.writeToStream() to handle chunked encoding transparently.
  2. Update *defaultLen with actual bytes received.
  3. Keep your good additions: http.begin() check, waitForWiFi(), timeouts, error logging.
  4. Drop the chunk-parsing code from trmnl.cpp.

Would you be open to reworking along these lines?

Thanks again!

@cellerich
Copy link
Copy Markdown
Author

I have to admit that I "vibe coded" the manual chunk parsing just to get the thing working.
Thank you for the deeper investigation. I will try to implement your suggested approach.

Will close this PR now and we'll open a new one once I'm able to get the new code.
I cannot commit on a timeline, but I'm sure I should be able to do it within the next seven days.

@cellerich cellerich closed this Mar 12, 2026
@lanrat
Copy link
Copy Markdown
Owner

lanrat commented Mar 12, 2026

No worries. I also used an LLM to review the submission and investigate if the underlying library already had support.

I'd be happy to take a look at another PR whenever you are ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants