Skip to content

feat(connectors): use TCP client for Flink sink#3657

Open
Standing-Man wants to merge 4 commits into
apache:masterfrom
Standing-Man:flink-transport
Open

feat(connectors): use TCP client for Flink sink#3657
Standing-Man wants to merge 4 commits into
apache:masterfrom
Standing-Man:flink-transport

Conversation

@Standing-Man

@Standing-Man Standing-Man commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Rationale

The Flink connector used different transports for reading and writing messages. The source already used the native TCP client, while the sink sent messages through the HTTP client and hardcoded the HTTP port to 3000.

This caused the same IggyConnectionConfig to behave differently between the source and sink. It also required deployments to expose the HTTP API even when the configured Iggy endpoint used the TCP port.

The source and sink also parsed server addresses independently. The source relied on splitting the address by :, while the sink used URI parsing. This produced inconsistent behavior and did not reliably support IPv6 or validate malformed endpoints.

What changed?

  • Replaced IggyHttpClient with the blocking IggyTcpClient in the Flink sink.
  • Configured the TCP client using the existing connection settings:server host and port
    • username and password
    • connection and request timeouts
    • retry count and backoff
    • TLS
  • Updated IggySinkWriter to send messages through TCP.
  • Updated sink shutdown to flush pending messages and close the TCP client, while preserving both flush and close failures.
  • Added a shared TcpEndpoint parser used by both the Flink source and sink.
  • Added support for:host-only addresses using the default TCP port 8090
    • host:port
    • tcp://host:port
    • bracketed IPv6 addresses
  • Added validation for unsupported schemes, missing hosts, invalid ports, and addresses containing user info, paths, queries, or fragments.
  • Added unit tests covering valid and invalid TCP endpoint formats.
  • Ignored Gradle-generated .kotlin cache directories.

Local Execution

  • Passed / not passed
    Passed
  • Pre-commit hooks ran / not ran
    Ran

AI Usage

Signed-off-by: StandingMan <jmtangcs@gmail.com>
@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 12, 2026
@Standing-Man
Standing-Man marked this pull request as draft July 12, 2026 08:24
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Jul 12, 2026
Signed-off-by: StandingMan <jmtangcs@gmail.com>
@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 41.93548% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.02%. Comparing base (313c50b) to head (4f54771).

Files with missing lines Patch % Lines
...ache/iggy/connector/flink/sink/IggySinkWriter.java 0.00% 17 Missing ⚠️
...org/apache/iggy/connector/flink/sink/IggySink.java 0.00% 13 Missing ⚠️
.../org/apache/iggy/connector/config/TcpEndpoint.java 89.65% 2 Missing and 1 partial ⚠️
...apache/iggy/connector/flink/source/IggySource.java 0.00% 3 Missing ⚠️

❌ Your patch check has failed because the patch coverage (41.93%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3657      +/-   ##
============================================
+ Coverage     73.98%   74.02%   +0.04%     
- Complexity      937      956      +19     
============================================
  Files          1301     1301              
  Lines        147354   147239     -115     
  Branches     122911   122768     -143     
============================================
- Hits         109013   108990      -23     
+ Misses        34868    34777      -91     
+ Partials       3473     3472       -1     
Components Coverage Δ
Rust Core 74.32% <ø> (+0.04%) ⬆️
Java SDK 62.59% <41.93%> (+0.14%) ⬆️
C# SDK 72.12% <ø> (-0.07%) ⬇️
Python SDK 92.17% <ø> (ø)
PHP SDK 84.29% <ø> (ø)
Node SDK 91.35% <ø> (+0.09%) ⬆️
Go SDK 42.71% <ø> (ø)
Files with missing lines Coverage Δ
.../org/apache/iggy/connector/config/TcpEndpoint.java 89.65% <89.65%> (ø)
...apache/iggy/connector/flink/source/IggySource.java 0.00% <0.00%> (ø)
...org/apache/iggy/connector/flink/sink/IggySink.java 0.00% <0.00%> (ø)
...ache/iggy/connector/flink/sink/IggySinkWriter.java 0.00% <0.00%> (ø)

... and 12 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Standing-Man
Standing-Man marked this pull request as ready for review July 12, 2026 08:52
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 12, 2026
Signed-off-by: StandingMan <jmtangcs@gmail.com>
@hubcio

hubcio commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@mmodzelewski and also please check this one :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flink connector - transport - http to tcp

2 participants