Skip to content

Conversation

@khaledh
Copy link
Contributor

@khaledh khaledh commented Nov 17, 2025

What is the purpose of the change

Flink's protobuf format fails when processing messages compiled with protobuf-java's field naming conflict resolution. When proto schemas contain certain field patterns, protoc appends field number suffixes to accessor methods (e.g., getStatus1(), getTags4List()), but Flink's codegen assumes canonical names (getStatus(), getTagsList()). Conflict scenarios:

  • Repeated field vs *_list/*_count fields -> numbered suffixes
  • Enum field + *_value field -> both get numbered suffixes (protobuf-java 4.30.0+)

This affects users with conflict-prone schemas on protobuf-java 3.x+ (repeated field conflicts) and 4.30.0+ (enum field conflicts).

Brief change log

  • Added PbFieldConflictResolver with caching for conflict-aware accessor resolution
  • Refactored deserializer/serializer codegen to use conflict-aware accessors

Verifying this change

This change added tests and can be verified as follows:

  • Added test_field_conflict.proto with field conflict scenarios
  • Generated Java classes with protoc 4.32.1 to verify accessor name matching
  • Added FieldConflictProtoToRowTest (14 tests) and FieldConflictRowToProtoTest (4 tests)
  • All tests pass with protobuf-java 4.32.1

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no (the changes only affect which accessor methods are called during codegen, not the serialization format or behavior)
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

@flinkbot
Copy link
Collaborator

flinkbot commented Nov 17, 2025

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants