1 by xyuanlu · Pull Request #52 · xyuanlu/helix

xyuanlu · 2023-05-24T01:07:21Z

Issues

My PR addresses the following Helix issues and references them in the PR description:

(apache#200 - Link your issue number here: You can write "Fixes #XXX". Please use the proper keyword so that the issue gets closed automatically. See https://docs.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue
Any of the following keywords can be used: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved)

Description

Here are some details about my PR, including screenshots of any UI changes:

(Write a concise description including what, why, how)

Tests

The following tests are written for this issue:

(List the names of added unit/integration tests)

The following is the result of the "mvn test" command on the appropriate module:

(If CI test fails due to known issue, please specify the issue and test PR locally. Then copy & paste the result of "mvn test" to here.)

Changes that Break Backward Compatibility (Optional)

My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:

(Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)

Documentation (Optional)

In case of new functionality, my PR adds documentation in the following wiki page:

(Link the GitHub wiki you added)

Commits

My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Code Quality

My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)

Add new submodule meta client - a generic metadata client

Add listener interfaces and factory class for metaclient interface

Add Meta client factory and ZkMetaClient constructor

…implementation (apache#2333) Prepare zkclient for meta-client DataChangeListener add new method in IZkDataListener Implement boolean flag connectOnInit in zkclient for decoupling Override equals and hashcode for the converter class

Add basic CRUD for ZkMetaClient and test

Metaclient - Op and OpResult Logic for transactional suppor

Metaclient - Implementation of Create from CRUD operations

* Implement data change listener for ZkMetaClient and test

… zkclient (apache#2346) Prepare zkclient and implement new adapter for child change listener.

1. Implementation of the transactionOp() operation method in zkMetaClient. This method enables transactional support by converting metaclient operations into zk ops and running the zk.multi() method. The results are converted back from zk opResults to metaclient opResults. 2. Created a separate util class zkMetaClientUtil for Util functions and the logic behind the implementation of transactionOp().

Refactoring MetaClient Exception. Creating separate exception directory.

Rename and reformat metaclient test and util

Implements the direct child change listener in ZkMetaClient using the native ZkClient.

Modified transactionOp test case variable for clearer understanding. Added more comments to test cases.

Fix ZkClient retry logic for customized callback and test

Update async API for MetaClient.

Add wrapper file for meta client async callback class

Create separate API for persistent and one-time listener

Implementation of Container Nodes in ZkMetaClient

Implement zk Meta client async crud

TTL Node Interface Methods Co-authored-by: mapeng <mapeng@linkedin.com>

This change adds an extra check in ZkClient.connect so that it will only be invoked once. Also avoid close called in between of isclosed check and connection establishment in connect().

Add support for state change in ZkMetaClient

…pache#2385)

TTL Node Lattice Implementation

Implement timeout for auto reconnect ZkClient does auto reconnect when it get's disconnected from Zookeeper. There is no timeout when reconnect can not be established. In ZkMetaclient, we have a separate thread to monitor reconnect status and close ZkClient when connection can not be reestablished when timed out.

Refactor ZkClient for persist watch

Implementing LockInfo Object for LockClient in MetaClient --------- Co-authored-by: mapeng <mapeng@linkedin.com>

…he#2443) Add comment and description for ZkMetaClient reconnect handling

…gured) (apache#2432) Before ZK 3.6, all zk watchers are one time watcher. ZkClient used to resubscribe every time when an event happens. This change adopts ZK persist watcher.

…ient (apache#2439) ZkPathRecursiveWatcherTrie will be used for a registry for ZK persist recursive watcher. When persist recursive watcher is registered on path /X, ZK will send out data change for any data/child changing under the tree structure of /X. The event only include the path of changed ZNode. In ZkClient, when ever we get a dataChange event for path /X/Y/Z, we need to track back the path and notify all registered recursive persist listener and notify them about the node change. ref: https://zookeeper.apache.org/doc/r3.7.1/zookeeperProgrammers.html#sc_WatchPersistentRecursive

Update snapshot version for metaclient align with TOT of master branch

Add metaclient supporting files.

Add meta-client module in version bump script

…r flag is on (apache#2465) When Zk Client register a one time listener on a path hat has already a persist watcher registered. ZK will over write the persist watcher and only trigger event once when change happen. In Helix ZkClient, we should disable one time wacher registration when _usePersistWatcher is set to true.

ZkPathRecursiveWatcherTrie will be used as a registry of persistent recursive watchers. When persist recursive watcher is registered on path /X, ZK will send out data change for any data/child changing under the tree structure of /X. The event only include the path of changed ZNode. In ZkClient, when ever we get a dataChange event for path /X/Y/Z, we need to track back the path and notify all registered recursive persist listener and notify them about the node change. ref: https://zookeeper.apache.org/doc/r3.7.1/zookeeperProgrammers.html#sc_WatchPersistentRecursive

Implementation of LockClient for Lattice. Co-authored-by: mapeng <mapeng@linkedin.com>

…PI only * add recursive persist listener API in ZkClient and test native ZK

ZkClient add recursive persist listener implementation

xyuanlu and others added 30 commits May 20, 2023 14:12

Add new submodule meta client - a generic metadata client (apache#2234)

c4dcd1a

Add new submodule meta client - a generic metadata client

Add listener interfaces and config class (apache#2249)

ee05f8c

Add listener interfaces and factory class for metaclient interface

Add Meta client factory and ZkMetaClient constructor (apache#2291)

4bd435f

Add Meta client factory and ZkMetaClient constructor

Setup testng for meta-client module (apache#2326)

1fd3aae

Meta client - implement CRUD for zkMetaClient

a34902b

Add basic CRUD for ZkMetaClient and test

MetaClient - Op and OpResult Logic for transactional support

b78c871

Metaclient - Op and OpResult Logic for transactional suppor

Metaclient - Implementation of Create from CRUD operations

0dd1b24

Metaclient - Implementation of Create from CRUD operations

Implement data change listener for ZkMetaClient and test

8a26da3

* Implement data change listener for ZkMetaClient and test

Create adapter package for data and child change listener and prepare…

32b0b39

… zkclient (apache#2346) Prepare zkclient and implement new adapter for child change listener.

Refactoring MetaClient Exceptions

5b33147

Refactoring MetaClient Exception. Creating separate exception directory.

Rename and reformat metaclient test and util

f238c1a

Rename and reformat metaclient test and util

Implement direct child change listener in ZkMetaClient

e5e18d3

Implements the direct child change listener in ZkMetaClient using the native ZkClient.

Add metaclient code and code translator from ZkException code

73ed25d

Modified transactionOp test case variable for clearer understanding

f7516e3

Modified transactionOp test case variable for clearer understanding. Added more comments to test cases.

Fix ZkClient retry logic for customized callback and test

d85320e

Fix ZkClient retry logic for customized callback and test

Update async API for MetaClient.

944c3ed

Update async API for MetaClient.

Add wrapper file for meta client async callback class (apache#2372)

fc26345

Add wrapper file for meta client async callback class

Create separate API for persistent and one-time listener (apache#2376)

eb1233e

Create separate API for persistent and one-time listener

MetaClient Container Node Implementation (apache#2377)

cd742a5

Implementation of Container Nodes in ZkMetaClient

Implement zk Meta client async crud (apache#2354)

fab43a9

Implement zk Meta client async crud

Fix typos and styles in metaclient (apache#2382)

25f4869

TTL Node Interface Methods (apache#2388)

78535b3

TTL Node Interface Methods Co-authored-by: mapeng <mapeng@linkedin.com>

Add check in ZkClient.connect so it only called once.

a8b17ae

This change adds an extra check in ZkClient.connect so that it will only be invoked once. Also avoid close called in between of isclosed check and connection establishment in connect().

Add support for state change in ZkMetaClient

f43d2bf

Add support for state change in ZkMetaClient

Add retry policy class and config for operation retry for MetaClient (a…

faa434d

…pache#2385)

TTL Node Lattice Implementation (apache#2401)

37d3d3b

TTL Node Lattice Implementation

use reconnect timeout for crud retry timeout (apache#2410)

11f1c65

Marcosrico and others added 16 commits May 20, 2023 14:12

MetaClient LockClient Interface

a18fc65

Refactor ZkClient for persist watch (apache#2426)

721f986

Refactor ZkClient for persist watch

Add an option in metaclient to use persist watcher (apache#2434)

acb6611

Implementing LockInfo Object for LockClient in MetaClient

b9ccf3b

Implementing LockInfo Object for LockClient in MetaClient --------- Co-authored-by: mapeng <mapeng@linkedin.com>

Add comment and description for ZkMetaClient reconnect handling (apac…

ad1429e

…he#2443) Add comment and description for ZkMetaClient reconnect handling

Use persist watcher for listener registration in ZkClient (when confi…

e789a75

…gured) (apache#2432) Before ZK 3.6, all zk watchers are one time watcher. ZkClient used to resubscribe every time when an event happens. This change adopts ZK persist watcher.

Change snalshot version for metaclient (apache#2461)

54e8491

Update snapshot version for metaclient align with TOT of master branch

Add metaclient supporting files. (apache#2464)

36c011c

Add metaclient supporting files.

Add meta-client module in version bump script (apache#2471)

a04477b

Add meta-client module in version bump script

Add leader election API and dummy class (apache#2488)

1162154

Change pom of metaclient pom file(apache#2500)

a22616f

Implementation of LockClient for Lattice. (apache#2457)

7aafc9d

Implementation of LockClient for Lattice. Co-authored-by: mapeng <mapeng@linkedin.com>

Add recursive persist listener API in ZkClient and test native ZK - A…

6c53496

…PI only * add recursive persist listener API in ZkClient and test native ZK

xyuanlu force-pushed the recursivePersistImpl branch 5 times, most recently from 3d7f9d4 to cc536b9 Compare May 27, 2023 01:03

ZkClient add recursive persist listener implementation (apache#2506)

1df540f

ZkClient add recursive persist listener implementation

xyuanlu force-pushed the recursivePersistImpl branch 2 times, most recently from 7bd0ee8 to 7a8fa24 Compare June 1, 2023 16:25

change undetermined check in unit test

481acbf

xyuanlu force-pushed the recursivePersistImpl branch from 7a8fa24 to 481acbf Compare June 1, 2023 20:48

xyuanlu force-pushed the metaclient branch from dedd6ad to d2023a7 Compare July 19, 2023 04:18

xyuanlu force-pushed the metaclient branch from 29337ef to 4a43edf Compare August 28, 2023 01:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1#52

1#52
xyuanlu wants to merge 48 commits into
metaclientfrom
recursivePersistImpl

xyuanlu commented May 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

xyuanlu commented May 24, 2023

Issues

Description

Tests

Changes that Break Backward Compatibility (Optional)

Documentation (Optional)

Commits

Code Quality

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants