[WIP] MOLT Replicator draft docs #20465
Conversation
Files changed:
|
✅ Deploy Preview for cockroachdb-api-docs canceled.
|
✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.
|
❌ Deploy Preview for cockroachdb-docs failed. Why did it fail? →
|
taroface
changed the title
There was a problem hiding this comment.
Excellent work @taroface . Not an easy doc to write, but you made it understandable and clean! Let's bottom out on some of these discussions and ensure the deprecation effort from @tuansydau reflects the reality of what we are documenting.
|
|
||
| {% include molt/molt-setup.md %} | ||
|
|
||
| ## Start Fetch |
There was a problem hiding this comment.
So an important note here is that as part of the deprecation of the wrapper, we're mainly removing the invocations of Replicator from MOLT. However, there is some source database Replication setup that we'll still need to perform for PostgreSQL specifically. The reason we have to do this is because we need to create the slot at the time we actually do the snapshot export so we don't have gaps in data.
So that means that we still need to document the behavior when we set certain pg-* flags for setting publication, slots and the relevant drop/recreate behavior. I think we'll need to discuss this a bit more in the next team meeting to clearly lay out what the behavior still is. CC @tuansydau @Jeremyyang920
There was a problem hiding this comment.
@taroface resolved here: #20465 (comment)
We resolved it during the call and Tuan's comment above should capture the behavior.
| ## Prepare the CockroachDB cluster | ||
|
|
||
| {{site.data.alerts.callout_success}} | ||
| For details on enabling CockroachDB changefeeds, refer to [Create and Configure Changefeeds]({% link {{ site.current_cloud_version }}/create-and-configure-changefeeds.md %}). |
There was a problem hiding this comment.
We need to also ensure that the license and organization are set:
SET CLUSTER SETTING cluster.organization = 'organization';
SET CLUSTER SETTING enterprise.license ='$LICENSE';
|
@taroface We should also make the user grab their own copy of instant client from Oracle. For the Linux instructions, we should replace: sudo apt-get install -yqq --no-install-recommends libaio1t64
sudo ln -s /usr/lib/x86_64-linux-gnu/libaio.so.1t64 /usr/lib/x86_64-linux-gnu/libaio.so.1
curl -o /tmp/ora-libs.zip https://replicator.cockroachdb.com/third_party/instantclient-basiclite-linux-amd64.zip
unzip -d /tmp /tmp/ora-libs.zip
sudo mv /tmp/instantclient_21_13/* /usr/lib
export LD_LIBRARY_PATH=/usr/libWith: sudo apt-get install -yqq --no-install-recommends libaio1t64
sudo ln -s /usr/lib/x86_64-linux-gnu/libaio.so.1t64 /usr/lib/x86_64-linux-gnu/libaio.so.1
# Download the Oracle Instant Client libraries from Oracle: (https://www.oracle.com/ca-en/database/technologies/instant-client.html) into /tmp/instantclient-basiclite-linux-amd64.zip for example
unzip -d /tmp /tmp/instantclient-basiclite-linux-amd64.zip
sudo mv /tmp/instantclient_21_13/* /usr/lib
export LD_LIBRARY_PATH=/usr/libLet me know if you have questions on this, it should be updated for each instance of the oracle instant client instructions throughout the docs. The actual links for Linux binaries: |
src/current/molt/molt-replicator.md
Outdated
|
|
||
| MOLT Replicator offers three consistency modes for balancing throughput and transactional guarantees: | ||
|
|
||
| 1. Consistent (default for CockroachDB sources): Preserves per-row order and source transaction atomicity. Concurrent transactions are controlled by `--parallelism`. |
There was a problem hiding this comment.
I know this is implied since the immediate section implies immediate is the default for postgres, mysql, and oracle, but should we call out that consistent and best effort only apply to "CRDB sources" or "failback mode"?
|
Can we link the replicator dashboard in the DOCS? |
| {% include_cached copy-clipboard.html %} | ||
| ~~~ sql | ||
| -- Query the current SCN from Oracle | ||
| SELECT CURRENT_SCN FROM V$DATABASE; |
There was a problem hiding this comment.
@noelcrl , can you please check the Fetch cursor logging to see if we end up printing the relevant SCN that folks should start from? I think right now this is a good step for folks who are using replicator directly (querying the scn directly), but I'm wondering if we need to include this if folks are doing this via Fetch since it should be able to access the current SCN and log out.
CC @tuansydau
There was a problem hiding this comment.
Yes, it is logged in dataexport/oracle.go:NewOracleSource() when starting Fetch:
logger.Info().Msgf(fmt.Sprintf("replication-only mode should include the following "+
"replicator flags: --backfillFromSCN %s --scn %s", replicationBackfillSCN, scn))The message should look like:
replication-only mode should include the following replicator flags: --backfillFromSCN 26685444 --scn 26685786
So if Fetch is being used for a bulk load, and this logic in Fetch isn't being removed, these SCNs can be used instead of doing all of these queries here to find it manually.
If just replication is necessary (no bulk data-load), the user can just grab the current SCN and just use it for both --backfillFromSCN and --scn
There was a problem hiding this comment.
I think we may need to change the logging here. Maybe we put it as part of the cursor specific logging like we do for MySQL and Postgres. CC @tuansydau . Tuan can you work with Noel to figure out the best way to do this?
There was a problem hiding this comment.
Yeah sure, I'll work with Noel on this
5 participants
DOC-13338
DOC-14748
This PR is still WIP.
Notes for reviewers: