-
Notifications
You must be signed in to change notification settings - Fork 37
[CHORE] CLI Command to replicate now_applications in local, dev or test #3834
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
matbusby-fw
wants to merge
8
commits into
develop
Choose a base branch
from
chore-cli-command
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
1bf1f47
feat: Implement data replication and generation tools, enhance migrat…
effe5b6
refactor: improve NOW application data replication by using explicit …
6621b82
refactor: simplify `create-test-data` command to exclusively generate…
d0fd1c9
feat: Add support for replicating NoW applications to OpenShift dev/t…
dc2fbc3
Add oc login step if user is not currently logged in.
35f340e
revert now_application_progress_resource changes (re-apply in subsequ…
5d6d804
remove unnecessary change
fb8293e
restore accidentally removed comment
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,225 @@ | ||
| #!/bin/bash | ||
| set -e | ||
|
|
||
| # Colors | ||
| GREEN='\033[0;32m' | ||
| BLUE='\033[0;34m' | ||
| YELLOW='\033[1;33m' | ||
| RED='\033[0;31m' | ||
| NC='\033[0m' # No Color | ||
|
|
||
| echo -e "${BLUE}--- MDS NoW Application Replicator ---${NC}" | ||
| echo "This script replicates a generic NoW Application from Production to your local environment OR a dev/test OpenShift environment." | ||
| echo "Prerequisites: 'oc' (logged into prod) and 'docker' must be installed." | ||
| echo "" | ||
|
|
||
| # Check dependencies | ||
| if ! command -v oc &> /dev/null; then | ||
| echo -e "${RED}Error: 'oc' command not found. Please install OpenShift CLI.${NC}" | ||
| exit 1 | ||
| fi | ||
| if ! command -v docker &> /dev/null; then | ||
| echo -e "${RED}Error: 'docker' command not found. Please install Docker.${NC}" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # --- Helper Function: Find Primary Crunchy DB Pod --- | ||
| find_primary_db_pod() { | ||
| local namespace=$1 | ||
| local selector="-n $namespace" | ||
| if [ -z "$namespace" ]; then | ||
| selector="" | ||
| fi | ||
|
|
||
| # Try to find the Crunchy Data Primary/Master pod | ||
| local candidates=$(oc get pods $selector -l postgres-operator.crunchydata.com/role=master -o custom-columns=":metadata.name" --no-headers 2>/dev/null | grep -v "clone" || true) | ||
| local master_pod=$(echo "$candidates" | grep "ha-" | head -n 1 || true) | ||
| master_pod=${master_pod:-$(echo "$candidates" | head -n 1)} | ||
|
|
||
| if [ -z "$master_pod" ]; then | ||
| # Fallback: legacy search | ||
| master_pod=$(oc get pods $selector -L postgres-operator.crunchydata.com/role --no-headers 2>/dev/null | grep "master" | grep -v "clone" | head -n 1 | awk '{print $1}') | ||
| fi | ||
| echo "$master_pod" | ||
| } | ||
|
|
||
| # --- Helper Function: Find Backend Pod --- | ||
| find_backend_pod() { | ||
| local namespace=$1 | ||
| # User specified we should look for core-api-*** but not core-api-celery-*** | ||
| # We list pods, grep for core-api, exclude celery, and take the first one. | ||
| local candidate=$(oc get pods -n "$namespace" -o name 2>/dev/null | grep "core-api" | grep -v "celery" | head -n 1 | cut -d/ -f2) | ||
|
|
||
| # Fallback to generic backend if core-api not found (just in case) | ||
| if [ -z "$candidate" ]; then | ||
| candidate=$(oc get pods -n "$namespace" -l app.kubernetes.io/name=backend -o custom-columns=":metadata.name" --no-headers 2>/dev/null | head -n 1) | ||
| fi | ||
|
|
||
| echo "$candidate" | ||
| } | ||
|
|
||
|
|
||
|
|
||
| # --- Helper Function: Check OpenShift Login --- | ||
| check_login() { | ||
| if ! oc whoami &> /dev/null; then | ||
| echo -e "${YELLOW}You are not logged into OpenShift.${NC}" | ||
| echo -e "Please generate a token from: ${BLUE}https://oauth-openshift.apps.silver.devops.gov.bc.ca/oauth/token/request${NC}" | ||
| echo -e "${YELLOW}...Paste Token Here...${NC}" | ||
| read -s OC_TOKEN | ||
| echo "" | ||
|
|
||
| if [ -z "$OC_TOKEN" ]; then | ||
| echo -e "${RED}Error: Token is required.${NC}" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "Logging in..." | ||
| oc login --token="$OC_TOKEN" --server=https://api.silver.devops.gov.bc.ca:6443 > /dev/null | ||
|
|
||
| if ! oc whoami &> /dev/null; then | ||
| echo -e "${RED}Error: Login failed.${NC}" | ||
| exit 1 | ||
| fi | ||
| echo -e "${GREEN}Successfully logged in as $(oc whoami)${NC}" | ||
| else | ||
| echo -e "Logged in as: ${GREEN}$(oc whoami)${NC}" | ||
| fi | ||
| } | ||
|
|
||
| echo "" | ||
| check_login | ||
| echo "" | ||
|
|
||
| # 1. Prompt for NoW Number | ||
| read -p "Enter NoW Number to be copied from Prod (e.g. 0400022-2025-01): " NOW_NUMBER | ||
| if [ -z "$NOW_NUMBER" ]; then | ||
| echo -e "${RED}Error: NoW Number is required.${NC}" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # 2. Prompt for Target Environment | ||
| echo "" | ||
| echo "Select Target Environment:" | ||
| echo "1) Local (default)" | ||
|
|
||
| # Fetch available OpenShift Namespaces in 4c2ba9, EXCLUDING prod | ||
| # Assuming the user has access to list projects. If not, we might need a manual entry or just fail gracefully. | ||
| PROJECTS=$(oc get projects --no-headers 2>/dev/null | awk '{print $1}' | grep "^4c2ba9-" | grep -vE "prod$|tools$" || true) | ||
|
|
||
| i=2 | ||
| declare -A PROJECT_MAP | ||
| if [ -n "$PROJECTS" ]; then | ||
| for proj in $PROJECTS; do | ||
| echo "$i) $proj" | ||
| PROJECT_MAP[$i]=$proj | ||
| i=$((i+1)) | ||
| done | ||
| fi | ||
|
|
||
| read -p "Select Target [1]: " TARGET_SELECTION | ||
| TARGET_SELECTION=${TARGET_SELECTION:-1} | ||
|
|
||
| TARGET_ENV="local" | ||
| TARGET_NAMESPACE="" | ||
|
|
||
| if [ "$TARGET_SELECTION" -ne 1 ]; then | ||
| TARGET_NAMESPACE=${PROJECT_MAP[$TARGET_SELECTION]} | ||
| if [ -z "$TARGET_NAMESPACE" ]; then | ||
| echo -e "${RED}Error: Invalid selection.${NC}" | ||
| exit 1 | ||
| fi | ||
| TARGET_ENV="openshift" | ||
| echo -e "Targeting OpenShift Namespace: ${GREEN}${TARGET_NAMESPACE}${NC}" | ||
| else | ||
| echo -e "Targeting: ${GREEN}Local Docker Environment${NC}" | ||
| fi | ||
|
|
||
| # 3. Prompt for Mine GUID | ||
| echo "" | ||
| echo "Enter the GUID of the Mine in the TARGET environment you want to attach this NoW to." | ||
| read -p "Mine GUID: " MINE_GUID | ||
| if [ -z "$MINE_GUID" ]; then | ||
| echo -e "${RED}Error: Mine GUID is required.${NC}" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # 4. Prompt for SOURCE Prod Pod | ||
| echo "" | ||
| echo "Select the SOURCE Production Database Pod (must be in 4c2ba9-prod)." | ||
| SOURCE_NAMESPACE="4c2ba9-prod" | ||
|
|
||
| # Auto-detect SOURCE pod in PROD namespace | ||
| MASTER_POD=$(find_primary_db_pod "$SOURCE_NAMESPACE") | ||
| PROD_POD=${MASTER_POD:-postgresql-prod-0} | ||
|
|
||
| echo "Detected Primary Pod Candidate in PROD: ${MASTER_POD:-(none)}" | ||
| echo "" | ||
| echo "Available Postgres Pods in PROD:" | ||
| oc get pods -n "$SOURCE_NAMESPACE" -L postgres-operator.crunchydata.com/role --no-headers 2>/dev/null | grep -E "postgres|crunchy" | grep "ha-" || echo " (none found)" | ||
| echo "" | ||
|
|
||
| read -p "Source DB Pod Name [$PROD_POD]: " USER_POD | ||
| PROD_POD=${USER_POD:-$PROD_POD} | ||
| if [ -z "$PROD_POD" ]; then | ||
| echo -e "${RED}Error: Source DB Pod not found or specified.${NC}" | ||
| exit 1 | ||
| fi | ||
|
|
||
|
|
||
| echo "" | ||
| echo -e "${YELLOW}Step 1: Generating SQL query from LOCAL COMPATIBLE backend...${NC}" | ||
| # We always generate the query using the LOCAL backend code to ensure compatibility with the logic we are running. | ||
| # NOTE: If we are targeting a remote env, we are sending data compatible with the LOCAL codebase. | ||
| # Ideally, the remote env code matches. | ||
| QUERY=$(docker compose exec -T backend flask generate-now-query --now-number "$NOW_NUMBER" | grep "SELECT 'now_application_guid=") | ||
|
|
||
| if [ -z "$QUERY" ]; then | ||
| echo -e "${RED}Error: Failed to generate SQL query.${NC}" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "$QUERY" > generated_query.sql | ||
| echo "Debug: SQL query saved to generated_query.sql" | ||
|
|
||
| echo -e "${YELLOW}Step 2: Fetching data from Prod and ingesting into Target...${NC}" | ||
|
|
||
| if [ "$TARGET_ENV" == "local" ]; then | ||
| # --- Execute for Local --- | ||
| (echo "SET search_path TO public;"; cat generated_query.sql) | \ | ||
| oc exec -n "$SOURCE_NAMESPACE" -i "$PROD_POD" -c database -- psql -d mds -At | \ | ||
| docker compose exec -T backend flask create-test-data --scenario now-application --mine-guid "$MINE_GUID" --non-interactive | ||
| else | ||
| # --- Execute for Remote OpenShift --- | ||
| echo "Locating backend pod in ${TARGET_NAMESPACE}..." | ||
| TARGET_POD_CANDIDATE=$(find_backend_pod "$TARGET_NAMESPACE") | ||
|
|
||
| echo "Detected Backend Pod Candidate: ${TARGET_POD_CANDIDATE:-(none)}" | ||
| echo "" | ||
| echo "Available Pods in Target Namespace (${TARGET_NAMESPACE}):" | ||
| echo "Available Pods in Target Namespace (${TARGET_NAMESPACE}):" | ||
| # List all pods to help user verify | ||
| oc get pods -n "$TARGET_NAMESPACE" --no-headers 2>/dev/null | grep "core-api" | grep -v "celery" || echo " (none found or access denied)" | ||
| echo "" | ||
|
|
||
| read -p "Target Backend Pod Name [${TARGET_POD_CANDIDATE}]: " USER_TARGET_POD | ||
| TARGET_POD=${USER_TARGET_POD:-$TARGET_POD_CANDIDATE} | ||
|
|
||
| if [ -z "$TARGET_POD" ]; then | ||
| echo -e "${RED}Error: Target Backend Pod not specified.${NC}" | ||
| rm -f generated_query.sql | ||
| exit 1 | ||
| fi | ||
| echo "Using target backend pod: $TARGET_POD" | ||
|
|
||
| echo "Ingesting data..." | ||
| (echo "SET search_path TO public;"; cat generated_query.sql) | \ | ||
| oc exec -n "$SOURCE_NAMESPACE" -i "$PROD_POD" -c database -- psql -d mds -At | \ | ||
| oc exec -n "$TARGET_NAMESPACE" -i "$TARGET_POD" -- flask create-test-data --scenario now-application --mine-guid "$MINE_GUID" --non-interactive | ||
| fi | ||
|
|
||
| rm -f generated_query.sql | ||
|
|
||
| echo "" | ||
| echo -e "${GREEN}Done!${NC}" | ||
| echo -e "${GREEN}Done!${NC}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was trying out a different IDE and thought I'd throw in these for convenience