Skip to content

[WIP DO NOT MERGE] Implement cluster registration workflows via Shepherd/golang #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 45 commits into
base: main
Choose a base branch
from
Open

[WIP DO NOT MERGE] Implement cluster registration workflows via Shepherd/golang #56

wants to merge 45 commits into from

Conversation

@git-ival
Copy link
Contributor

@git-ival git-ival commented May 13, 2025 •
edited
Loading

Features:

  • "Natural" sorting of slices based on some string contained in each element
    • Currently used for imported downstream_cluster_templates to trigger import cluster registration workflow in-order based on name
  • Batched, sequential and concurrent cluster registration
    • Imported Clusters: fully functional
    • Provisioned Clusters: implemented, not yet tested
    • Custom Clusters: fully functional
  • Idempotency for all cluster registrations via custom state handling logic within dartboard
    • Uses the newly-introduced ClusterStatus struct that contains basic state information for each Cluster that has been sent to the cluster registration batch runner

Closes #72

@git-ival git-ival changed the title [WIP DO NOT MERGE] Implement cluster regirstration workflows via Shepherd/golang [WIP DO NOT MERGE] Implement cluster registration workflows via Shepherd/golang May 20, 2025
@git-ival git-ival mentioned this pull request May 28, 2025
Open
2 tasks
@git-ival git-ival force-pushed the implement-provisioned-clusters branch from edeec2f to 5f1d44e Compare May 31, 2025 19:29
moio
moio reviewed Jun 6, 2025
View reviewed changes
moio
moio reviewed Jun 6, 2025
View reviewed changes
moio
moio reviewed Jun 6, 2025
View reviewed changes
@git-ival git-ival force-pushed the implement-provisioned-clusters branch 2 times, most recently from ae1bc50 to 31c8bc0 Compare June 12, 2025 20:34
git-ival added 22 commits July 8, 2025 18:32
@git-ival
implement Imported Cluster creation via golang, add Cluster state han…
9cc70db 
…dling logic within dartboard, add kubeconfig parsing support, add shepherd as dependency
@git-ival
WIP support provisioning clusters via dartboard, WIP support for usin…
146a8cb 
…g Harvester VMs to provision clusters
@git-ival
update state handling to utilize a map[string]*ClusterStatus to avoid…
241bc71 
… pointer issues on append, add concurrency handling to speed up batch processing, add yaml struct tags
@git-ival
add missing wait.go file handling backoff logic
a39a707
@git-ival
change post-batch sleep logic, update some log messages
44ccb68
@git-ival
move update channels to separate struct that can be passed as needed,…
72d4fe0 
… add sequenced updates per-batch
@git-ival
add TODO for destroy command
2c3f33a
@git-ival
Moved batch Cluster registration logic into a separate file, added a …
0406a1d 
…SequencedBatchRunner struct to handle all the logic
@git-ival
reduce size of ClusterStatus struct, genericize batchrunner logic to …
91938d1 
…enable handling of both imported and provisioned cluster flows
@git-ival
support 'naturally' sorting slices by string, cleanup parameters and …
78d9c9a 
…unused code
@git-ival
replace pod status function from shepherd with custom action
3b88e39
@git-ival
replace pod status function from shepherd with custom action
7c5188b
@git-ival
WIP custom cluster registration flow
dc22bf5
@git-ival
update shepherd dep
851686e
@git-ival
fix clusters.go assert
d81ea83
@git-ival
add import nicknames to fix golangci-lint
82d25b6
@git-ival
fix various golangci-lint errors and warnings
136cd75
@git-ival
add new stages, various func updates
9baf89e
@git-ival
WIP, updates to various interal/dart and internal/tofu types
71156b4
@git-ival
updates to harvester modules, add node_templates
277ad90
@git-ival
add missing dart types
5f176dc
@git-ival
update tofu format parsing
1f4f208
git-ival added 22 commits July 8, 2025 18:32
@git-ival
WIP massive overhauls to dart types, added in internal/harvester for …
711d027 
…creating/operating on harvester VMs, misc
@git-ival
update deps
6a2cbc9
@git-ival
WIP modifications to type constraints, VM handling logic
6be2594
@git-ival
update test_env to support node-only creation for custom clusters
c0214b4
@git-ival
update test_env to support node-only creation for custom clusters
6ae5fc7
@git-ival
WIP update types, custom cluster prov support
063dcdd
@git-ival
update deps
d1c46c9
@git-ival
WIP custom cluster flow, SO CLOSE
9aceac4
@git-ival
fix get cluster
3525d6d
@git-ival
cleanup some comments
e3fce0c
@git-ival
ignore tester cluster deploys if no tester, fix custom cluster batchi…
ee6b35c 
…ng logic, start using logrus instead of fmt for debug logs
@git-ival
update image dataresource logic, do not use datasource if image_id is…
c0386e6 
… passed in
@git-ival
remove lots of unused stuff
c3bcea3
@git-ival
ensure tunnels are not created by default, added bool flag to control…
8dfc180 
… this behavior
@git-ival
removed unused 'node_templates' var
74b5a8f
@git-ival
update upstream and tester clusters with new create_tunnels input
9232302
@git-ival
delay Stage.Create update until after cluster is found post-create ca…
24d9043 
…ll, add tofu.Output() func
@git-ival
add new skip-charts flag for so that repeated can bypass chart reinst…
a415bca 
…alls
@git-ival
replace shepherd deps with upstream
8b5f2b3
@git-ival
fix upstream cluster URL output when tunnels are not in use
2f0f796
@git-ival
add skip-refresh dartboard argument for tofu.Apply()
396eb4c
@git-ival
replace image_id with node_module_variables field
a9f959f
@git-ival git-ival force-pushed the implement-provisioned-clusters branch from 31c8bc0 to a9f959f Compare July 8, 2025 18:33
@git-ival git-ival requested review from MSpencer87 and moio July 9, 2025 17:36
@git-ival git-ival force-pushed the implement-provisioned-clusters branch from d7d54b0 to 053f2d7 Compare July 9, 2025 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MSpencer87 MSpencer87 Awaiting requested review from MSpencer87

@moio moio Awaiting requested review from moio

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Refactor cluster registration workflow

2 participants

@git-ival @moio