feat(ansible): journald永続化をcontrol-planeに追加 [BOXP-127] - #11562
Merged
Conversation
…very Implements a GitHub Actions workflow that downloads an Orange Pi image from S3 and writes it to an SD card writer connected to a worker node. Adds S3 read permission to the Ansible Apply IAM role to support image download without needing a separate role. Safety guards: removable device check, mount state check, checksum verification, dry-run mode, and explicit device path requirement. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add shellcheck disable=SC2029 for intentional client-side variable expansion in SSH commands
- Group consecutive GITHUB_STEP_SUMMARY redirects with { } >> file (SC2129)
- Add `set -o pipefail` to the write step so xzcat failure propagates correctly - Track write and verify completion with IMAGE_WRITTEN/WRITE_VERIFIED flags - Write summary now only reports success when both write and verify succeeded Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The build action uploaded the timestamp-named image checksum but omitted a corresponding checksum for latest.img.xz, causing write-sdcard-image.yml to always skip verification. Now creates latest.img.xz.sha256 with the correct filename reference before uploading.
…ailable Replace `|| true` silent fallback with a mandatory checksum download. If `latest.img.xz.sha256` cannot be fetched from S3, the workflow now exits with an error rather than writing an unverified image to the SD card. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace `mount | grep '^$DEVICE'` with `lsblk -no MOUNTPOINT` to detect partitions mounted via UUID/label/by-* paths that string matching would miss, preventing accidental writes to in-use SD cards - Add workflow-level concurrency group keyed by worker_node+device_path to prevent two simultaneous runs from both passing safety checks and running dd concurrently Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Verify write step now checks that lsblk reports at least one partition with TYPE=part, failing if none are found. Previously, partprobe failures were silently ignored and lsblk was called without checking its output content — a blank or corrupt card would still set WRITE_VERIFIED=true since lsblk always shows the disk itself regardless of partition state. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… verify partitions - Remove DEVICE_PATH=... prefix before ssh which does not forward env vars to remote; expand $DEVICE_PATH locally in double-quoted command strings instead (format already validated as /dev/sd[a-z] or /dev/mmcblk[0-9]+, so local expansion is safe) - Replace unconditional WRITE_VERIFIED=true with explicit partition count check using wc -l (always exits 0), avoiding grep -c returning "0\n0" on no matches and ensuring the step fails when no TYPE=part partitions are found on the written device Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SD カード障害（INC-2/INC-4/BOXP-127）発生時に障害直前のカーネル・ストレージ エラーログが揮発的に失われる問題を解消するため、全 control-plane ノードに journald 永続ストレージを設定する Ansible タスクを追加する。 - /var/log/journal ディレクトリ作成 + Storage=persistent 設定 - SystemMaxUse=200M / MaxRetentionSec=1month で SD カード消費を抑制 - journald_persistent_storage フラグでオプトアウト可能（デフォルト有効） - Restart systemd-journald ハンドラを追加 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lve conflicts) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
- defaults/main.yml: journald_persistent_storage デフォルトをfalseに変更 (kubernetes_componentsはworker-image.ymlでも使用されるためworkerへの影響を排除) - control-plane.yml: journald_persistent_storage: trueをcontrol-plane専用で明示設定 - tasks/journald.yml: false時にStorage=volatileドロップインを配置し/var/log/journalを削除 (既存設定が残存して永続化を解除できない問題を修正) - molecule/default/converge.yml: journald_persistent_storage: trueを設定してテストパスを有効化 - molecule/default/verify.yml: journaldディレクトリ・ドロップイン内容・Storage=persistentを検証するテストを追加 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Owner
Author
codex-reviewの指摘事項修正 (BOXP-127 retry)前回のレビューで指摘された3点をすべて修正しました。 P1修正: journald設定をcontrol-plane限定に変更
P2修正: false時の収束処理を追加
P2修正: Moleculeテスト追加
|
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
converge.ymlではtrueに設定しているが、verifyプレイブックはロールのdefaultsを 自動継承しないためwhen条件式で未定義エラーが発生していた。 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
デフォルトはfalseで、control-planeプレイブック側でのみtrueに設定される。 worker nodeには適用されないことを明記。 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…le scenario - Add `notify: Restart systemd-journald` to the persistent journal directory creation task so journald picks up the drop-in even when only the directory is recreated (drop-in unchanged → no change event without this notify). - Add molecule/journald-disabled scenario that converges with journald_persistent_storage=false and asserts the drop-in is absent, covering the code path that was previously untested. - Run `molecule test --all` in CI so both default and journald-disabled scenarios are exercised on every ansible role change. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
Remove drop-in on disable was insufficient: if /var/log/journal already exists, journald Storage=auto continues persistent logging. Write Storage=volatile explicitly so the role always converges correctly. Update Molecule journald-disabled verify to assert Storage=volatile is present in the drop-in rather than checking the file is absent.
This comment has been minimized.
This comment has been minimized.
When journald_persistent_storage=false, remove the role-managed drop-in (state: absent) instead of writing Storage=volatile. This aligns with the plan.md safety policy: only remove role-owned config, never overwrite external journald configuration. Update Molecule journald-disabled verify to assert drop-in is absent. Document in plan.md that Storage=auto delegation is intentional: new nodes without /var/log/journal default to volatile. Hosts that previously had persistence can use journald_purge_persistent_logs=true to fully revert. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…n-destructive disabled path - default/verify.yml: assert SystemMaxUse=200M and MaxRetentionSec=1month in the journald drop-in (in addition to Storage=persistent) - journald-disabled/prepare.yml: pre-create /var/log/journal with a sentinel file to simulate an existing persistent journal setup - journald-disabled/verify.yml: assert /var/log/journal and sentinel file are preserved after role runs with journald_persistent_storage=false Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…urge paths - journald.yml: when disabled, remove role-managed drop-in (state: absent) instead of creating a Storage=volatile file; Storage=auto takes effect, which is correct for nodes with no /var/log/journal directory - journald-disabled/prepare.yml: pre-create role-managed drop-in to prove the converge step deletes it (deletion is now load-bearing in the test) - journald-disabled/verify.yml: assert drop-in is absent after converge - default/verify.yml: add SystemMaxUse=200M and MaxRetentionSec=1month assertions alongside the existing Storage=persistent check - molecule/journald-purge/: new scenario that pre-creates /var/log/journal with a sentinel file and the drop-in, then runs the role with journald_purge_persistent_logs=true and verifies both are deleted Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
purge タスクで /var/log/journal を削除しても journald に再起動通知がなければ、 稼働中の journald が以前のストレージ設定を保持し続けてしまう問題を修正。 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
Resolve conflicts: keep origin/main version of write-sdcard-image.yml and update test-ansible.yml to use matrix approach with molecule test --all.
Contributor
Ansible Plan ResultsMode: golyat-4: worker-image
2 changed Changed Tasks (2)
shanghai-1: control-plane
4 changed Changed Tasks (4)
shanghai-1: node-shanghai-1
No changes shanghai-2: control-plane
4 changed Changed Tasks (4)
shanghai-2: node-shanghai-2
No changes shanghai-3: control-plane
4 changed Changed Tasks (4)
shanghai-3: node-shanghai-3
No changes Plan executed on all nodes in parallel. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kubernetes_componentsAnsible ロールに journald 永続ストレージ設定を追加変更内容
tasks/journald.yml:/var/log/journal作成 +Storage=persistent+ 容量上限設定（新規）tasks/main.yml: journald タスクを includedefaults/main.yml:journald_persistent_storage: true,SystemMaxUse=200M,MaxRetentionSec=1monthhandlers/main.yml:Restart systemd-journaldハンドラ追加docs/project_docs/BOXP-127/plan.md: 恒久対策計画（新規）背景
OrangePI Zero3 の control-plane（shanghai-1/2/3）が繰り返し SD カード障害で停止している（INC-2, INC-4, BOXP-127）。journald が volatile のため、障害後に OS 再起動すると直前のエラーログが消える。SD カードが読み取り可能な状態でリブートした場合でも証拠が残らない状態を改善する。
容量:
SystemMaxUse=200Mで SD カードの過度な消費を防止。Test plan
cd ansible && uv run ansible-playbook --syntax-check playbooks/control-plane.ymlcd ansible && uv run ansible-lint roles/kubernetes_componentsjournalctl --disk-usageで永続ログ確認ls /var/log/journal/でディレクトリ存在確認🤖 Generated with Claude Code