Skip to content

Commit e0cafed

Browse files
committed
feat: R10 입력 고정과 평가 봉인을 분리
상황: 평가 입력 고정이 사람 검수와 평가자 준비를 요구하는 round seal에 결합돼 03 leaf가 자체 책임을 끝내지 못했다. 변경: current source와 허용된 기계 리포트 및 참조 산출물만 해시로 고정하는 --freeze-input 전이, 변조 검증, TODO 삭제 보호 계약을 추가했다. 영향: input-frozen은 사람 검수나 R10 통과를 주장하지 않으며 기존 --seal과 독립 report 게이트는 계속 모든 외부 조건을 요구한다. 검증: preflight 3/3, 관련 pytest 45개, docs gate, ruff, py_compile, git diff check를 통과했다.
1 parent 911f4e3 commit e0cafed

7 files changed

Lines changed: 441 additions & 26 deletions

File tree

docs/skills/ops/foundation/testing-and-gates.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -116,8 +116,8 @@ uv run python -X utf8 tests/run.py gate attempts
116116
| --- | --- | --- |
117117
| `docs` | fast | 운영 문서 포인터, gate 정의, CI 연결 상태를 확인한다. |
118118
| `root-clean` | fast | 저장소 루트가 canonical tree와 맞고 로컬 실습 파일, 로그, 임시 산출물이 남지 않았는지 확인한다. |
119-
| `evaluation-contract` | fast | clean implementation commit에서 목표 점수 없는 frozen rubric, closed raw report·canonical finding ledger schema, bundle exclusion과 raw report·scope·maturity·score/severity 보존 negative fixture를 검증한다. R10 bundle의 허용 목록 machine report와 직접 참조 JSON·이미지는 current source commit, 원본 hash, read-only entry로 묶고 stale·dirty scope는 seal을 차단한다. 독립 evaluator 배정과 R10 report 생성은 실행하지 않는다. |
120-
| `plan-quality` | fast | mainPlan이 미완료 TODO만 보존하는지, 현재 사실과 R10 draft가 일관적인지 검사한다. evaluator 배정·sealed scope·raw report 같은 외부 readiness blocker만 남은 draft는 통과하지만 blocker를 report에서 제거하지 않는다. |
119+
| `evaluation-contract` | fast | clean implementation commit에서 목표 점수 없는 frozen rubric, closed raw report·canonical finding ledger schema, bundle exclusion과 raw report·scope·maturity·score/severity 보존 negative fixture를 검증한다. R10 bundle의 허용 목록 machine report와 직접 참조 JSON·이미지는 current source commit, 원본 hash, read-only entry로 묶고 stale·dirty scope는 input freeze와 round seal을 모두 차단한다. 독립 evaluator 배정과 R10 report 생성은 실행하지 않는다. |
120+
| `plan-quality` | fast | mainPlan이 미완료 TODO만 보존하는지, 현재 사실과 R10 bundle이 일관적인지 검사한다. current source와 machine evidence만 고정한 `input-frozen` 상태는 evaluator 배정·사람 검수·raw report·round seal을 주장하지 않는다. 이 외부 readiness blocker가 명시돼 있으면 통과하지만 blocker를 report에서 제거하지 않는다. |
121121
| `r10-independent-review` | release | 서명 hash와 가용 기간이 있는 독립 evaluator roster, sealed scope, 세 raw report byte seal·결정적 Markdown view와 손실 없는 canonical finding ledger를 목표 점수 없이 엄격하게 검사한다. open P0·P1은 점수와 무관하게 `product-release`를 차단한다. |
122122
| `backend` | fast | Python backend 전체 테스트를 최대 1800초 안에서 실행한다. `tests/_attempts``--ignore`로 수집하지 않고, 0.25초 이상 걸린 항목 중 가장 느린 25개를 출력해 preflight 병목을 관찰한다. |
123123
| `attempts` | experiment | 운영과 분리된 `tests/_attempts` 실험 샌드박스를 실행한다. preflight/quality-cycle/CI 비포함이며 `tier` 스윕에도 끼지 않는다. |
@@ -256,4 +256,5 @@ uv run python -X utf8 tests/run.py gate attempts
256256
- `landing-build`는 공개 문서 surface가 generated docs 본문 HTML을 nav chunk에 싣지 않는지도 확인한다. `docsNav.js`는 metadata와 `contentModule`만 담고, 각 문서 본문 module은 화면과 prerender가 소비하는 `html`만 가진 채 `landing/src/lib/generated/docsPages/page*.js`로 분리되어 slug route에서 동적 로딩되어야 한다. 검색 원문 `text``searchIndex.js`가 소유하므로 페이지 module에 중복하지 않는다. `docs/skills` 핵심 SSOT 문구가 generated docs에 반영되지 않은 stale 상태도 실패로 본다.
257257
- 기존 부채를 새 테스트로 한 번에 해결하지 못하면 별도 baseline 또는 명시적 TODO 문서로 분리한다.
258258
- CI YAML은 세부 명령을 소유하지 않고 `tests/run.py gate <name>`만 호출한다.
259-
- `evaluation-contract`은 rubric·raw report schema·canonical finding ledger schema·negative fixture 자체만 봉인한다. sealed bundle, evaluator roster, 실제 raw report와 finding ledger는 `08-r10-independent-review`가 소유하며 이 downstream 산출물을 계약 packet의 완료 조건으로 되돌리지 않는다.
259+
- `buildPrdEvaluationBundle.py --freeze-input`은 clean source commit, current machine report 9개, 직접 참조 artifact, rubric과 exclusion manifest만 `input-frozen`으로 고정한다. archive와 scope hash가 바뀌면 `--check``plan-quality`가 실패한다. 이 전이는 `sealed: false`, `roundState: blocked`, `roundSealEligible: false`를 유지하므로 evaluator 배정, 사람 검수, 독립 report나 R10 통과를 대신하지 않는다. 이후 모든 remediation과 evaluator 조건이 갖춰진 경우에만 별도 `--seal`이 round scope를 봉인한다.
260+
- `evaluation-contract`은 rubric·raw report schema·canonical finding ledger schema·negative fixture 자체만 봉인한다. round seal, evaluator roster, 실제 raw report와 finding ledger는 `08-r10-independent-review`가 소유하며 이 downstream 산출물을 입력 고정 packet의 완료 조건으로 되돌리지 않는다.

docs/skills/ops/tools/buildPrdEvaluationBundle.py

Lines changed: 178 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -790,6 +790,34 @@ def remediationSealBlockers(inputManifest: dict[str, Any]) -> list[str]:
790790
return blockers
791791

792792

793+
def rubricFreezeBlockers(inputManifest: dict[str, Any], rubricBytes: bytes) -> list[str]:
794+
rubric = inputManifest.get("rubric")
795+
if (
796+
not isinstance(rubric, dict)
797+
or rubric.get("sha256") != sha256Bytes(rubricBytes)
798+
or rubric.get("targetScore") is not None
799+
or rubric.get("passThreshold") is not None
800+
):
801+
return ["frozen rubric metadata is invalid"]
802+
return []
803+
804+
805+
def inputFreezeReadiness(
806+
inputManifest: dict[str, Any],
807+
rubricBytes: bytes,
808+
machineEvidenceBlockers: Iterable[str] = (),
809+
) -> dict[str, Any]:
810+
freezeBlockers = sorted(
811+
set(list(machineEvidenceBlockers) + rubricFreezeBlockers(inputManifest, rubricBytes))
812+
)
813+
return {
814+
"freezeEligible": not freezeBlockers,
815+
"inputFrozen": False,
816+
"freezeBlockingReasons": freezeBlockers,
817+
"blockingReasons": list(freezeBlockers),
818+
}
819+
820+
793821
def roundReadiness(
794822
inputManifest: dict[str, Any],
795823
roster: dict[str, Any],
@@ -800,15 +828,8 @@ def roundReadiness(
800828
remediationSealBlockers(inputManifest)
801829
+ verifyEvaluatorRoster(roster)
802830
+ list(machineEvidenceBlockers)
831+
+ rubricFreezeBlockers(inputManifest, rubricBytes)
803832
)
804-
rubric = inputManifest.get("rubric")
805-
if (
806-
not isinstance(rubric, dict)
807-
or rubric.get("sha256") != sha256Bytes(rubricBytes)
808-
or rubric.get("targetScore") is not None
809-
or rubric.get("passThreshold") is not None
810-
):
811-
sealBlockers.append("frozen rubric metadata is invalid")
812833
blockers = list(sealBlockers)
813834
if inputManifest.get("sealed") is not True or inputManifest.get("roundState") != "ready":
814835
blockers.append("R10 input manifest is not sealed and ready")
@@ -839,6 +860,32 @@ def buildEvaluationScopeManifest(entries: tuple[BundleEntry, ...], *, gitHead: s
839860
}
840861

841862

863+
def inputFreezeMatches(
864+
binding: Any,
865+
scope: dict[str, Any],
866+
archive: dict[str, Any],
867+
machineEvidence: dict[str, Any],
868+
) -> bool:
869+
return (
870+
isinstance(binding, dict)
871+
and binding.get("state") == "frozen"
872+
and binding.get("manifestPath") == relativePath(MANIFEST_PATH)
873+
and binding.get("archivePath") == archive["path"]
874+
and binding.get("gitCommit") == scope["gitCommit"]
875+
and binding.get("dirtyDiffHash") == scope["dirtyDiffHash"]
876+
and binding.get("manifestHash") == scope["manifestHash"]
877+
and binding.get("evaluationBundleHash") == archive["sha256"]
878+
and binding.get("fileCount") == scope["fileCount"]
879+
and binding.get("machineEvidenceIndexPath") == EVIDENCE_INDEX_PATH
880+
and binding.get("requiredReportCount") == machineEvidence["requiredReportCount"]
881+
and binding.get("includedReportCount") == machineEvidence["includedReportCount"]
882+
and binding.get("artifactCount") == machineEvidence["artifactCount"]
883+
and binding.get("allCurrent") is True
884+
and machineEvidence["allCurrent"] is True
885+
and machineEvidence["blockingReasons"] == []
886+
)
887+
888+
842889
def buildPrdEvaluationBundle() -> tuple[dict[str, Any], bytes]:
843890
gitHead = currentGitHead()
844891
scopeCommit = latestIncludedScopeCommit(gitHead)
@@ -861,9 +908,23 @@ def buildPrdEvaluationBundle() -> tuple[dict[str, Any], bytes]:
861908
beforeDiffHash = dirtyDiffHash(entries, beforeStatuses)
862909
scope = buildEvaluationScopeManifest(entries, gitHead=scopeCommit, diffHash=beforeDiffHash)
863910
archiveBytes = buildZipBytes(entries)
911+
archive = {
912+
"path": relativePath(ARCHIVE_PATH),
913+
"sha256": sha256Bytes(archiveBytes),
914+
"bytes": len(archiveBytes),
915+
"format": "zip",
916+
"compression": ZIP_COMPRESSION_NAME,
917+
"compressionMethod": ZIP_COMPRESSION,
918+
"readOnlyEntries": True,
919+
}
864920
if currentGitHead() != gitHead or changedScopeStatuses() != beforeStatuses:
865921
raise BundleError("evaluation scope changed while the bundle was being built")
866922
inputManifest = loadMapping(INPUT_PATH)
923+
inputReadiness = inputFreezeReadiness(
924+
inputManifest,
925+
sourceBytes[relativePath(RUBRIC_SOURCE)],
926+
machineEvidence["blockingReasons"],
927+
)
867928
readiness = roundReadiness(
868929
inputManifest,
869930
loadMapping(ROSTER_PATH),
@@ -879,29 +940,52 @@ def buildPrdEvaluationBundle() -> tuple[dict[str, Any], bytes]:
879940
and inputScope.get("gitCommit") == scope["gitCommit"]
880941
and inputScope.get("dirtyDiffHash") == scope["dirtyDiffHash"]
881942
and inputScope.get("manifestHash") == scope["manifestHash"]
882-
and inputScope.get("evaluationBundleHash") == sha256Bytes(archiveBytes)
943+
and inputScope.get("evaluationBundleHash") == archive["sha256"]
883944
)
945+
frozenBinding = inputManifest.get("inputFreeze")
946+
frozenBindingPresent = isinstance(frozenBinding, dict) and frozenBinding.get("state") == "frozen"
947+
frozenInputMatches = inputFreezeMatches(frozenBinding, scope, archive, machineEvidence)
948+
inputReadiness["inputFrozen"] = inputReadiness["freezeEligible"] and frozenInputMatches
949+
if frozenBindingPresent and not frozenInputMatches:
950+
inputReadiness["blockingReasons"] = sorted(
951+
set(inputReadiness["blockingReasons"] + ["frozen input scope does not match the current bundle"])
952+
)
884953
if inputManifest.get("sealed") is True and not sealedScopeMatches:
885954
readiness["blockingReasons"] = sorted(
886955
set(readiness["blockingReasons"] + ["sealed input scope does not match the current bundle"])
887956
)
957+
if not inputReadiness["inputFrozen"] and not sealedScopeMatches:
958+
readiness["sealEligible"] = False
959+
readiness["sealBlockingReasons"] = sorted(
960+
set(readiness["sealBlockingReasons"] + ["R10 input bundle is not frozen"])
961+
)
962+
readiness["blockingReasons"] = sorted(
963+
set(readiness["blockingReasons"] + ["R10 input bundle is not frozen"])
964+
)
888965
readiness["roundReady"] = readiness["sealEligible"] and sealedScopeMatches
889-
scope["sealState"] = "sealed" if sealedScopeMatches else "draft"
890-
state = "sealed" if sealedScopeMatches else "ready-to-seal" if readiness["sealEligible"] else "draft"
966+
scope["sealState"] = (
967+
"sealed"
968+
if sealedScopeMatches
969+
else "input-frozen"
970+
if inputReadiness["inputFrozen"]
971+
else "draft"
972+
)
973+
state = (
974+
"sealed"
975+
if sealedScopeMatches
976+
else "input-frozen"
977+
if inputReadiness["inputFrozen"]
978+
else "ready-to-seal"
979+
if readiness["sealEligible"]
980+
else "draft"
981+
)
891982
manifest = {
892983
"schemaVersion": 1,
893984
"roundId": "R10",
894985
"state": state,
895986
"scope": {key: value for key, value in scope.items() if key != "files"},
896-
"archive": {
897-
"path": relativePath(ARCHIVE_PATH),
898-
"sha256": sha256Bytes(archiveBytes),
899-
"bytes": len(archiveBytes),
900-
"format": "zip",
901-
"compression": ZIP_COMPRESSION_NAME,
902-
"compressionMethod": ZIP_COMPRESSION,
903-
"readOnlyEntries": True,
904-
},
987+
"archive": archive,
988+
"inputReadiness": inputReadiness,
905989
"roundReadiness": readiness,
906990
"machineEvidence": {
907991
"indexPath": EVIDENCE_INDEX_PATH,
@@ -1026,6 +1110,66 @@ def draftInputManifest(inputManifest: dict[str, Any], bundleManifest: dict[str,
10261110
return updated
10271111

10281112

1113+
def frozenInputManifest(inputManifest: dict[str, Any], bundleManifest: dict[str, Any]) -> dict[str, Any]:
1114+
inputReadiness = bundleManifest.get("inputReadiness")
1115+
if not isinstance(inputReadiness, dict) or inputReadiness.get("freezeEligible") is not True:
1116+
reasons = (
1117+
inputReadiness.get("freezeBlockingReasons", [])
1118+
if isinstance(inputReadiness, dict)
1119+
else ["input freeze readiness is absent"]
1120+
)
1121+
raise BundleError("R10 input cannot be frozen: " + "; ".join(reasons))
1122+
scope = bundleManifest["scope"]
1123+
archive = bundleManifest["archive"]
1124+
machineEvidence = bundleManifest["machineEvidence"]
1125+
updated = draftInputManifest(inputManifest, bundleManifest)
1126+
remainingSealBlockers = [
1127+
blocker
1128+
for blocker in bundleManifest["roundReadiness"]["sealBlockingReasons"]
1129+
if blocker != "R10 input bundle is not frozen"
1130+
]
1131+
updated["draftBundle"]["sealEligible"] = not remainingSealBlockers
1132+
updated["inputFreeze"] = {
1133+
"state": "frozen",
1134+
"manifestPath": relativePath(MANIFEST_PATH),
1135+
"archivePath": archive["path"],
1136+
"gitCommit": scope["gitCommit"],
1137+
"dirtyDiffHash": scope["dirtyDiffHash"],
1138+
"manifestHash": scope["manifestHash"],
1139+
"evaluationBundleHash": archive["sha256"],
1140+
"fileCount": scope["fileCount"],
1141+
"machineEvidenceIndexPath": machineEvidence["indexPath"],
1142+
"requiredReportCount": machineEvidence["requiredReportCount"],
1143+
"includedReportCount": machineEvidence["includedReportCount"],
1144+
"artifactCount": machineEvidence["artifactCount"],
1145+
"allCurrent": machineEvidence["allCurrent"],
1146+
}
1147+
return updated
1148+
1149+
1150+
def freezeInputBundle(manifest: dict[str, Any], archiveBytes: bytes) -> dict[str, Any]:
1151+
inputManifest = loadMapping(INPUT_PATH)
1152+
updatedInput = frozenInputManifest(inputManifest, manifest)
1153+
originalInputBytes = INPUT_PATH.read_bytes()
1154+
updatedInputBytes = yaml.safe_dump(updatedInput, allow_unicode=True, sort_keys=False, width=120).encode("utf-8")
1155+
try:
1156+
writeAtomic(INPUT_PATH, updatedInputBytes)
1157+
frozenManifest, frozenArchiveBytes = buildPrdEvaluationBundle()
1158+
if (
1159+
frozenManifest["state"] != "input-frozen"
1160+
or frozenManifest["inputReadiness"]["inputFrozen"] is not True
1161+
):
1162+
raise BundleError("R10 input was updated but the current bundle did not reach an input-frozen state")
1163+
if frozenArchiveBytes != archiveBytes:
1164+
raise BundleError("evaluation archive changed during the input freeze transition")
1165+
except Exception:
1166+
writeAtomic(INPUT_PATH, originalInputBytes)
1167+
raise
1168+
writeAtomic(ARCHIVE_PATH, frozenArchiveBytes, readOnly=True)
1169+
writeAtomic(MANIFEST_PATH, dumpManifest(frozenManifest))
1170+
return frozenManifest
1171+
1172+
10291173
def sealBundle(manifest: dict[str, Any], archiveBytes: bytes) -> dict[str, Any]:
10301174
inputManifest = loadMapping(INPUT_PATH)
10311175
updatedInput = sealedInputManifest(inputManifest, manifest)
@@ -1051,6 +1195,11 @@ def buildParser() -> argparse.ArgumentParser:
10511195
action = parser.add_mutually_exclusive_group(required=True)
10521196
action.add_argument("--write", action="store_true", help="Write the tracked manifest and generated ZIP archive.")
10531197
action.add_argument("--check", action="store_true", help="Verify the tracked manifest against the current scope.")
1198+
action.add_argument(
1199+
"--freeze-input",
1200+
action="store_true",
1201+
help="Freeze current source and machine evidence without claiming evaluator or round readiness.",
1202+
)
10541203
action.add_argument("--seal", action="store_true", help="Seal only after remediation and evaluator evidence is ready.")
10551204
return parser
10561205

@@ -1065,6 +1214,12 @@ def main(argv: list[str] | None = None) -> int:
10651214
f"sealed R10 bundle: {sealed['scope']['fileCount']} files, "
10661215
f"sha256={sealed['archive']['sha256']}"
10671216
)
1217+
elif args.freeze_input:
1218+
frozen = freezeInputBundle(manifest, archiveBytes)
1219+
print(
1220+
f"froze R10 input bundle: {frozen['scope']['fileCount']} files, "
1221+
f"sha256={frozen['archive']['sha256']}"
1222+
)
10681223
elif args.write:
10691224
inputManifest = loadMapping(INPUT_PATH)
10701225
updatedInput = draftInputManifest(inputManifest, manifest)
@@ -1074,14 +1229,16 @@ def main(argv: list[str] | None = None) -> int:
10741229
)
10751230
writeAtomic(ARCHIVE_PATH, archiveBytes, readOnly=True)
10761231
writeAtomic(MANIFEST_PATH, dumpManifest(manifest))
1232+
state = "input-frozen" if manifest["inputReadiness"]["inputFrozen"] else "draft"
10771233
print(
1078-
f"wrote draft R10 bundle: {manifest['scope']['fileCount']} files, "
1234+
f"wrote {state} R10 bundle: {manifest['scope']['fileCount']} files, "
10791235
f"sha256={manifest['archive']['sha256']}"
10801236
)
10811237
else:
10821238
verifyWrittenManifest(manifest)
10831239
print(
10841240
f"ok: R10 bundle manifest matches {manifest['scope']['fileCount']} files; "
1241+
f"inputFrozen={str(manifest['inputReadiness']['inputFrozen']).lower()}, "
10851242
f"sealEligible={str(manifest['roundReadiness']['sealEligible']).lower()}"
10861243
)
10871244
except BundleError as exc:

docs/skills/ops/tools/buildPrdRoundFactAudit.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -304,6 +304,8 @@ def buildPrdRoundFactAudit() -> dict[str, Any]:
304304
"auditComplete": True,
305305
"scoreThresholdApplied": False,
306306
"productCompletionEligible": False,
307+
"inputFreezeEligible": actualManifest["inputReadiness"]["freezeEligible"],
308+
"inputFrozen": actualManifest["inputReadiness"]["inputFrozen"],
307309
"roundSealEligible": actualManifest["roundReadiness"]["sealEligible"],
308310
"scope": {
309311
"gitCommit": scope["gitCommit"],
@@ -366,7 +368,7 @@ def main(argv: list[str] | None = None) -> int:
366368
"R10 fact audit is stale; run "
367369
"uv run --no-sync python -X utf8 docs/skills/ops/tools/buildPrdRoundFactAudit.py --write"
368370
)
369-
print("ok: R10 fact audit matches the current draft bundle")
371+
print("ok: R10 fact audit matches the current evaluation bundle")
370372
except FactAuditError as exc:
371373
print(f"FAIL: {exc}", file=sys.stderr)
372374
return 1

0 commit comments

Comments
 (0)