Skip to content

image.builder.pipeline — project history

Frozen archive — retired 2026-09-10

This is CHANGELOG.md from ericcames/image.builder.pipeline as it stood when the per-PR changelog obligation was retired (image.builder.pipeline#119). Nothing is appended to it again.

Bare issue numbers refer to ericcames/image.builder.pipeline. They do not autolink on this site — read (#112) as https://github.com/ericcames/image.builder.pipeline/issues/112. References to sales.demos are written out in full.

Entries are verbatim and newest first. For anything after 2026-09-10, see where things live now.

Fixed

  • The Windows row said "built and published" with a caveat that the tag's hardening had not been re-verified. It had been, and thoroughly (#112). win2k22-cis-l1-golden:20260908-1853 measures 10 of 10 controls impossible on a clean install -- off the media and off the booted guest's own disk -- and 27 of 27 (100%) across the full set on a clone confirmed rebuilt from the DataSource serving that image (sales.demos#358, #382). The row now says Complete.
  • The caveat was written from the artifact and the label, and missed the record. skopeo inspect and a green consumer scan genuinely prove nothing, which is what I checked; the verification lives in sales.demos' CHANGELOG and in playbooks/scripts/verify_cis_disk.py, which runs inside the publish and refuses to apply an L1 label the disk does not support. The label is a gate output, not an assertion -- that is what changed after #91, and it is the part the previous wording missed.
  • The two genuine non-evidence points are kept, because both still look like proof: a green Windows Day 1 - 4 Compliance Scan (the consumer's windows_compliance_fail_on_noncompliant defaults to false, so it reports rather than gates) and a bare cis.level=L1 label, which is exactly what #91 was.
  • Also records that sysprep /generalize strips nothing, measured off the sysprepped guest's disk.

Changed

  • The platform table said Windows Server 2022 was "Phase 3", and the tag it points at has been in production use for days (#112). Every task that row depends on is Done in ROADMAP.md — unattended install, ISO remaster (#40), CIS L1 hardening over WinRM with 44 controls, sysprep and publish — and sales.demos consumes the published tag. "Phase 3" understated it.
  • It does not say "Complete" either, and that is the point. The RHEL 9 row quotes a score because one exists (OpenSCAP 98.07 against a 95 gate). Windows has no equivalent yet, so the row says built and published and links to a short section explaining exactly what is and is not evidenced.
  • Three things that look like proof and are not, now written down so the question is not re-litigated:
    • The cis.level=L1 label. That is the label #91 was about: it was present on 20260907-0516 while the disk underneath was the unhardened Sep 5 build, and the guest scored 9 of 27.
    • A green Windows Day 1 - 4 Compliance Scan. The consumer's windows_compliance_fail_on_noncompliant defaults to false — the scan is a report, not a gate, deliberately, so a red node never appears in front of a customer. Job 591 succeeded on 2026-09-09 and would have succeeded at 9 of 27 too.
    • The layer size. 5,438,513,415 bytes is a compressed layer; the known-bad artifact was a 9,307,619,328-byte uncompressed qcow2. Not comparable either way.
  • What is actually known, and is now in the README: the current tag was built sixty-one minutes after the #91 fix landed (dd8116b, 2026-09-08 17:52:31 UTC; image created 18:53:28 UTC), so it is the first publish with the stale-creates: path removed. The section names the one command that would turn "published" into "verified".
  • ## Quay.io private repo entitlement was a troubleshooting write-up for a resolved problem — the free plan's zero private-repo allowance, a screenshot of the Quay warning banner, and instructions to open a support case. The Unlimited Repositories subscription is active to 2027-08-14. It is now a two-row table saying which image is public, which is private, and why; the lapse runbook and the screenshot moved to docs/operations.md rather than being deleted.
  • sales.demos#3 was described as "shipped and waiting on a tag". It has the tag and consumes it.
  • ### Working across both repos was ~30 lines near-duplicated in sales.demos' README. Cut to the part specific to this repo — that its skills are not reachable from a session started elsewhere — with a link to the canonical copy. Two copies with nothing keeping them in step had already begun to drift, which is the same problem ericcames/sales.demos-docs#12 was opened for.

Changed

  • Getting started opened with AWS prerequisites, but most readers only consume what this repo publishes (#114). The README already says four lines above it that this repo is the producer and the dependency runs outward — Getting started did not act on that. It now opens with three doors: consuming the images (you do not need this repo — go to sales.demos), building or publishing one (the previous content, pipeline commands unchanged), and working across both repos. Mirrors the shape applied in sales.demos#418.
  • The cross-repo working shape is now stated, and it points away from here. A session spanning both repos must start in sales.demos, because its .mcp.json is project-scoped and this repo has no MCP servers at all. From there you can cd here and run these playbooks, since the working directory does not restrict shell access — better in one direction only. The new ### Working across both repos subsection also says plainly that this repo's skills are not reachable from that session: first-time, dev-workflow, rhel9-containerdisk and windows-image-build are discovered only from a session started here, so /first-time typed over there gets nothing.
  • The git clone step was missing. Getting started began at claude . with no instruction to obtain the repo first.
  • The by-hand fallback is now specific instead of reassuring. It said the skill "reads perfectly well as a checklist"; it now names the Step 0 audit as the plain shell block to paste, which is the part that actually is one. Claude Code stays named as the primary path — deliberately not "your favorite AI agent", which would be false, since other agents do not discover .claude/skills/.

Added

  • Issue chooser (#111). .github/ISSUE_TEMPLATE/config.yml sets blank_issues_enabled: false and adds contact links to the roadmap, the operational runbook, the security policy, and sales.demos for anyone whose question is about consuming an image rather than building one. bug_report.md and feature_request.md already existed, but GitHub offered "Open a blank issue" beside them, so people took it.
  • .github/CODEOWNERS (#111). There was none, so no review was ever auto-requested on a PR. sales.demos has had one; this repo now matches.
  • LVMS operator Day 0 manifest (#97). sno-manifests/lvms/ adds the lvms-operator Subscription so fresh SNO installs get a StorageClass out of the box. Needed for Compliance Operator scan PVCs and VM disks.
  • Root partition sizing (#97, corrected by #101). New MachineConfig template (98-lvms-partition) reserves the tail of the boot disk for LVMS by declaring a partition that starts at sno_root_partition_size_gb (default 200 GB), which is what caps root. Without this, RHCOS grows root to fill the disk and LVMS has nothing to use. Set to 0 to disable.

Changed

  • README opens with what the factory is for, not how it works (#111). It led with "This pipeline automates four stages" -- accurate, and it told a reader nothing about who the repo is for or what they get. Replaced with a value-first executive summary and an at-a-glance table (For / Produces / Run it / Status). The four-stage list is unchanged, now under ## Overview.
  • Quick Start is now Getting started, and it moved (#111). It sat fifth, about 55 lines down, behind Overview, Architecture, Supported platforms and the skills table -- so the section a new operator needs first was the last one they reached. It now follows the executive summary directly, matching sales.demos and sales.demos-docs. Prerequisites folded in beneath it as ### Prerequisites rather than standing alone before it.
  • Getting started leads with the first-time skill (#111). The skill was listed in the Claude skills table but never referenced from the setup path, so the one thing that validates every local prerequisite was invisible to someone actually setting up.
  • .github/pull_request_template.md renamed to .github/PULL_REQUEST_TEMPLATE.md (#111). Both work -- GitHub is case-insensitive here -- but sales.demos used the uppercase form and the inconsistency invited the question of which one is correct. Content unchanged.
  • ## Supported Platforms is now ## Supported platforms, so the anchor the new at-a-glance table links to resolves.
  • Default cluster name changed from demo to edge (#97). Base domain changed from example.com to internal.ames.net. Aligns with the sales.demos inventory environment name and avoids collision with the RHDP demo environment.
  • OCP channel bumped from stable-4.17 to stable-4.22 (#97). Matches the proven NUC install (OCP 4.22.13).

Fixed

  • The SNO root partition still filled the disk (#101). The Day 0 MachineConfig sized partition 4 directly. Ignition did that correctly — and ignition-ostree-growfs ran a second later and grew it back, because growpart expands root into any free space that follows it. Measured on the NUC: Ignition set 419430400 sectors (200 GiB), growfs logged CHANGED: partition=4 ... new: size=1999358607. Sizing root was never the right lever. The template now declares partition 5 at that offset instead, so growpart is bounded by it. Verified on the rebuilt NUC: sda4 199.5 GiB, sda5 753.9 GiB, 1007 KiB free.

growpart still logs CHANGED — that is the fix working, not the bug (#104). An earlier draft of this entry said it would report NOCHANGE; root is written at ~9 GiB, so it does grow, just no further than partition 5. Tell them apart by the end sector, not the word: end=2000409230 is the bug (end of disk), end=419430399 is correct (stops at partition 5).

This also fixes a second defect that would have survived the first: the design doc claimed LVMS auto-discovers unallocated space. It discovers unused block devices, and skips devices with children, so /dev/sda was never eligible. Partition 5 is left unformatted and unmounted so LVMS claims it with no deviceSelector change in install_lvms.yml. - SNO Day 0 manifest errors found during NUC boot (#95). Four issues blocked or delayed the first ABI ISO bootstrap: (1) AAP subscription used package name aap-operator instead of ansible-automation-platform-operator, (2) CNV was missing a Namespace manifest — openshift-cnv does not exist by default, (3) CNV was missing an OperatorGroup manifest, (4) the ScanSettingBinding cannot be a Day 0 manifest because the Compliance Operator CRD does not exist during bootstrap, causing bootkube to retry indefinitely and block the pivot. Fixed by correcting the package name, adding CNV namespace and operatorgroup manifests, and moving the ScanSettingBinding to Day 1 documentation. - The #92 gate looked for a control where it cannot exist (#93). verify_cis_disk.py checked DisableWebPnPDownload at \Policies\Microsoft\Windows; the CIS role writes it to HKLM:\SOFTWARE\Policies\Microsoft\Windows Nt\Printers (rule 18.9.20.1.1, level1-memberserver, enabled). The path omitted both Nt and \Printers, so the value could not be found on any machine. It was invisible because the only disk ever measured was unhardened, where the honest answer and the bug are both VALUE ABSENT — the same failure shape as #91 one level up, with the checker validated against an artifact that could not tell a pass from a bug. Caught the first time genuinely hardened media was read, which returned 9 of 10. Key paths now also resolve case-insensitively; regipy already does this, so it is belt-and-braces against an undocumented behaviour, not load-bearing. The same defect is fixed in sales.demos #370, which this reader was ported from.

Added

  • The Windows CIS level is now measured, not declared (#91). playbooks/scripts/verify_cis_disk.py reads the SOFTWARE and SYSTEM hives out of the qcow2 a publish is about to package — qemu-img, ntfscat and regipy, with no root, no libguestfs, no cluster and no booting the guest — and checks ten CIS controls that cannot be set on a clean install, so a pass cannot be a Windows default in disguise. publish_windows_containerdisk.yml refuses to apply com.redhat.cis.level=L1 unless the disk supports it, and fails equally when the check cannot reach a verdict, because "unverified" is the state that let #91 ship. A preflight check fails in a second if regipy is missing rather than forty minutes into the download. The verdict lands in cis_verify.json and in publish_output.json beside the level being claimed; the script also reports the disk's sysprep count and dates, which is how #91 was caught. Ported from sales.demos utilities/inspect-golden-image.py. -e cis_level=none publishes unhardened media ungated and says so out loud.
  • Phase 5: OpenShift SNO installer kit (#86). Agent-Based Installer ISO pipeline for bare-metal Single Node OpenShift with AAP 2.7, OpenShift Virtualization, Compliance Operator, and CIS L1 node hardening as Day 0 manifests. Initial scaffolding: build_sno_installer.yml playbook, Jinja2 templates for install-config.yaml and agent-config.yaml, Day 0 operator manifests (AAP, CNV, Compliance Operator), generate-iso.sh script, sno_defaults.yml, ROADMAP Phase 5, and docs/design.md §11.

Fixed

  • A publish repackaged a previous run's disk, and labelled it CIS L1 (#91). publish_windows_containerdisk.yml guarded the qcow2 conversion with creates: disk.qcow2 while its cleanup deleted only disk.img.gz and disk.raw — so a qcow2 survived between runs and the next publish skipped the conversion and packaged the stale disk. win2k22-cis-l1-golden:20260907-0516 is the 2026-09-05 unhardened build, byte-identical to win2k22-golden:20260905-2217 at 9307619328 bytes, and the demo guest built from it scored 9 of 27 CIS controls (sales.demos#358). Both creates: guards are gone, intermediates are purged before the run starts, disk.qcow2 and the verification scratch are removed in the always: block as build_cis_containerdisk.yml has always done, and the packaged qcow2 is asserted to be newer than the run that packaged it. The cluster was never at fault — the export selected the correct, freshly built PVC.
  • CIS controls that break WinRM mid-hardening (#79). Disabled four controls in cis_profile.yml that kill the WinRM session during the build: win22cis_rule_2_3_17_1 and _2 (UAC Admin Approval Mode — NTLM credentials rejected), win22cis_rule_18_9_19_4 and _5 (security policy background refresh — triggers immediate reprocessing of all security settings, dropping the connection). Not covered by the role's win_skip_for_test. Also added a fail: task in Play 3 so a lost WinRM connection fails the build instead of exiting 0.
  • Edge AppX package blocks sysprep after CIS hardening (#79). CIS hardening installs or updates Microsoft Edge as a per-user AppX package. Sysprep refuses to generalize with per-user packages not provisioned for all users (0x80073cf2). Added an AppX cleanup step after CIS and before sysprep.
  • Port-forward dies mid-CIS hardening (#79). kubectl port-forward uses a WebSocket that OpenShift drops after ~15 minutes. The CIS role takes ~15 minutes. Wrapped the port-forward in a keepalive loop that restarts it automatically; WinRM reconnects on the next task. Eliminates the need to re-run the playbook when the tunnel drops.

Documentation

  • How the Windows CIS claim is evidenced (#91). docs/design.md §10.2 now states that com.redhat.cis.level is an observation of the disk rather than an input to the publish, and §10.5 splits compliance evidence by OS. The section previously deferred per-format scanning on the grounds that "the same profile applied to the same distribution produces the same compliance posture regardless of output format" — true for RHEL, where Image Builder runs OpenSCAP, and false for Windows, which has no compose and no scan and was taking the operator's intent as evidence.
  • CIS L1 image verified end-to-end (#84). Updated docs/design.md §10.2 to reflect that win2k22-cis-l1-golden:20260907-0516 is published and consumed. win_ping from AAP returned ok: 1 against a clone provisioned from the CIS image (sales.demos #270, #294). Removed stale "until PR 2" references.

Changed

  • Worktrees mandatory for code changes (#77). Strengthened CLAUDE.md from defensive habits only to an unconditional worktree rule — the main checkout stays on main as read-only. Matches sales.demos #267.

Added

  • CIS L1 Member Server hardening in the Windows build (#24 PR 2). build_windows_image.yml now applies ansible-lockdown/Windows-2022-CIS to the build VM over WinRM before sysprep. Default is windows_cis_harden=true. Three plays: install, harden, cleanup. The hardening profile forces non-cloud lockout order (hosted_virtual_system_override: false) to avoid the secedit failure on KubeVirt, and win_skip_for_test: true skips 11 controls that would kill WinRM mid-run. Verified 2026-09-06: 44 controls applied, idempotent on re-run.
  • playbooks/vars/cis_profile.yml — CIS role configuration extracted to a vars file. Documents the KubeVirt auto-detect bug and the 11 skipped controls with reasons.
  • roles/requirements.yml — installs the CIS role from ansible-lockdown/Windows-2022-CIS.
  • collections/requirements.yml gains ansible.windows, community.windows, community.general — the CIS role's dependencies, pinned to the versions verified against.
  • publish_windows_containerdisk.yml now defaults to cis_level=L1 and win2k22-cis-l1-golden, matching the build default. Override both with -e cis_level=none and QUAY_WINDOWS_REPO for unhardened builds.

Changed

  • windows_sysprep_at_first_logon defaults to false (was true). The build now syspreps via WinRM after hardening rather than from the answer file. Pass -e windows_cis_harden=false -e windows_sysprep_at_first_logon=true for the old PR 1 behavior.

Fixed

  • README: corrected Quay.io repo visibility (#73). The pipeline diagram said Quay.io (private) but the RHEL CIS containerDisk repo is public — only the Windows repo is private. Added a Quay.io private repo entitlement section with screenshot documenting that other SEs need at least a Developer plan or Red Hat developer subscription for private repos.
  • Removed hardcoded RHDP cluster ID from build_windows_image.yml (#64). The demo-cluster guard defaulted demo_cluster_marker to a live cluster ID committed in a public repo. Now requires demo_cluster_hostname as an extra-var with no default — the build fails closed when it is absent, consistent with how windows_eval_expires is handled. The guard logic and override escape hatch are unchanged.
  • The cached answer file delete now works — Windows SMI was rejecting the CommandLine as too long (#69). PR #70's single PowerShell command was 1031 characters; Windows SMI rejects SynchronousCommand/CommandLine values over ~1024 characters with error 0x80220005 during the oobeSystem pass, which invalidates the entire pass — AutoLogon, FirstLogonCommands, everything — dropping OOBE to the manual region screen. Split into three short commands (Orders 5–7, each under 410 chars): delete+log, cleanup other locations, and assert. Root cause diagnosed by parsing UnattendGC\setupact.log off the raw NTFS disk.

Added

  • The post-#59 rebuild reproduces the measured build time, so the fix is free (#63). Rebuilt on cluster-kbjvc 2026-09-05, teardown of the spoiled namespace through to a Stopped generalized VM: 21m25s, against the 21m26s measured in #56. One second apart on an unattended 21-minute build is the same number twice, which is what makes it a measurement rather than an anecdote — n=2 on the figure ROADMAP.md quotes. The guest phase reproduces on its own too: the power-off watch took 31 polls at 30s ≈ 16m, against #56's 16m43s for install through sysprep.
  • That is the point worth recording: deleting the cached answer files costs no measurable build time. #59 adds five del targets to FirstLogonCommands at <Order>5</Order>, and a reasonable worry about putting work between first logon and sysprep /generalize is that it lengthens the slowest, least observable stretch of the build. It does not — the delta is inside the noise of two runs. PR 2's rebuild loop is unaffected, which matters because CIS hardening is the phase that pays 21 minutes per iteration.
  • What this does NOT establish. The rebuild proves the image builds; it does not prove a clone specializes. #59's whole failure mode was a build that looked perfect and a consumer that silently ignored the sysprep CD, and nothing in this playbook can see that — the Order 5 delete is fire-and-forget (2>nul & exit /b 0) by design, so it cannot fail the build even if it does nothing. The proof is downstream in sales.demos, and #63 stays open until a clone reaches the desktop instead of the OOBE region screen.
  • Cross-repo sessions should start in sales.demos (#65). This repo has no MCP servers at all — no .mcp.json, nothing in settings — while sales.demos/.mcp.json defines openshift-sandbox (read-write, toolsets core,config,kubevirt) and openshift-demo (read-only). Both are project-scoped and load only when Claude Code starts in that directory; a session started there can still cd here and run these playbooks, so it is strictly better than the reverse. Recorded in CLAUDE.md and the windows-image-build skill.
  • With the caveat that has to travel with it: MCP does not supply the credentials. The playbooks read K8S_AUTH_HOST and K8S_AUTH_API_KEY from the environment and assert them non-empty. Recording the pattern without this would read as "start there and the credentials sort themselves out", which is worse than not recording it.
  • docs/design.md §4.1 — where the Windows build's three variables actually come from (#61). §4 listed K8S_AUTH_HOST, K8S_AUTH_API_KEY and WINDOWS_ADMIN_PASSWORD and said credentials are env vars, which is the playbook's whole contract — but never said where an operator gets them. All three are maintained in sales.demos and nowhere else, and rediscovering that costs a search of another repo's vault; it has now been done twice. §4.1 names the three files, and deliberately does not reproduce the API URL, which embeds a live RHDP cluster ID.
  • The distinction the note preserves: the playbook's contract is not the same question as where a human fills it in. build_windows_image.yml loads no vault and imports nothing from sales.demos, so it stays runnable by someone who has never seen that repo — its header said so, but in a way that read as "these values have nothing to do with sales.demos", which is wrong. Both halves are now stated separately.
  • And the rule that falls out of it: do not materialise them into a file. sales.demos deliberately allows exactly one secrets file and no sourceable second copy. A plaintext .env here would be that second copy, in a second repo, expiring never. Export them into the shell that runs the playbook.
  • playbooks/publish_windows_containerdisk.yml — PR 3 of #24: the Windows golden image is published. VirtualMachineExport → disk.img.gz → sparse expand → qemu-img convert → FROM scratch + COPY to /disk/disk.img → podman build → podman push. Published and verified from the registry: quay.io/zigfreed/win2k22-golden:20260905-1826, digest sha256:61c8d5ef…, private, all labels present. Whole run 27 minutes.
  • docs/design.md §10 gains its Windows half — §10.1 and §10.2 carry the Windows repository, the two Windows-only labels, and cis.level=none; §10.2.1 documents the export path and the four things about it that are not obvious.
  • The publish is deliberately not win2k22-cis-l1-golden. What it holds is the unhardened build, and tags are immutable — a repository name claiming L1 would make that claim permanently, on media that never had it. PR 2 publishes win2k22-cis-l1-golden separately. com.redhat.cis.level=none is stated rather than omitted, because an absent label reads as an oversight while none is a claim, and it is the true one.
  • image-factory/eval-expires is stamped on the published image. The 180-day clock is the thing people forget, and a label on a build VM in a namespace that no longer exists helps nobody.
  • docs/operations.md — operational runbook for manual rebuild triggers, secret rotation, and troubleshooting. Closes #52
  • README.md CI/Automation section documenting both GitHub Actions workflows
  • The Windows build no longer needs anyone at the console (#40). The stock ISO's UEFI boot image asks "Press any key to boot from CD or DVD" on a ~5 second timeout and fails silently when nobody answers -- the console just sits there, so a build that needs a human is indistinguishable from a build that is merely slow. It cost a wasted 25-minute run. A pod on the cluster now fetches the ISO, verifies windows_iso_sha256, swaps Microsoft's own efisys_noprompt.bin and cdboot_noprompt.efi over their prompting twins, rebuilds with xorriso and serves the result over HTTP; the existing source.http DataVolume points at that Service instead of at Microsoft. Only the URL differs -- DataVolume, VM, autounattend.xml, FirstLogonCommands and sysprep are untouched and already verified. On the cluster and not the laptop for the reason #36 measured: ~246 MiB/s there against ~1 MiB/s up from here.
  • playbooks/scripts/remaster_iso.sh -- the recipe is Red Hat's, verbatim, from the modify-windows-iso-file task in kubevirt-tekton-tasks v0.26.0. It does not re-point El Torito at the no-prompt image; it deletes efisys.bin and cdboot.efi and renames the _noprompt twins over them, then rebuilds. Both files matter: efisys.bin is the El Torito boot image and cdboot.efi is what lives inside it. Deviating from a shipped, working recipe would buy nothing and risk media that boots on one firmware and not another.
  • .github/workflows/containerdisk-rebuild.yml — monthly GitHub Actions scheduled rebuild for the RHEL 9 CIS L1 containerDisk. Runs 1st of every month at 06:00 UTC; also supports manual workflow_dispatch triggers. Zero playbook changes — secrets provide the RH token and Quay credentials. Closes #48
  • playbooks/build_cis_containerdisk.yml — builds RHEL 9 CIS L1 qcow2 via Image Builder guest-image type, wraps as containerDisk, pushes to private Quay.io repo. Closes #21
  • wait_for_compose.py handles guest-image compose results (download URL) in addition to AMI compose results — backward-compatible
  • docs/design.md §10 — containerDisk contract for OpenShift Virtualization: distribution model, OCI labels, credential pattern, compliance evidence strategy. Parallel to §9's AMI contract
  • ROADMAP.md Phase 1.7 — RHEL 9 CIS L1 containerDisk
  • .claude/skills/rhel9-containerdisk/SKILL.md — preflight, API validation, run, verify, and troubleshooting for the RHEL 9 containerDisk build
  • .claude/skills/windows-image-build/SKILL.md — the Windows build had no skill, so its preflight existed only as asserts inside the playbook, discovered one failure at a time. Covers acquiring the media, the run, verification that asks the cluster rather than trusting the recap, and the teardown. Closes #30
  • playbooks/scripts/wim_images.py — prints the image names inside a .wim by parsing the WIM header's XML resource. windows_image_name must match one of those names exactly; get it wrong and Windows Setup stops on the edition-selection screen, which reads as a hang because there is no console output to explain it.
  • README.md now lists the four skills. There was no skills section at all, despite three already existing.
  • playbooks/build_windows_image.yml + playbooks/templates/autounattend.xml.j2 — unattended Windows Server 2022 build for the Phase 3 containerDisk. PR 1 of 3 on #24: build only, unhardened. Hardening is PR 2, export and publish are PR 3.
  • Built on the cluster, not on a laptop. #21 flags "a local libvirt/KVM scan path — new hypervisor dependency" as its High-risk item; a KubeVirt cluster is a hypervisor, so this avoids that dependency rather than incurring it, and the image is exercised by KubeVirt before any consumer sees it. Plain VirtualMachine objects driven with kubernetes.core, the same way this repo already drives EC2 with amazon.aws — no operator is installed, so there is no cross-repo dependency on the consumer's cluster configuration.
  • The answer file is delivered as a ConfigMap. KubeVirt renders a configMap volume as an iso9660 CD and Windows Setup reads autounattend.xml from removable media, so no ISO-authoring tool is needed on the machine running the playbook.
  • Sysprep is flag-guarded. windows_sysprep_at_first_logon defaults true so PR 1 finishes without a WinRM round trip and stays free of the Windows collections. PR 2 sets it false and syspreps after hardening — generalizing first would throw the hardening away.
  • collections/requirements.yml pins kubernetes.core 6.4.0, matching sales.demos so producer and consumer cannot disagree about the client library.
  • docs/design.md §4 documents K8S_AUTH_HOST, K8S_AUTH_API_KEY and WINDOWS_ADMIN_PASSWORD — env vars, consistent with the model that section already states. Nothing is read from sales.demos' vault.
  • The playbook refuses to build on the demo cluster. Builds run on sandbox; demo consumes the published tag and never hosts a build. Asserted rather than merely documented, because a 45-minute Windows install on a cluster someone is presenting from is the mistake worth making impossible.
  • windows_eval_expires is required. Evaluation media expires after 180 days and an expired Windows guest nags and then shuts down hourly. The date is asserted, stamped onto the VM as a label, and printed at the end so it reaches the run-sheet.
  • .claude/skills/collections-sync/SKILL.md — pin, install, and verify Ansible collections; audit script detects drift between pinned and installed versions. Closes #27
  • .claude/skills/first-time/SKILL.md — prerequisite validation for new sessions: Hub token in ~/.ansible.cfg, collections, AWS credential pattern, troubleshooting table
  • CLAUDE.md skills table listing all three Claude agent skills
  • .claude/skills/dev-workflow/SKILL.md — mandatory development cycle for Claude agents working in this repo
  • CLAUDE.md workflow section — documents protected main, branch naming, multi-session safety, standing merge authorization
  • main branch protection: required CI checks (yamllint, ansible-lint), PRs required, enforce admins, no force pushes
  • GitHub community health files: CONTRIBUTING.md, .github/SECURITY.md, .github/ISSUE_TEMPLATE/bug_report.md, .github/ISSUE_TEMPLATE/feature_request.md, .github/pull_request_template.md — tailored to pipeline context, ROADMAP phases, and the two-consumer contract
  • playbooks/vars/exempt_controls.yml — curated exempt list with canonical reasons for AWS-inherent CIS rules (grub2_password, ensure_root_password_configured, partition_for_tmp). Merged into data.json at generate time; curated entries take precedence over parser-auto-emitted candidates. Closes #11
  • build_cis_image.yml Image Builder customizations.packages list — installs aide, firewalld, systemd-journal-remote at build time. First experiment toward closing the CIS L1 packaging gaps tracked in #10
  • Two more curated exempts (closes #12): file_permission_user_init_files (P2; rule targets deployed-system user homes, not the image artifact) and sshd_limit_user_access (P3; SSH access policy is a consumer decision)
  • docs/cis-l1-rhel9-status.md — snapshot of the latest validated compliance state (AMI, score, exempt rules, reproduction steps, validation history). Linked from README's platform table
  • Project docs synced to Phase 1 completion state: ROADMAP marks Phase 1 Complete with all new tasks logged; CLAUDE.md current-state paragraph updated; design.md §3 data.json example replaced with a real entry instead of placeholder text
  • Phase 1.5: pipeline applies the AMI tagging contract (6 tags: Pipeline, OS, CIS-Level, BuildDate, ComposeID, Name) via amazon.aws.ec2_tag after compose. Design changed from name-based to tag-based discovery — docs/design.md §9.1-9.2 rewritten. DC1's eventual filter becomes a tag filter, not a name filter
  • Initial repository structure, MIT License, Contributor Covenant Code of Conduct
  • README with architecture overview and quick start
  • ROADMAP — Phase 1 (RHEL 9 L1), Phase 1.5 (DC1 integration), Phase 2 (L2 / RHEL 8), Phase 3 (Windows), Phase 4 (other platforms)
  • docs/design.md — full pipeline design, including §6 (OPA consumer) and §9 (demo.datacenter consumer / AMI naming contract)
  • playbooks/build_cis_image.yml — Image Builder API integration: token exchange, blueprint creation, compose trigger, polling, AMI capture
  • playbooks/deploy_and_scan.yml — EC2 deploy, SCAP result extraction, fresh OpenSCAP scan fallback when build-time results are missing
  • playbooks/generate_policy_data.yml — XCCDF results parser, hardening-score computation, data.json generator per docs/design.md §3
  • playbooks/filter_plugins/xccdf.py — namespace-agnostic XCCDF parser; returns rule counts, severity breakdown, hardening score (pass / (pass + fail), excluding N/A and notchecked), and AWS exempt-control candidates (auto-emitted for low-severity failures)
  • Sample inventory — Red Hat offline token via ~/.ansible/ansible.cfg, AWS credentials via env vars
  • collections/requirements.yml pinning amazon.aws to 11.2.0
  • .gitignore covering credentials, collections, and generated output
  • GitHub Actions lint workflow (ansible-lint, yamllint)
  • Project-level CLAUDE.md capturing repo conventions and the two-consumer contract

Changed

  • The re-master is now a byte-exact patch rather than a rebuild (#44), and it is smaller than the design it replaces. Measured on the real medium: efisys.bin and efisys_noprompt.bin are the same size (1,474,560), and the hidden El Torito boot image is a byte-identical copy of the first (3c9eda1a…). So the whole job is a same-length overwrite of 720 sectors at one LBA — no 4.7 GB extraction, no re-authoring, no discarding UDF, and iso_remaster_scratch_size drops from 20Gi to 8Gi. Red Hat rebuilds because their pipeline has already extracted to a PVC; there is no such reason here.
  • playbooks/scripts/remaster_iso.sh locates the EFI boot image by parsing the Boot Record Volume Descriptor at LBA 17 and walking the El Torito catalogue for the 0xEF platform section — verified against the real medium, which reports catalogue LBA 22 and load RBA 523, matching what libisofs independently printed. Nothing is written until the bytes at that address are proved byte-identical to efisys.bin, so a misread catalogue fails instead of corrupting the medium. After the patch it re-checks the extent, the ISO's length, and that the medium still mounts and still holds install.wim.
  • The re-master pod becomes an initContainer on tekton-tasks-disk-virt (guestfish, plus devices.kubevirt.io/kvm) and a serve container on tekton-tasks (python3, which the guestfish image lacks). The KVM request is an extended resource from KubeVirt's device plugin, not a privileged securityContext — the pod still runs under restricted-v2 as an arbitrary non-root UID. Both images pinned by digest, verified against what this cluster resolves the v0.26.0 tags to.
  • .claude/skills/windows-image-build/SKILL.md still described the pre-#36 media path in every paragraph that only implied a local ISO, after #41 corrected the section that said so outright (#42). "On the specific ISO you downloaded" and a 7z e -so ~/Downloads/... snippet both assume a copy that no longer exists by default -- the cluster imports the media, pinned by windows_iso_url and windows_iso_sha256, and the hand-fetch is now shown only where it is still true: after repointing that URL. The Provisioning troubleshooting row said "the ISO upload is ~4.7 GB" when there is no upload on the default path, and did not mention that the ISO DataVolume now waits on the re-master pod.
  • The Run section contradicted itself, and that is the sharp one: "about forty-five minutes" four lines above the breakdown #41 added, which totals about twenty-two. A skill that quotes two runtimes teaches the reader to trust neither, and the timing is not decoration -- a build with nobody watching needs a reliable "is this stuck?" threshold, which is precisely what #40 made the normal case. Now one number, thirty minutes, with what to look at past it.
  • README.md "Related Projects" renamed to Related repositories and rewritten to say what each consumer receives and which way the dependency runs, rather than listing them in bare lines. Names the Windows producer/consumer split explicitly: building the image is #24 here, pointing a cluster at it is sales.demos#3, and the only thing binding them is one string -- a containerdisk tag. Reciprocal to sales.demos#204, so the link works both ways. Closes #38
  • The re-master is on by default and asserts rather than assumes: windows_iso_remaster requires windows_iso_source=url, because the pod fetches from windows_iso_url and cannot re-master a file that exists only on the operator's laptop. -e windows_iso_remaster=false returns the stock ISO and a keypress, and the final recap now says which media the build used.
  • The DataVolume records which media it holds, as image-factory/iso-variant: noprompt|stock, and a Succeeded import of the wrong variant is re-imported. This is defect 3 from #34 -- existence is not success -- in its next disguise: an import made before the re-master existed holds the prompting ISO, is Succeeded, is the right size, and produces a build that stops dead at the console. Phase alone cannot tell them apart. A DataVolume with no annotation counts as a mismatch and is re-imported once, which costs about two minutes and is the right way round to be wrong.
  • The re-master's scratch space is a PVC, not an emptyDir. Peak use is the extracted tree plus the rebuilt ISO -- about 9.4 GB -- and an emptyDir spends the node's ephemeral storage, where crossing the line evicts the pod under disk pressure and reads as an unexplained restart rather than as "out of space". Pod, Service, ConfigMap and PVC are all deleted as soon as the import succeeds.
  • The re-master image is pinned by digest (quay.io/kubevirt/tekton-tasks@sha256:87743fb6..., tag v0.26.0), on the same reasoning execution-environment.yml pins its base: a factory that cannot say what it consumed cannot say what it produced.
  • .claude/skills/windows-image-build/SKILL.md -- the media section still told the operator to download 4.7 GB by hand and passed -e windows_iso_path=, both left over from before #36 made url the default. The cluster fetches the media; the manual download now appears only where it is still true, on the upload escape hatch. Adds the no-keypress section, the iso-variant check to the verification block (jsonpath form measured against a live cluster -- the single-quoted bracket form and the plain dot form both work for this key, the double-quoted bracket form does not), and four troubleshooting rows.
  • ROADMAP.md Phase 3 -- the keypress row, marked done.
  • windows_iso_sha256 now defaults to the verified checksum rather than empty, on the same reasoning execution-environment.yml pins terraform: a factory that does not know what it consumed cannot say what it produced. Pass -e windows_iso_sha256='' to skip.
  • build_windows_image.yml records the four WIM images the evaluation ISO actually contains, so the windows_image_name default is documented as verified rather than assumed.
  • All cross-repo references updated: aap.as.code and demo.datacenter → sales.demos throughout CLAUDE.md, ROADMAP.md, README.md, CONTRIBUTING.md, and docs/design.md §9
  • docs/design.md §9 header renamed from "Integration with demo.datacenter (DC1)" to "Downstream consumers" — the mechanism is consumer-agnostic
  • Phase 3 roadmap rewritten: Windows Server 2022 will ship as a CIS-hardened containerDisk on Quay.io, not an AWS AMI. Tracked in #21
  • CLAUDE.md consumer table expanded to three rows (rego_policy_libraries, sales.demos AMIs, sales.demos containerDisks)
  • Branch naming convention aligned with sales.demos: <type>-<issue>-<slug> replaces <type>/<description>
  • CONTRIBUTING.md updated: main is now protected, PRs always required
  • Dynamic VPC and subnet discovery instead of relying on default VPC
  • Async oscap execution for long-running scans
  • IdentitiesOnly=yes for SSH; unique per-run EC2 instance names
  • CI lint workflow restored to green (closes #2): .yamllint relaxes line-length to 120 (Ansible community norm); ansible/ansible-lint action bumped v24 → v26 for ansible-core 2.19 compatibility
  • build_cis_image.yml: build-output dict moved to a vars: block (was a 224-char inline Jinja expression)
  • Phase 1 end-to-end run completed 2026-05-11 — first real RHEL 9 CIS L1 AMI through the full pipeline. Score 94.94 vs gate 95; follow-ups tracked in #4, #5, #6, #7

Fixed

  • The cached answer file delete is now observable and guarded (#69). #60's cmd /c del ran, had permissions (proved by a clone test on the running image), and left the file untouched — but 2>nul & exit /b 0 discarded the error, so the reason could not be determined from the build. Replaced with PowerShell Remove-Item that logs before/after state to C:\Windows\Temp\panther-delete.log, and added a new <Order>6</Order> assert that fails the build if the file survives. Sysprep moves to <Order>7</Order>. The assert is the guard #60 should have shipped with — nothing in the build verified the delete worked, which is why a broken fix published, reached a consumer, and was only caught by a human at the OOBE screen.
  • Every clone ignored the consumer's sysprep answer file, because the build left its own cached (#59). Windows Setup caches the answer file it used into %WINDIR%\Panther, and Microsoft's implicit search order puts that cache at precedence 3 — ahead of removable read/write media at 4 and read-only removable media at 5, which is exactly where KubeVirt's sysprep API presents the consumer's CD. So sales.demos built a correct sysprep Secret, attached it, saw the 1 MiB ISO materialise as sdb — and every clone still specialized from this build's answer file and stopped on the OOBE region screen. The consumer half (sales.demos#201 / PR #227) was right all along and needed no change. autounattend.xml.j2 now deletes the cached copies at <Order>5</Order>, and sysprep moves to <Order>6</Order>.
  • The delete is deliberately outside the windows_sysprep_at_first_logon guard. The hardening work sets that flag false and syspreps later over WinRM; that path needs the cache gone just as much, and nothing after first logon re-caches it. Guarding it would mean rediscovering #59 the first time an image is hardened.
  • It also stops shipping windows_admin_password in clear text inside the published containerDisk. Microsoft gives that as the second reason the cleanup is mandatory before delivery. The password is a throwaway by design, so this is a real fix and not an urgent one.
  • The export's token secret is named in status.tokenSecretRef, not spec. virtctl sets spec itself, so a manual check with virtctl exercises a path a playbook creating the export directly never takes — the same shape as #44's synthetic ISO: testing with a different tool than the code uses proves the wrong thing.
  • A no_log: true task that fails tells you nothing. The token read is correctly no_log — its result carries the token — but the failure came back as censored and cost a run to diagnose. no_log stays; asserts either side of it now fail with real messages.
  • The export offers every volume the VM had, the install ISO included. The root volume is selected by name — taking the first would publish a Windows installer as a golden image, and it would look plausible until a consumer booted it.
  • Running out of disk happens at the END, after the slow download has already succeeded. output_dir defaults inside the repo, and the intermediates are tens of GB where the RHEL containerDisk's are ~2. There is now a free-space assert that fails in a second, and the gzip and raw are deleted as soon as each is consumed — peak ~26 GiB instead of ~40.
  • Teardown of a wedged build took an hour, and a build started behind it silently reused the old VM (#54). Two defects, one sequence, both hit while rebuilding for #50.

The hour: the build VM inherited terminationGracePeriodSeconds: 3600 from KubeVirt's Windows preference. Measured — a launcher pod deleted at 16:41 carried deletionTimestamp 17:41. A guest wedged mid-Setup never shuts down cleanly, so the pod waits out the entire grace period with the namespace Terminating behind it. An hour is right for a machine someone cares about and wrong for a throwaway that exists to be rebuilt; now 60 seconds via build_termination_grace_seconds, which still lets a healthy guest flush and halt.

The quiet part: the build then ran anyway, against the terminating namespace. Kubernetes refuses to create objects there — but every object the build needs already existed from the previous run, so state: present was a no-op patch and nothing errored. The playbook waited for the previous, looping VM to power off, having reported it had created a new one. It looked completely normal. There is now an assert that refuses to build into a Terminating namespace.

  • The safe way to clear a wedged guest is virtctl stop <vm> --force --grace-period=0, which asks virt-handler to destroy the domain through the KubeVirt API. Used here, it cleared in about five minutes what had fifty left to run. It is not the same as oc delete pod --force, which wedges virt-handler so it then refuses to start any new domain with nothing surfaced anywhere. The new failure message names the safe path and warns off the other.

  • Teardown staying asynchronous is deliberate and was not changed. wait: false is right — a namespace with PVCs can take many minutes and nothing in the teardown path needs to watch it go. The fix belongs in the build refusing to start, not in the teardown blocking.

  • Removing the keypress made the install CD boot unconditionally, so Windows Setup restarted forever (#50). The first build after #40 ran four hours with the progress bar moving and never finished. It was not slow — progress went backwards: 51% at 12:36, 82% at 14:08, 49% at 16:33. That figure is monotonic within a run, so a decrease means a new run.

Windows Setup reboots partway through every installation. The firmware re-runs the boot order, and with installcd bootOrder=1 and no prompt left to time out, it booted the CD again and Setup began from zero. virt-launcher showed one continuous domain the whole time, so nothing above the guest could see it.

"Press any key to boot from CD or DVD" is what made CD-first survivable. It is the mechanism that lets a machine with bootable install media boot the installed OS on later reboots. Fixed by ordering rootdisk bootOrder: 1, installcd bootOrder: 2 — which is what Red Hat's windows-efi-installer ships, and which is self-resolving: the blank disk has no ESP so the firmware falls through to the CD on the first boot, and once Setup has written an ESP the installed OS boots and carries on to sysprep.

  • This is #36 and #40 interacting, and #36 was right at the time. #36 moved the CD ahead of the disk because with the disk first "EFI tried the blank root disk, found no ESP, fell through to the CD, and by then the only thing on screen was a prompt nobody could answer." That is the same fall-through the fix now relies on — measured, on this cluster. #40 removed the reason for #36's order and should have reverted it in the same PR. The two settings are one decision and are now commented as one.

  • A build that never reaches Stopped now says what that usually means. The 60-minute timeout already existed and would have caught this, but with a generic message. It now names the loop, tells you to read the console percentage twice, and points at the boot order.

  • The guestfish image ships no CA bundle, so the re-master could not fetch the ISO at all (#46). curl: (77) error setting certificate file. Probed against the pinned digest: neither /etc/ssl/certs/ca-certificates.crt nor /etc/pki/tls/certs/ca-bundle.crt exists in tekton-tasks-disk-virt, and its curl is RHEL-built expecting a trust store the image does not carry — so every HTTPS fetch from it fails, whatever the URL. Split into three stages: fetch on tekton-tasks, which had already pulled this exact URL from Microsoft successfully; remaster on tekton-tasks-disk-virt with the KVM device, which now needs no network at all; and serve. Each image does only what it is demonstrably able to do.
  • #45 was verified with ISO_URL=file://, because the ISO was already on the PVC from the failed run — so the fix was proven against the real medium and the transport went untested. Same shape as #44 one layer over: the part that did not have to be re-run is the part that broke. Rejected curl -k as the fix: windows_iso_sha256 is pinned and checked, so TLS is belt-and-braces here, but turning verification off to work around a missing trust store is the wrong habit to write into a factory.
  • The failure rescue now reads both initContainer logs. Kubernetes refuses an unqualified log request once a pod has more than one container, and either stage can be the one that failed — asking for only remaster would have returned an empty log for every fetch-side failure, including this one.
  • The #40 re-master failed on real media: the Windows ISO is UDF and xorriso cannot read it (#44). The first real run extracted one node, 135 bytes, and the extraction guard fired. guestfish ... list-filesystems on the medium the importer had already fetched says /dev/sda: udf; the ISO 9660 tree is a stub. It has to be UDF — sources/install.wim is 4,340,202,461 bytes, past ISO 9660's 4 GiB single-extent limit. libisofs, which xorriso reads with, has no UDF reader. This also explains a detail #41 had read past: Red Hat's modify-windows-iso-file extracts with guestfish and asks for devices.kubevirt.io/kvm, and #41's PR body treated avoiding that device as a win. It is not a win — the device is what buys UDF support, because libguestfs boots a kernel that has a UDF driver.
  • The verification in #41 used a synthetic ISO built by the tool under test, so it proved the mechanism and never the medium. It established that xorriso reads Joliet when Rock Ridge is absent — true, and irrelevant, because the real medium's ISO 9660 side carries nothing to read. Same lesson as the seven defects on #24, one level up. The guard added in #41 is what caught it, and it caught it in ninety seconds with nothing built.
  • "{{ x | int }}" inside a nested definition reaches the module as the STRING '8080', and bare "{{ x }}" reaches it as the integer. The opposite of what the filter name suggests, measured rather than assumed. containerPort is int32 and both the readiness probe's port and the Service's ports are IntOrString, where a string means a named port -- so the wrong one does not error anywhere. The probe simply never passes and the pod never becomes Ready. Caught before the first run; it would have looked like a slow re-master.
  • spec.running: true restarted the VM after sysprep, booting the generalized image into OOBE (#37). running is a desired state, not one-shot: when the guest powered itself off at the end of sysprep /generalize /oobe /shutdown, KubeVirt started it straight back up. This is the most dangerous defect found so far, because it produces a disk that looks fine and is not — OOBE consumes the generalization, and a "golden image" that was never generalized surfaces much later as cloned VMs colliding on SIDs and machine names. Caught only because someone had the console open. Replaced with runStrategy: Once, which starts the VMI exactly once and leaves the VM Stopped when the guest halts — also making "wait for Stopped" a real completion signal rather than a race it could never win.
  • The ISO was transferred the wrong way across the network (#36). The playbook had the operator's machine download 4.7 GB from Microsoft and push the same 4.7 GB back up a domestic uplink. Measured: ~1 MiB/s up, ~75 min, against ~246 MiB/s for CDI importing directly on the cluster — 117 s end to end, verified. windows_iso_source now defaults to url and CDI imports via source.http; the upload path remains for a cluster with no egress. This was not a bug — the code did what it said. It was the wrong shape, and only a stopwatch revealed it.
  • The virtio-win image reference was wrong three ways (#36): the repo is virtio-win-rhel9 not virtio-win, it refuses the :latest tag so an untagged reference cannot pull at all, and CNV pins it by digest which moves with the CNV version. No hardcoded string stays correct. The playbook now reads the virtio-win ConfigMap that HCO publishes in openshift-cnv, which is right on any cluster at any version; virtio_win_image remains an override for air-gapped mirrors.
  • Boot order put the blank root disk ahead of the install CD (#36), so EFI tried an empty disk first and fell through to a prompt nobody could answer.
  • cnv_namespace was referenced as win_cnv_namespace, a variable that never existed. ansible-lint passes this at the production profile — undefined runtime variables are not something static analysis catches, which is worth remembering when a lint-green playbook has never been executed.
  • virtctl ignored K8S_AUTH_* and silently targeted localhost:8080 (#33). Those are the python client's variables, consumed by kubernetes.core; virtctl is a Go client that reads a kubeconfig or its own flags. The play-level environment: block covered every kubernetes.core task and nothing else — invisible until the playbook actually ran, which #29 said in as many words had not happened. Fixed with a mode 0600 kubeconfig written to a temp file and deleted in an always: block; --token would have worked and would have put the bearer token in the process list where any local user can read it with ps.
  • The ISO upload deadlocked on WaitForFirstConsumer storage — cannot upload to DataVolume in PendingPopulation phase. The PVC will not bind until a pod consumes it, and the upload needs it bound first. --force-bind is the imperative twin of the cdi.kubevirt.io/storage.bind.immediate.requested annotation. Measured on sandbox: the default class is ocs-external-storagecluster-ceph-rbd, WFFC.
  • The idempotence check tested existence rather than success, and that was the dangerous one. A failed upload leaves the DataVolume behind in PendingPopulation, so a re-run that only asked "does it exist?" would have skipped the upload and installed Windows from an empty disk — a broken image that looks like a successful build. The playbook now reads the DataVolume's phase, deletes a non-Succeeded one, and re-uploads.
  • ROADMAP.md Phase 3 and CLAUDE.md tracked the Windows producer as sales.demos#193. That issue was transferred into this repo and is now #24, so the roadmap asserted "permanent home is this repo" while the tracker pointed elsewhere. GitHub redirects a transferred issue so neither link was broken, but the reference read as "the work lives in the other repo" — the opposite of the decision Phase 3 records — and sent a reader out of this repository to be bounced back into it. sales.demos updated its 24 references in its PR #200. Closes #25
  • build_cis_image.yml token lookup path — was reading from frozen ~/.ansible/ansible.cfg (stale Oct 2025 token) instead of active ~/.ansible.cfg. Latent bug that would 401 on next token rotation. Closes #22
  • Stale "Same AWS account as DC1" claim in CLAUDE.md — Image Builder builds in service account 463606842039 and shares with the consumer account; CLAUDE.md now matches the corrected design.md §9.1
  • docs/design.md §9.1 and ROADMAP.md AMI ownership contract — Red Hat Image Builder is a hosted service that builds AMIs in its own service account (463606842039) and shares them with the consumer account via the API's share_with_accounts field. The consumer never owns the AMI, only receives launch permission. Previous text claimed "pipeline and DC1 share one AWS account; no cross-account sharing" and prescribed owners = ["self"] for the DC1 data source — both wrong. Surfaced when demo.datacenter PR #14 ran end-to-end and terraform plan returned "no results" against the shared AMI.
  • AMI region taken from compose result instead of assumed
  • network_interfaces deprecation in amazon.aws module
  • Empty TARGET_PLATFORM env var handled with default filter
  • Duplicate vars block in deploy_and_scan.yml merged
  • generate_policy_data.yml now reads scap/scap-results.xml (matches the path deploy_and_scan.yml writes)
  • build_cis_image.yml no longer fails mid-compose when the access token expires — polling moved to scripts/wait_for_compose.py which refreshes the token every iteration. Closes #4
  • deploy_and_scan.yml default instance type bumped t3.micro → t3.medium; the smaller size OOMs reliably during the RHEL 9 SCAP scan. Closes #5
  • deploy_and_scan.yml cleanup now lives in a block:/always: so AWS resources are torn down even when the scan task fails. Closes #6

Verified

  • The publish path, on the real artifact (2026-09-05): disk.img.gz 4.6 GB at ~15 MB/s from the export proxy; sparse raw 60 GiB apparent, 8.7 GiB on disk; qcow2 8.65 GiB, reproduced byte-identically from a second independent export. Registry read-back confirms the digest, all seven labels, one layer, and that anonymous access returns 401 — private.
  • Measured rather than assumed, and it reversed a design decision. The laptop uploads at ~4.97 MB/s (~40 Mbps), not the ~1 MiB/s of #36 — that figure was a cluster-side limit through cdi-uploadproxy, not this uplink. So the publish stays laptop-side like build_cis_containerdisk.yml, and an in-cluster build was rejected: it would have put registry credentials in the cluster to save about twenty minutes.
  • The Windows build completes unattended, and the numbers are now measured rather than guessed (2026-09-05, cluster-kbjvc, clean namespace to a Stopped generalized VM): re-master 2m23s, CDI import 2m20s, install + virtio-win tools + guest agent + sysprep + shutdown 16m43s — 21m26s total. guestOSInfo reported Windows Server 2022 Standard Evaluation, which proves in one field that Setup completed, the right edition came out of install.wim, the machine rebooted into the OS, and FirstLogonCommands ran. The VM ended Stopped and stayed there, so runStrategy: Once is still holding and the #37 OOBE defect did not recur.
  • Two documented timings were wrong and are replaced: "about forty-five minutes" (inherited) and "about thirty" (added in #42). Both predated any completed run.
  • "External Ceph is the bottleneck" was wrong and is retracted. It was computed from three console readings assuming monotonic progress; the progress was not monotonic, it was a boot loop (#50). The storage is fine — the same cluster does the whole build in 21 minutes.
  • Before any of it touched a cluster. xorriso -osirrox on -extract reads Joliet names when Rock Ridge is absent, which is the case on Windows media -- so sources/install.wim and cdboot_noprompt.efi survive the round trip with case and length intact. That was the one thing that could have quietly mangled the ISO; Red Hat sidesteps it with guestfish, which wants devices.kubevirt.io/kvm, while xorriso keeps the pod unprivileged. Confirmed on a synthetic Joliet-only ISO, then end to end inside the pinned image itself: -report_el_torito reports the UEFI entry at /efi/microsoft/boot/efisys.bin with Ldsiz 2624 -- 1,343,488 bytes, the no-prompt image, not the 1,376,256-byte original. The missing-bootloader guard was proved by building an ISO without it and watching the script exit 1.
  • Windows Server 2022 evaluation ISO, from Microsoft's own fwlink (LinkID=2195280, redirecting to software-static.download.prss.microsoft.com): 5,044,094,976 bytes, SHA256 3e4fa6d8507b554856fc9ca6079cc402df11a8b79344871669f0251535255325, volume SSS_X64FREE_EN-US_DV9.
  • sources/install.wim holds 4 images. Windows Server 2022 SERVERSTANDARD is index 2, the Desktop Experience variant — which is what the playbook already defaulted to, now confirmed instead of guessed. Every EDITIONID is ServerStandardEval / ServerDatacenterEval, which is how evaluation media is told apart from licensed media at a glance.
  • The full preflight chain runs green against the real ISO — state, connection, demo-cluster refusal, inputs, checksum match, virtctl — failing only at the first cluster API call, which is as far as it can get without a live cluster.

Lesson

  • CLAUDE.md gains a Verification section. Six defects landed on #24 in one day (#42, #44, #46, #50, #54, and the export token), ansible-lint passed at the production profile through every one of them, and the same few mistakes produced most of them: testing against a fixture built by the tool under test; testing with a different tool than the code uses; not re-running the part that was already known to work; reading a moving indicator once; and removing a manual step without asking what it did on the paths nobody was looking at. Those are rules about how to work in this repo rather than facts about one playbook, so they now live where the next session will read them before starting.
  • The keypress was described as the last manual step — noise to be removed. It was also load-bearing. Nothing recorded what the prompt did on the second boot, so removing it changed behaviour nobody had written down, and the replacement looked healthy for four hours. Before removing a manual step, ask what it does on the paths you are not looking at. Three of this session's four defects (#44, #46, #50) were found by executing, none by lint; #50 needed the same measurement taken twice to be visible at all, because a single reading of a moving progress bar is indistinguishable from progress.
  • A rate computed from an assumption is not a measurement. "~3 minutes per percentage point, so external Ceph is the bottleneck" was derived from three samples assuming monotonic progress. With restarts the inference was void, and it had already been repeated to the user as a reason demos might be slow. The true install speed remains unmeasured.