Skip to content

P02-T04 - QGA Runner and Golden Image Validation

Objective

Validate a fresh clone of the Windows Server 2025 Desktop Experience image before finalization or publication. QGA provides transport; the reusable validation logic remains PowerShell-first and versioned in /scripts.

Inputs

Key Default
Source template 4000 / TPL-WS2025
Temporary audit VMID 4030
Build VLAN 110
Runner scripts/qga-run.sh
Validation scripts/Validate-WS2025-GoldenImage.ps1
Expected PowerShell 7.6.3

Current validated boundary

The workflow and failure detection were executed on July 18, 2026 against a fresh linked clone of 4000:

  • clone boot completed
  • QGA responded
  • Windows Server 2025 Standard was detected
  • QEMU Guest Agent was running and automatic
  • OpenSSH was stopped and disabled
  • PowerShell 7.6.3 was detected
  • no problem devices, forbidden roles, domain membership, or pending reboot were detected
  • validation correctly failed because the clone inherited persistent route 0.0.0.0/0 through 172.20.110.1

The failed audit was remediated on July 18, 2026. The corrected image was generalized through interactive Sysprep, a full clone completed first boot with IMAGE_STATE_COMPLETE, and the clone reported zero manual IPv4 addresses and zero persistent default routes. It was then published as the protected 4000 / TPL-WS2025 template. Do not weaken this validation if future drift is detected.

Before replacing the old template, HQ-FS01 volumes were promoted to independent ZFS datasets and the running file server was revalidated. This prevented removal of the old base volumes from affecting the operational server.

Prechecks

  • Run on Proxmox host H1 from the repository root.
  • Confirm the source VM is a stopped, protected template.
  • Select an unused temporary VMID.
  • Confirm jq, iconv, and base64 exist on H1.
  • Never boot the source template directly.
SOURCE_VMID=4000
AUDIT_VMID=4030

qm config "$SOURCE_VMID" | grep -E '^(name|template|protection|agent|net0):'
qm status "$AUDIT_VMID" >/dev/null 2>&1 && {
  echo "ERROR: temporary VMID $AUDIT_VMID already exists"
  exit 1
}

Step 1 - Install the QGA transport helper

The helper encodes and transports a local PowerShell file, waits for guest completion, returns guest output, and propagates a non-zero exit code.

Show full script: scripts/qga-run.sh 
#!/usr/bin/env bash

set -Eeuo pipefail

usage() {
    cat <<'EOF'
Usage:
  qga-run.sh <VMID> <PowerShell-script.ps1> [timeout-seconds]

Example:
  qga-run.sh 4000 validate-golden-image.ps1 900
EOF
}

fail() {
    echo "ERROR: $*" >&2
    exit 1
}

[[ $# -ge 2 && $# -le 3 ]] || {
    usage
    exit 2
}

VMID="$1"
SCRIPT_FILE="$2"
TIMEOUT_SECONDS="${3:-900}"
POLL_SECONDS=5

command -v qm >/dev/null 2>&1 || fail "qm was not found."
command -v jq >/dev/null 2>&1 || fail "jq was not found. Install it with: apt install jq"
command -v iconv >/dev/null 2>&1 || fail "iconv was not found."
command -v base64 >/dev/null 2>&1 || fail "base64 was not found."

[[ -f "$SCRIPT_FILE" ]] || fail "PowerShell script not found: $SCRIPT_FILE"
[[ "$TIMEOUT_SECONDS" =~ ^[0-9]+$ ]] || fail "Timeout must be an integer."

qm status "$VMID" >/dev/null 2>&1 || fail "VMID $VMID does not exist."
[[ "$(qm status "$VMID" | awk '{print $2}')" == "running" ]] || fail "VMID $VMID is not running."
qm guest cmd "$VMID" ping >/dev/null 2>&1 || fail "QEMU Guest Agent is not responding for VMID $VMID."

ENCODED_COMMAND="$(iconv -f UTF-8 -t UTF-16LE "$SCRIPT_FILE" | base64 -w 0)"

START_RESPONSE="$(
    qm guest exec "$VMID" -- powershell.exe -NoLogo -NoProfile -NonInteractive \
        -ExecutionPolicy Bypass -EncodedCommand "$ENCODED_COMMAND"
)"

if jq -e '.exited == 1' >/dev/null 2>&1 <<<"$START_RESPONSE"; then
    FINAL_RESPONSE="$START_RESPONSE"
else
    PID="$(jq -r '.pid // empty' <<<"$START_RESPONSE")"
    [[ -n "$PID" ]] || fail "Unable to determine guest execution PID. Response: $START_RESPONSE"
    echo "Guest execution started with PID $PID."
    START_TIME="$(date +%s)"

    while true; do
        FINAL_RESPONSE="$(qm guest exec-status "$VMID" "$PID")"
        jq -e '.exited == 1' >/dev/null 2>&1 <<<"$FINAL_RESPONSE" && break

        CURRENT_TIME="$(date +%s)"
        if (( CURRENT_TIME - START_TIME >= TIMEOUT_SECONDS )); then
            echo "Timeout reached; the guest process may still be running."
            echo "Check it with: qm guest exec-status $VMID $PID"
            exit 124
        fi
        sleep "$POLL_SECONDS"
    done
fi

STDOUT="$(jq -r '."out-data" // empty' <<<"$FINAL_RESPONSE")"
STDERR="$(jq -r '."err-data" // empty' <<<"$FINAL_RESPONSE")"
EXIT_CODE="$(jq -r '.exitcode // 1' <<<"$FINAL_RESPONSE")"

[[ -z "$STDOUT" ]] || printf '%s\n' "$STDOUT"
[[ -z "$STDERR" ]] || printf '%s\n' "$STDERR" >&2
[[ "$EXIT_CODE" -eq 0 ]] || fail "PowerShell script failed with exit code $EXIT_CODE."

echo "QGA execution completed successfully."

install -d -m 700 /root/gntech-qga
install -m 700 scripts/qga-run.sh /root/gntech-qga/qga-run.sh
bash -n /root/gntech-qga/qga-run.sh

For Proxmox VE 9, the health check is qm guest cmd <vmid> ping. Do not substitute qm guest ping <vmid>.

Step 2 - Review the validation logic

The sysadmin may change the expected caption pattern and PowerShell version through parameters. The remaining gates are mandatory for this image standard.

Show full script: scripts/Validate-WS2025-GoldenImage.ps1 
[CmdletBinding()]
param(
    [string]$ExpectedCaptionPattern = '*Windows Server 2025 Standard*',
    [string]$ExpectedPowerShellVersion = '7.6.3'
)

$ErrorActionPreference = 'Stop'
$ProgressPreference = 'SilentlyContinue'
$failures = [System.Collections.Generic.List[string]]::new()

Write-Output '=== Operating system ==='
$operatingSystem = Get-CimInstance Win32_OperatingSystem
Write-Output "Caption: $($operatingSystem.Caption)"
Write-Output "Version: $($operatingSystem.Version)"
Write-Output "Build: $($operatingSystem.BuildNumber)"
if ($operatingSystem.Caption -notlike $ExpectedCaptionPattern) {
    $failures.Add("Unexpected Windows edition: $($operatingSystem.Caption)")
}

Write-Output '=== QEMU Guest Agent ==='
$qemuAgent = Get-Service -Name 'QEMU-GA' -ErrorAction Stop
Write-Output "QEMU-GA: $($qemuAgent.Status) / $($qemuAgent.StartType)"
if ($qemuAgent.Status -ne 'Running') { $failures.Add('QEMU-GA is not running.') }
if ($qemuAgent.StartType -ne 'Automatic') { $failures.Add('QEMU-GA is not automatic.') }

Write-Output '=== OpenSSH ==='
$sshd = Get-Service -Name 'sshd' -ErrorAction SilentlyContinue
if ($sshd) {
    Write-Output "sshd: $($sshd.Status) / $($sshd.StartType)"
    if ($sshd.Status -ne 'Stopped') { $failures.Add('sshd is not stopped.') }
    if ($sshd.StartType -ne 'Disabled') { $failures.Add('sshd is not disabled.') }
} else {
    Write-Output 'sshd: Not installed'
}

Write-Output '=== PowerShell 7 ==='
$pwshPath = 'C:\Program Files\PowerShell\7\pwsh.exe'
if (-not (Test-Path $pwshPath)) {
    $failures.Add('PowerShell 7 executable is missing.')
} else {
    $pwshVersion = & $pwshPath -NoProfile -Command '$PSVersionTable.PSVersion.ToString()'
    Write-Output "PowerShell 7 version: $pwshVersion"
    if ($pwshVersion -ne $ExpectedPowerShellVersion) {
        $failures.Add("Unexpected PowerShell 7 version: $pwshVersion")
    }
}

Write-Output '=== Problem devices ==='
$pnpJob = Start-Job -ScriptBlock {
    @(Get-PnpDevice -PresentOnly | Where-Object Status -ne 'OK')
}
if (-not (Wait-Job -Job $pnpJob -Timeout 60)) {
    Stop-Job -Job $pnpJob
    Remove-Job -Job $pnpJob -Force
    $failures.Add('PnP device enumeration exceeded 60 seconds.')
    $problemDevices = @()
} else {
    $problemDevices = @(Receive-Job -Job $pnpJob)
    Remove-Job -Job $pnpJob -Force
}
if ($problemDevices.Count -gt 0) {
    $problemDevices | Select-Object Class, FriendlyName, Status, Problem |
        Format-Table -AutoSize | Out-String | Write-Output
    $failures.Add("$($problemDevices.Count) problem device(s) detected.")
} else {
    Write-Output 'No problem devices detected.'
}

Write-Output '=== Forbidden roles ==='
$forbiddenRoles = @(
    Get-WindowsFeature AD-Domain-Services, DNS, DHCP, Web-Server, FS-FileServer, Hyper-V |
        Where-Object InstallState -eq 'Installed'
)
if ($forbiddenRoles.Count -gt 0) {
    $forbiddenRoles | Select-Object Name, DisplayName |
        Format-Table -AutoSize | Out-String | Write-Output
    $failures.Add('One or more forbidden roles are installed.')
} else {
    Write-Output 'No forbidden roles detected.'
}

Write-Output '=== Domain membership ==='
$computerSystem = Get-CimInstance Win32_ComputerSystem
Write-Output "PartOfDomain: $($computerSystem.PartOfDomain)"
Write-Output "Domain or workgroup: $($computerSystem.Domain)"
if ($computerSystem.PartOfDomain) { $failures.Add('Golden image clone is joined to a domain.') }

Write-Output '=== Network state ==='
$manualAddresses = @(
    Get-NetIPAddress -AddressFamily IPv4 -ErrorAction SilentlyContinue |
        Where-Object { $_.InterfaceAlias -ne 'Loopback Pseudo-Interface 1' -and $_.PrefixOrigin -eq 'Manual' }
)
$persistentDefaultRoutes = @(
    Get-NetRoute -PolicyStore PersistentStore -AddressFamily IPv4 -DestinationPrefix '0.0.0.0/0' -ErrorAction SilentlyContinue
)
Write-Output "Manual IPv4 addresses: $($manualAddresses.Count)"
Write-Output "Persistent default routes: $($persistentDefaultRoutes.Count)"
if ($manualAddresses.Count -gt 0) { $failures.Add('A manual IPv4 address remains in the image clone.') }
if ($persistentDefaultRoutes.Count -gt 0) { $failures.Add('A persistent default route remains in the image clone.') }

Write-Output '=== Pending reboot ==='
$pendingReboot = (
    (Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending') -or
    (Test-Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired') -or
    (Test-Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations')
)
Write-Output "Pending reboot: $pendingReboot"
if ($pendingReboot) { $failures.Add('A reboot is pending.') }

Write-Output '=== Result ==='
if ($failures.Count -gt 0) {
    $failures | ForEach-Object { Write-Error $_ }
    exit 1
}

Write-Output 'Golden image validation passed.'
exit 0

The script checks:

  • OS edition
  • QEMU Guest Agent state
  • OpenSSH capture state
  • PowerShell version
  • present PnP devices, with a bounded enumeration timeout
  • forbidden server roles
  • domain membership
  • manual IPv4 addresses
  • persistent default routes
  • pending reboot state

Step 3 - Create and boot an isolated audit clone

This modifies only the temporary clone. It does not alter the protected source template.

SOURCE_VMID=4000
AUDIT_VMID=4030

qm clone "$SOURCE_VMID" "$AUDIT_VMID" \
  --name AUDIT-TPL-WS2025 \
  --full 0

qm set "$AUDIT_VMID" \
  --protection 1 \
  --onboot 0 \
  --net0 virtio,bridge=GEILLAN,tag=110,firewall=0

qm start "$AUDIT_VMID"

for attempt in $(seq 1 90); do
  qm guest cmd "$AUDIT_VMID" ping >/dev/null 2>&1 && break
  sleep 5
done

qm guest cmd "$AUDIT_VMID" ping

Step 4 - Execute the validation

AUDIT_VMID=4030

/root/gntech-qga/qga-run.sh \
  "$AUDIT_VMID" \
  scripts/Validate-WS2025-GoldenImage.ps1 \
  300

Optional overrides require creating an operator copy with the desired parameter invocation. The reference script defaults match the approved lab standard.

Success must end with:

Golden image validation passed.
QGA execution completed successfully.

Any reported failure blocks publication. Record the exact failure and correct the source build through P02-T05; do not repair only the audit clone and call the template validated.

Step 5 - Inspect network drift explicitly

Run this when the validation reports a manual address or persistent route:

AUDIT_VMID=4030

qm guest exec "$AUDIT_VMID" -- powershell.exe -NoProfile -Command "
Get-NetIPAddress -PolicyStore PersistentStore -AddressFamily IPv4 -ErrorAction SilentlyContinue |
  Where-Object InterfaceAlias -ne 'Loopback Pseudo-Interface 1' |
  Select-Object InterfaceAlias,IPAddress,PrefixLength,PrefixOrigin;
Get-NetRoute -PolicyStore PersistentStore -AddressFamily IPv4 -DestinationPrefix '0.0.0.0/0' -ErrorAction SilentlyContinue |
  Select-Object InterfaceAlias,DestinationPrefix,NextHop,RouteMetric
"

The approved result contains no manual non-loopback IPv4 address and no persistent default route.

Step 6 - Remove the audit clone

Review the evidence before cleanup, then remove only the temporary VMID:

AUDIT_VMID=4030

qm stop "$AUDIT_VMID" --timeout 60 || true
qm set "$AUDIT_VMID" --protection 0
qm destroy "$AUDIT_VMID" --purge 1
qm status "$AUDIT_VMID" 2>&1 || true

Acceptance criteria

  • Source remains stopped, protected, and unmodified.
  • Audit clone boots without interactive recovery.
  • QGA responds through the documented Proxmox VE 9 command.
  • The reference PowerShell script returns exit code 0.
  • No forbidden roles or domain membership exist.
  • No manual build IPv4 state or persistent default route remains.
  • No pending reboot exists.
  • Evidence is retained before deleting the temporary clone.

Evidence

  • qm config for the source template and audit clone
  • complete qga-run.sh output
  • explicit persistent network-state output
  • audit clone cleanup result

Rollback

  • Destroy the temporary clone if it cannot reach a clean QGA state.
  • Never boot or modify the protected source template to troubleshoot the audit.
  • Rebuild or recapture the source through the approved image workflow when a clone exposes inherited drift.