Ops AI Showcases

# ManualTestDnsEdge — Analysis
Status: firing · Severity critical
edge: metrics-ui
host: metrics-guest

## 1. What happened?
The DNS + host smoke for metrics-ui is failing. Virtualization looks healthy:
platform guest metrics-guest is running, no hypervisor shutdown logs. Read-only
host triage reports `systemctl is-system-running: running`.

## 2. Likely cause
1. **DNS or edge path:** guest is up and SSH-RO works — smoke (HTTPS at edge)
   still fails. More likely resolution/proxy than power state.
2. **Side finding:** journal shows expired client certificates — relevant for
   agents, not necessarily for the edge smoke.
3. **App down while OS up:** local web UI may be down while the OS is fine.

## 3. Runbook — next steps
1. Check DNS for the edge role.
2. Check upstream health on the guest (listen port / ready endpoint).
3. Re-run smoke — apply stays human + snapshot gate.

## 5. Do not
- Blindly reboot the guest while power state is already running.
- Reboot the hypervisor without hypervisor evidence.
- Change app config before edge vs upstream is separated.

LLM Follow-up
CRIT · ManualTestDnsEdge
edge: metrics-ui / host: metrics-guest

Hypothesis
Guest is up; smoke more likely fails on DNS/edge path or local upstream —
not hypervisor power state. Expired client certificates are a side finding.

Next check
1. DNS for metrics-ui
2. Upstream ready on metrics-guest
3. Re-run smoke — no blind apply

Full report: email [Homelab Analyse]

# ManualTestDnsOps — Analysis
Status: resolved · Severity critical
edge: secrets-ui
host: secrets-guest

## 1. What happened?
DNS/host smoke is green again. Guest is running; no shutdown logs.
Read-only triage still shows recurring snapshot-unit failures and expired
client certificates — smoke ok, ops hygiene not.

## 2. Likely cause
1. **Transient smoke / test trigger:** resolved + guest running contradicts
   a lasting outage.
2. **Background:** snapshot job and cert expiry may fire later alerts but
   do not explain the resolved status.

## 5. Do not
- Debug edge/app config while smoke is green and the guest is stable.
- Permanently ignore snapshot-job failures (data risk).

LLM Follow-up
CRIT · ManualTestDnsOps
edge: secrets-ui / host: secrets-guest

Hypothesis
Smoke is green again. Side findings (snapshot unit, certificates) need follow-up —
but they are not why the alert resolved.

Next check
1. Snapshot unit status/logs
2. Plan certificate rotation
3. No hypervisor reboot without evidence

Full report: email [Homelab Analyse]

# WazuhRule40112 — Analysis
Status: resolved · Severity critical
edge: git-ui
host: git-guest

## 1. What happened?
SIEM reports a successful root SSH login from the ops network. Evidence shows
`Certificate invalid: expired` first, then an accepted public key. Guest is stable;
resources are green. Parallel OIDC init errors (upstream 502) — separate from SSH.

## 2. Likely cause
1. **Expired client certificate + fallback key:** login from ops network after
   a failed certificate attempt.
2. **Planned maintenance login:** source on the management path; alert resolved.
3. **OIDC 502 correlation unclear:** check IdP upstream; do not blindly block SSH.

## 5. Do not
- Reboot the guest — fixes neither certificates nor OIDC 502.
- Block SSH from the ops network without confirming the login was illegitimate.
- Change git app config before the IdP upstream is checked.

LLM Follow-up
CRIT · WazuhRule40112
edge: git-ui / host: git-guest

Hypothesis
Root SSH from ops network after expired client certificate; alert resolved.
OIDC 502 is a parallel side finding on the IdP upstream.

Next check
1. Certificate status / rotation
2. IdP discovery/upstream health
3. No blind block of the ops source

Full report: email [Homelab Analyse]

# Grafana Alloy Upgrade 1.17.1 → 1.18.0

Minor bump with **real breaking changes** in OTel config.
Without config edits Alloy may fail to start or behave unexpectedly.

## 1. Upgrade path
- v1.18.0

## 2. Breaking changes / migrations (cumulative)

| Breaking change | Release | Action |
| --- | --- | --- |
| OTel HTTP timeouts: default from `0s` to upstream limits (`idle/read 1m`, `write 30s`) | v1.18.0 | Set `0s` explicitly if unlimited is required |
| Kafka `resolve_canonical_bootstrap_servers_only` is a no-op | v1.18.0 | Remove the argument; keep bootstrap addresses resolvable |
| Splunk HEC `batcher` block removed | v1.18.0 | Move fields under `sending_queue.batch` |

## 4. Homelab risk
**Medium to high** due to config breaks. Snapshot before apply. Short downtime on restart.
Bad migration → Alloy will not start. No blind image pull — bump SoT first.

# Keycloak Upgrade 26.2 → 26.7.0

Five minor releases at once: security, WebAuthn/Passkeys, deprecations,
template changes. Direct upgrade is possible — but review cumulatively.

## 1. Upgrade path
26.2 → … → 26.7.0 (patches between minors included)

## 2. Breaking changes / migrations (selection)

| Breaking change | Release | Action |
| --- | --- | --- |
| New `locale` field on user-profile registration | 26.3.x | Review custom themes |
| Twitter IdP deprecated | 26.7.0 | Replace/remove social login |
| Fine-grained admin permissions v1 / token exchange v1 deprecated | 26.7.0 | Migrate clients/providers |
| Client-initiated renegotiation off by default | 26.4.x | Re-enable only if required |
| DB migration may take long on large realms (no progress UI) | cumulative | Snapshot/PBS; watch logs |

## 4. Homelab risk
**Medium to high.** Snapshot before apply is mandatory. Planned downtime for DB migration.
Review themes and custom providers in staging before the jump — no autopilot apply.

# Forgejo Upgrade 15.0.4 → 16.0.2

**Major upgrade.** API/config breaks and automatic DB migration on start.

## 1. Upgrade path
- v15.0.5 / v15.0.6 (final 15.x patches)
- v16.0.0 (major, breaking)
- v16.0.1 → v16.0.2 (stabilization)

## 2. Breaking changes / migrations (cumulative)

| Change | Release | Action |
| --- | --- | --- |
| API changes | v16.0.0 | Re-check integrations/scripts against v16 |
| Legacy features removed/deprecated | v16.0.0 | Clean up `app.ini` |
| DB schema migration on start | v16.0.0 | Backup/snapshot; enough disk; watch logs |

## 4. Homelab risk
Short downtime is unavoidable; longer on large repos. **Snapshot before apply.**
Stop → backup → binary/image → start (migration) → smoke. Fail → snapshot rollback.

# fleet-control — OS update summary (pending)
Short: **19 packages** → no security flags, no reboot. Risk **medium**
(score 20), signal on PostgreSQL core.

## 1. What changes? (selection)
| Package | Current | Candidate | Repo |
| --- | --- | --- | --- |
| postgresql / -server / -contrib | 13.23-2.el9_7 | 13.23-3.el9_8 | appstream |
| ruby / ruby-libs | 3.0.7-166.el9_7 | 3.0.7-167.el9_8 | appstream |
| rubygems / bundler / … | revision bump | revision bump | appstream |

Revision only (`-2` → `-3`), no major bumps spotted.

## 2. Risks
| Signal | Detail |
| --- | --- |
| `core_packages` | postgresql-* |
| `security_count` | **0** |
| `risk_level` | medium (20) |

## 3. Recommendation
> Snapshot + apply now — revision updates, no reboot; still use the snapshot gate.

# tutor-guest — OS update summary (applied / history)
**Short:** 114 package updates, 40 security updates, no reboot —
risk level **high** (score 100).

| Packages | Security updates | Reboot needed? |
| --- | --- | --- |
| 114 | 40 | No |

## 1. Changes
1. **Core/runtime libs** — glibc (moderate), glib2/expat/libarchive/libnghttp2 (important)
2. **Auth stack** — krb5-libs, pam/pam-libs
3. **Languages** — python3 / python3-libs security
4. **tzdata** — major bump

## 2. Risks
| Signal | Meaning |
| --- | --- |
| `security_packages=40` | many advisories in the batch |
| `large_batch=114` | higher chance of surprises |
| `severity_important:*` | e.g. expat, glib2, krb5, pam, python3 |

## 3. Recommendation
> **Snapshot + apply** — high security signal, no reboot; take a guest snapshot first.
Apply stays human behind the snapshot gate (no autopilot).

# overlay-gateway — OS update summary (applied / history)
**Short:** 3 packages, 0 confirmed security, no reboot. Risk **low** (score 15).

## 1. What changes?
1. `openssh`, `openssh-clients`, `openssh-server`: 9.9p1-23 → 9.9p1-25
2. Source: baseos
3. No other core/runtime packages in the batch

## 2. Risks
| Signal | Detail |
| --- | --- |
| `core_packages=openssh*` | critical service, but no proven advisories in the manifest |
| `security_count=0` | classified as non-security |
| `risk_level` | low (15) |

## 3. Recommendation
> Snapshot + apply — small batch; history stays visible in the dashboard as last applied.