https://www.millabs.net/blog/commercial-app-hits-hardened-baseline/

Your application runs in Docker. It has a Helm chart. It deploys to commercial Kubernetes without issues. You've been told the next step is getting it into a DoW enclave — Platform One, Game Warden, a program office environment — and it shouldn't be that different.

It is.

I recently deployed an open-source project management application — OpenProject, a well-built Ruby on Rails platform with Docker support, Kubernetes documentation, and a Helm chart — against my private Big Bang instance. The same hardened Kubernetes substrate underlying most DoW program-office platforms. Istio service mesh, Kyverno policy enforcement, cosign image signing, CycloneDX SBOM attestation.

Before deploying, I modeled the expected findings based on architecture analysis. The model predicted 27–32 discrete findings: root containers, missing managed services, service mesh conflicts, image provenance failures.

The deployment produced 10 actual findings. But here's the thing — the model was wrong in both directions. It predicted problems that didn't exist and missed problems that only deployment reveals. That's the point: architecture analysis gets you in the ballpark. Only deployment gets you the answer.

What the model got wrong

Five remediation work items were scoped, and in some cases would have been budgeted, based on assumptions the deployment refuted:

What would have been done Why it was unnecessary Effort avoided
Rebuild image to run non-root Already non-root, uid 1000 2–3 days
Rebuild Hocuspocus on hardened base Already hardened, 0 criticals, 0 highs 1–2 days
Containerize PostgreSQL Platform provides managed PostgreSQL 17 3–5 days
Build cache sidecar Native Redis-compatible support, configuration only 1–2 days
Design TLS passthrough for mesh Application does not terminate TLS 0.5–1 day

8–13 days of engineering effort$11,000–$18,000 at GSA Schedule rates for mid-senior DevSecOps engineers (~$175/hr). And that's the government rate. Cleared engineers on the commercial market bill $200–$325/hr, pushing that wasted work to $16,000–$34,000. Either way, it would have been scoped, budgeted, and in some cases executed before anyone discovered it was unnecessary.

This is what happens when estimates are built on architecture reviews instead of deployment data. The assumptions aren't unreasonable — most commercial applications do run as root, most enclaves don't offer managed PostgreSQL. But this application didn't, and this platform did. Without deploying, you can't know.

What the model missed

Seven findings surfaced only by deploying. None appear in documentation reviews, architecture assessments, or vendor deployment guides.

Finding 1 — Secrets silently absent at runtime. The entrypoint does not load platform-injected secrets. Both sides report success independently — the gap is between them. The symptom is a database connection error, not a secrets error. You debug the wrong thing.

Finding 2 — Ruby rejects the default temporary directory. Ruby's Dir.tmpdir skips directories with incompatible permissions. Rails fails at boot with no indication of the cause. You're reading stack traces that point nowhere useful.

Finding 3 — Collaborative editor on an undocumented port. Hocuspocus listens on port 1234, not the documented 3000. Health checks kill a healthy process. The pod restarts in a loop and you think the application is crashing.

Finding 4 — Database migrations not in deployment manifests. Approximately 1,500 migrations required. The application will not start without them. The vendor's deployment documentation assumes you run them manually — on a hardened platform, there's no shell access to run them from.

Finding 5 — Cache key prefix mismatch. Every cache read/write denied. The application's default cache prefix is correct for the application but outside the scope of the provisioned credentials.

Finding 6 — Real-time channel prefix hardcoded. ActionCable's channel prefix is a literal string in application config — separate from the cache prefix. Teams fix Finding 5 and miss this one. Pub/sub stays broken.

Finding 7 — Storage backend crashes at boot without credentials. The S3 backend initializes eagerly, not lazily. The application crashes before reaching the code that uses storage. You see a crash on startup and assume the application is fundamentally broken.

Each of these findings surfaces serially. Each fix reveals the next failure. Without the lab, each would be a separate submission to the target platform's review cadence — weeks to months of serial iteration.

The real blocker: 685 CVEs and the hardened base image alternative

The model predicted "200+ CVEs." The actual scan found 670 CVEs in the Rails image — 28 critical, 96 high — driven entirely by the Debian trixie base and the Ruby/Node.js toolchain, not by application code.

The production vulnerability gate blocks any image with critical CVEs. No override.

The Hocuspocus collaborative editing service — which the model predicted would need a full image rebuild — was already built on a distroless Node.js base with 0 criticals and 0 highs. It was production-admissible as shipped.

The solution: rebuild the Rails image on Chainguard wolfi. The result: 0 CVEs across both images. 670 → 0 for Rails. 15 → 0 for Hocuspocus. Every CVE was in the base image and toolchain — not in application code.

Without the rebuild, each of those 685 CVEs would require a security impact statement for the authorization package — approximately 1 hour each for initial writing, then re-validation on every release. Over a 5-year ATO lifecycle with monthly releases, that's an estimated 5,365 hours of CVE impact statement work — roughly $940,000 at GSA Schedule rates ($175/hr), and north of $1.3M at commercial cleared rates. With 0 residual CVEs, that entire obligation is eliminated.

What deploying commercial software to a hardened baseline proved

All workloads passed 39 Kyverno admission policies, cosign signature verification, and CycloneDX SBOM attestation. The application is serving on the hardened platform — Rails web server, background worker, collaborative editor, database migrations complete.

No application code changes were required for deployment. Two bounded code changes were identified for FIPS 140 compliance (password hashing from bcrypt to PBKDF2-HMAC-SHA256, attachment checksum from MD5 to SHA-256) — proactive, not deployment blockers.

The fit study delivered working code: hardened Dockerfiles, FIPS code changes, a CI/CD pipeline, operational manifests (seed Job, break-glass credential reset Job), and a configuration playbook for all 7 findings.

The gap between the model and reality

The model overestimated infrastructure problems and underestimated image vulnerability severity. It predicted work that didn't need to happen and missed work that only deployment reveals.

This is not a criticism of architecture analysis — the model's assumptions were reasonable. It's a demonstration of why deployment data replaces assumptions:

  • 8–13 days of unnecessary work avoided — $11,000–$18,000 at GSA rates, $16,000–$34,000 at commercial cleared rates
  • 7 findings discovered — problems invisible without deployment, ~$10,000 each to discover serially on a live platform
  • 685 CVEs quantified — worse than modeled, but solvable with a base image rebuild
  • ~800–850 hours of engineering eliminated — ~$148,000 one-time at GSA rates
  • ~5,365 hours of CVE maintenance eliminated — ~$940,000 over a 5-year ATO lifecycle

The fit study cost surfaces the question. The full study answers it with working code — not a second round of recommendations.


Robert Burckner is the founder of Millabs Corporation, a Service-Disabled Veteran-Owned Small Business that deploys commercial software products into hardened DoW enclaves. He previously served as Division Chief at the Space Warfighting Analysis Center (USSF/NRO) and Cybersecurity Chief at the Air Force Lifecycle Management Center.

If your product needs to deploy to an authorized environment and you want to know what the gap looks like, contact Millabs.

Share this post
Email LinkedIn
GET IN TOUCH WITH US

Find out what the gap between your product and an authorized environment actually looks like.