Container Security: Advanced Best Practices for Production Environments

Published 8 min read
container securityDevSecOpsruntime securityKubernetes security

Container security is the practice of protecting containerized applications across the entire lifecycle: from image build through runtime operation. A secure container environment enforces trusted image sources, minimal base images, least-privilege runtime configurations, network segmentation, and continuous vulnerability monitoring. Teams that implement these controls reduce their attack surface by 60-80% compared to default container configurations.

Why Container Security Requires a Different Approach

Containers change the security model in fundamental ways. Traditional host-based security tools cannot see inside containers. Containers are ephemeral, so forensic analysis must happen in real-time. And shared kernel architecture means a container escape affects every workload on the node.

The container threat landscape includes:

  • Supply chain attacks: Compromised base images or dependencies injected during build
  • Container escape: Exploiting kernel vulnerabilities or misconfigured privileges to reach the host
  • Lateral movement: Using overly permissive network policies to pivot between services
  • Secret exposure: Credentials hardcoded in images, environment variables, or logs
  • Resource abuse: Cryptomining or denial-of-service from compromised containers

Threat VectorLikelihoodImpactPrimary Mitigation

Vulnerable base imageHighHighImage scanning + minimal base
Privileged containerMediumCriticalPod security standards
Network lateral movementHighHighNetwork policies
Exposed secretsHighCriticalSecret management + scanning
Container escapeLowCriticalKernel patching + seccomp
Supply chain compromiseMediumCriticalImage signing + SBOM

Phase 1: Secure the Build Pipeline

Use Minimal Base Images

Every package in your base image is a potential vulnerability. Minimal images like Alpine, Distroless, or scratch-based builds reduce attack surface dramatically.

<h1>Bad: Full Ubuntu base (200+ MB, 100+ CVEs typical)</h1>

FROM ubuntu:22.04

<h1>Better: Alpine base (5 MB, 5-10 CVEs typical)</h1>

FROM alpine:3.19

<h1>Best: Distroless (no shell, no package manager)</h1>

FROM gcr.io/distroless/static-debian12:nonroot

Image size comparison for a typical Go application:

Base ImageSizeKnown CVEs (typical)Shell Access

ubuntu:22.04220 MB80-120Yes
alpine:3.197 MB5-15Yes (ash)
distroless/static2 MB0-3No
scratch0 MB0No

Scan Images in CI

Run vulnerability scanning on every image build. Block deployments that contain critical or high-severity CVEs.

<h1>Trivy scan with severity threshold</h1>

trivy image --severity CRITICAL,HIGH --exit-code 1 \

--ignore-unfixed myapp:${GIT_SHA}

<h1>Grype scan with SBOM generation</h1>

syft myapp:${GIT_SHA} -o spdx-json > sbom.json

grype sbom:sbom.json --fail-on high

Sign and Verify Images

Image signing ensures that only images built by your CI pipeline can be deployed. Use Cosign (from the Sigstore project) for keyless signing integrated with OIDC identity.

<h1>Sign image in CI</h1>

cosign sign --yes myregistry.io/myapp:${GIT_SHA}

<h1>Verify signature before deployment</h1>

cosign verify --certificate-identity [email protected] \

--certificate-oidc-issuer https://token.actions.githubusercontent.com \

myregistry.io/myapp:${GIT_SHA}

For deployment platform options that integrate image verification, review Services Overview.

Phase 2: Harden Runtime Configuration

Run as Non-Root

Running containers as root is the single most common and dangerous misconfiguration. A container running as root that achieves a container escape has root access to the host.

<h1>Kubernetes Pod Security Context</h1>

apiVersion: v1

kind: Pod

spec:

securityContext:

runAsNonRoot: true

runAsUser: 65534

runAsGroup: 65534

fsGroup: 65534

containers:

- name: app

securityContext:

allowPrivilegeEscalation: false

readOnlyRootFilesystem: true

capabilities:

drop: ["ALL"]

Apply Pod Security Standards

Kubernetes Pod Security Standards define three levels: Privileged (unrestricted), Baseline (minimally restrictive), and Restricted (hardened). Production namespaces should enforce Restricted mode.

<h1>Label namespace to enforce restricted pod security</h1>

kubectl label namespace production \

pod-security.kubernetes.io/enforce=restricted \

pod-security.kubernetes.io/warn=restricted \

pod-security.kubernetes.io/audit=restricted

Configure Seccomp and AppArmor Profiles

Seccomp filters restrict the system calls a container can make. The default Docker/containerd seccomp profile blocks approximately 44 of 300+ syscalls. Custom profiles can further restrict based on application needs.

<h1>Apply runtime default seccomp profile</h1>

securityContext:

seccompProfile:

type: RuntimeDefault

For most applications, the RuntimeDefault profile provides good protection without compatibility issues. Only create custom profiles for workloads with very specific syscall requirements.

Phase 3: Network Segmentation

By default, Kubernetes allows all pod-to-pod communication within a cluster. This means a compromised container can reach every other service. Network policies restrict this lateral movement.

Default Deny Policy

Start with a default-deny policy on every namespace, then explicitly allow required communication paths.

<h1>Default deny all ingress and egress</h1>

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: default-deny-all

namespace: production

spec:

podSelector: {}

policyTypes:

- Ingress

- Egress

<h1>Allow specific service-to-service communication</h1>

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: allow-api-to-database

namespace: production

spec:

podSelector:

matchLabels:

app: database

ingress:

- from:

- podSelector:

matchLabels:

app: api-server

ports:

- port: 5432

protocol: TCP

Network policy design principles:

  • Start with default-deny in every namespace
  • Allow only the specific ports and protocols each service needs
  • Separate tenant workloads into different namespaces with cross-namespace deny rules
  • Allow DNS (port 53 to kube-dns) in egress policies or applications will fail to resolve hostnames
  • Test network policies in staging before enforcing in production

For architecture patterns that include network segmentation by design, see Cloud Computing.

Phase 4: Secret Management

Secrets in containers are a critical attack target. Hardcoded credentials in images, environment variables visible in pod specs, and unencrypted Kubernetes secrets are all common weaknesses.

Best practices for container secret management:

  • Never bake secrets into images: Use runtime injection through mounted volumes or init containers
  • Use external secret managers: HashiCorp Vault, AWS Secrets Manager, or GCP Secret Manager with Kubernetes integration (External Secrets Operator)
  • Rotate secrets automatically: Set maximum credential lifetime and automate rotation
  • Encrypt Kubernetes secrets at rest: Enable KMS encryption for etcd
  • Scan for leaked secrets: Run Gitleaks or TruffleHog in CI to prevent credential commits

<h1>External Secrets Operator example</h1>

apiVersion: external-secrets.io/v1beta1

kind: ExternalSecret

metadata:

name: app-secrets

spec:

refreshInterval: 1h

secretStoreRef:

name: vault-backend

kind: ClusterSecretStore

target:

name: app-secrets

data:

- secretKey: database-url

remoteRef:

key: production/database

property: connection_string

Phase 5: Runtime Monitoring and Detection

Static scanning catches known vulnerabilities before deployment. Runtime monitoring catches actual exploitation attempts, anomalous behavior, and zero-day attacks in production.

Runtime Security Tools

ToolTypeDetection MethodOverhead

FalcoOpen sourceSyscall monitoring via eBPFLow
TetragonOpen sourceeBPF-based observabilityVery Low
Sysdig SecureCommercialSyscall + network + fileMedium
Aqua SecurityCommercialFull lifecycle + runtimeMedium

Key runtime events to detect:

  • Shell spawned inside a container (potential intrusion)
  • Unexpected outbound network connections (data exfiltration or C2)
  • File modifications in read-only filesystems (tampering)
  • Privilege escalation attempts
  • Sensitive file access (/etc/shadow, /etc/passwd, service account tokens)

<h1>Example Falco rule: Detect shell in container</h1>

<ul><li>rule: Terminal shell in container</li>

</ul> desc: Detect a shell spawned in a container

condition: >

spawned_process and container and

proc.name in (bash, sh, zsh, ash)

output: >

Shell spawned in container

(user=%user.name container=%container.name image=%container.image.repository)

priority: WARNING

Security Compliance and Auditing

Maintain an audit trail of all security-relevant events: image deployments, policy changes, secret access, and security scan results. This supports compliance requirements (SOC 2, ISO 27001, HIPAA) and incident investigation.

Minimum audit logging:

  • All Kubernetes API server requests (audit policy)
  • Image pull and deployment events
  • Network policy changes
  • Secret access and rotation events
  • Security scan results with timestamps

Review Documentation for implementation guides on security controls and compliance frameworks.

Security Implementation Roadmap

WeekFocus AreaKey Actions

1-2Image securityMinimal base images, CI scanning, severity gates
3-4Runtime hardeningNon-root, read-only FS, Pod Security Standards
5-6Network segmentationDefault-deny policies, service-level allow rules
7-8Secret managementExternal secrets operator, rotation automation
9-10Runtime monitoringFalco/Tetragon deployment, alert tuning
11-12ComplianceAudit logging, policy documentation, first review

For platform options that include security controls by default, compare plans on Pricing. For architecture review, Contact the team.

Frequently Asked Questions

Q: Should we use Distroless or Alpine as our base image?

Use Distroless for production services where you do not need shell access for debugging. Use Alpine for services where you need occasional shell access in staging or during incident investigation. Many teams use Alpine for staging and Distroless for production images of the same service.

Q: How do we handle vulnerability scanning for third-party images we cannot rebuild?

Accept the risk with documented exceptions, or build wrapper images that copy binaries from third-party images into your own minimal base. For databases and infrastructure components, rely on vendor patching SLAs and monitor for CVEs.

Q: What is the performance overhead of runtime security tools like Falco?

eBPF-based tools like Falco and Tetragon typically add less than 1-2% CPU overhead and negligible latency to application requests. The overhead is almost entirely in the monitoring agent, not in the monitored containers.

Q: How do network policies interact with service meshes like Istio?

Kubernetes network policies and service mesh policies operate at different layers. Network policies work at L3/L4 (IP and port). Service meshes work at L7 (HTTP headers, paths). Use both: network policies for coarse segmentation and service mesh policies for fine-grained application-layer authorization.

Q: How often should we rotate container secrets?

Database credentials and API keys should rotate every 30-90 days. TLS certificates should rotate every 90 days or sooner. Service account tokens should be short-lived (hours, not days). Automate all rotation to eliminate the human factor.