[Umbrella] Artifact Vulnerability Scanning and Triage Policy

[PushkarJ]

Goal: Implement automated scanning capabilities that are tool agnostic for identifying vulnerabilities in Kubernetes related artifacts, followed by a documented private triage process to resolve the identified vulnerability, with a programmable way for Kubernetes users to consume this vulnerability information.

Background

Over the years, multiple different community members in Security Response Committee (formerly PSC), SIG Release, Architecture, Security, Auth have contributed to several standalone efforts related to vulnerability management for https://github.com/kubernetes/kubernetes. We have made tremendous progress but there are still some opportunities to improve :-)

Scope

This issue is created to act as a single place to find current state of the work, in progress and planned work that fall in the overall theme of vulnerability management of Kubernetes artifacts. In scope artifacts include but are not limited to build time dependencies and container images. Adding any missing issues or related work as a comment is encouraged :-)

Artifact Vulnerability Scanning

Build time Dependencies

• Implement automated scanning with prow and test-grid for k/k HEAD (main branch) (POC for vulnerability scanning using snyk kubernetes#101528)
• Parsing improvements (snyk-periodic: Fix style and jq nits test-infra#22756)
• Ensure scan fails when a vulnerability is found (Update scanning to triage privately test-infra#22833)
• Send alerts for security tooling to group under kubernetes.io test-infra#23112
• Scan kubernetes/kubernetes with govulncheck #95

Container Images

• Scanning Container images in a Kubernetes Release with Snyk #4
• Explore and identify scanners that can detect vulnerabilities in distroless++ images
• Explore using SBOM to programmatically get a list of images in each kubernetes release (Generate the first SBOM protoype from the Kubernetes release process release#2095)
• Implement automated scanning with prow and test-grid for k/k HEAD
• Ensure scan fails when a vulnerability is found

Ongoing Maintenance

• Revert to registry.k8s.io and upgrade snyk test-infra#27309
• Force Redirect Registry URL because of redirect failures test-infra#26777
• Fix failing job using GH issue lookup test-infra#24857
• Removes printing of snyk scan results test-infra#24446

Triage Policy Definition and Implementation

• Solicit feedback from SRC and SIG Security Co-chairs for Triage and Resolution policy (Private Triage Process and Vulnerability Scanning for Build Time Dependencies community#5853)
• Create a new group for private triage (Request new group for sig-security-tooling k8s.io#2342)
• Drive an end to end triage for an identified vulnerability to resolution
• Update the triage and resolution policy based on end to end experience
• Create a periodically auto-refreshing list of fixed CVEs #1
• Define and Measure mean time to triage, false positive rate for each identified vulnerability
• Create a rotating triage role for taking action on identified vulnerability

Related Issues and PRs

• Original issue to track kubernetes build time dependencies: Track CVEs for kubernetes dependencies... community#2992
• CVE RSS feed broken: Kubernetes Security and Disclosure Information website#29142
• Request for base image patching: Patching for base images release#1833
• Examples of CVE fixes:
  • Build time dependency bumps: CVE-2023-27561 CVE-2023-25809 CVE-2023-28642: Bump runc v1.1.4 -> v1.1.5 kubernetes#117094
  • Debian base image bumps: Update the Debian images to pick up CVE fixes in the base images kubernetes#102302
  • Distroless base image bumps: Update image base to gcr.io/distroless/base-debian10:latest kubernetes#100566
  • Build time dependency bumps: Moving to a fork for jwt-go with a CVE fix (transient dependency) kubernetes#100401
• CNCF TAG Security discussion: Suggestion/Recommendation for CVE Announcement enhancements cncf/tag-security#170

/sig security release architecture auth
/area config testing code-organization dependency release-eng release-eng/security
/committee product-security
/kind feature

[k8s-ci-robot]

@PushkarJ: The label(s) area/config, area/testing, area/release-eng, area/release-eng/security cannot be applied, because the repository doesn't have them.

In response to this:

Goal: Implement automated scanning capabilities that are tool agnostic for identifying vulnerabilities in Kubernetes related artifacts, followed by a documented private triage process to resolve the identified vulnerability, with a programmable way for Kubernetes users to consume this vulnerability information.

Background

Over the years, multiple different community members in Security Response Committee (formerly PSC), SIG Release, Architecture, Security, Auth have contributed to several standalone efforts related to vulnerability management for github.com/kubernetes/kubernetes. We have made tremendous progress but there are still some opportunities to improve :-)

Scope

This issue is created to act as a single place to find current state of the work, in progress and planned work that fall in the overall theme of vulnerability management of Kubernetes artifacts. In scope artifacts include but are not limited to build time dependencies and container images. Adding any missing issues or related work as a comment is encouraged :-)

Artifact Vulnerability Scanning

Build time Dependencies

• Implement automated scanning with prow and test-grid for k/k HEAD (main branch) (POC for vulnerability scanning using snyk kubernetes#101528)
• Parsing improvements (snyk-periodic: Fix style and jq nits test-infra#22756)
• Ensure scan fails when a vulnerability is found (Update scanning to triage privately test-infra#22833)
• Update alerting to send alerts to group under kubernetes.io
• Evaluate go vuln-db tool as an additional scanning tool

Container Images

• Identify a list of container images managed by github.com/kubernetes/release
• Explore and identify scanners that can detect vulnerabilities in distroless++ images
• Explore using SBOM to programmatically get a list of images in each kubernetes release (Generate the first SBOM protoype from the Kubernetes release process release#2095)
• Implement automated scanning with prow and test-grid for k/k HEAD
• Ensure scan fails when a vulnerability is found

Triage Policy Definition and Implementation

• Draft Triage and Resolution Policy and solicit feedback from SRC and SIG Security Co-chairs (Private Triage Process and Vulnerability Scanning for Build Time Dependencies community#5853)
• Create a new group for private triage (Request new group for sig-security-tooling k8s.io#2342)
• Drive an end to end triage for an identified vulnerability to resolution
• Update the triage and resolution policy based on end to end experience
• Create a periodically auto-refreshing list of fixed CVEs and publish to github.com/kubernetes/website
• Define and Measure mean time to triage, false positive rate for each identified vulnerability
• Create a rotating triage role for taking action on identified vulnerability

Related Issues and PRs

• Original issue to track kubernetes build time dependencies: Track CVEs for kubernetes dependencies... community#2992
• CVE RSS feed broken: Kubernetes Security and Disclosure Information website#29142
• Examples of CVE fixes:
• Debian base image bumps: Update the Debian images to pick up CVE fixes in the base images kubernetes#102302
• Distroless base image bumps: Update image base to gcr.io/distroless/base-debian10:latest kubernetes#100566
• Build time dependency bumps: Moving to a fork for jwt-go with a CVE fix (transient dependency) kubernetes#100401
• CNCF TAG Security discussion: Suggestion/Recommendation for CVE Announcement enhancements cncf/tag-security#170

/sig security release architecture auth
/area config testing code-organization dependency release-eng release-eng/security
/committee product-security
/kind feature

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[PushkarJ]

cc @kubernetes/sig-security-leads / @tabbysable / @IanColdwater , @dims , @navidshaikh , @puerco , @justaugustus

[PushkarJ]

/assign @PushkarJ

[puerco]

In addition to the SBOM item, you can find the images we produce in the SBOM to close this one:

Identify a list of container images managed by github.com/kubernetes/release

I can help with that!

[tengqm]

/cc

[vinayakankugoyal]

I'd love to help with. Evaluate go vuln-db tool as an additional scanning tool and any of the Container Images - tasks.
I remember use talking about automation for updating k8s with latest debian-[base, iptables] images. Are we tracking that in this? I'd love to help with that.

[PushkarJ]

/transfer sig-security