2019-032-kubernetes security audit dicussion with Jay Beale and Aaron Small
BrakeSec Education Podcast · Bryan Brake
Audio is streamed directly from the publisher (traffic.libsyn.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
Topics:Infosec Campout report
Derbycon Pizza Party (with podcast show!) https://www.eventbrite.com/e/brakesec-pizza-party-at-the-derbycon-mental-health-village-tickets-69219271705
Mental health village at Derbycon
Jay Beale (co-lead for audit) *Bust-a-Kube*
Aaron Small (product mgr at GKE/Google)
Atreides Partners
Trail of Bits
What was the Audit?
How did it come about?
Who were the players?
Kubernetes Working Group
Aaron, Craig, Jay, Joel
Outside vendors:
Atredis: Josh, Nathan Keltner
Trail of Bits: Stefan Edwards, Bobby Tonic , Dominik
Kubernetes Project Leads/Devs
Interviewed devs -- this was much of the info that went into the threat model
Rapid Risk Assessments - let's put the GitHub repository in the show notes
What did it produce?
Vuln Report
Threat Model - https://github.com/kubernetes/community/blob/master/wg-security-audit/findings/Kubernetes%20Threat%20Model.pdf
White Papers
https://github.com/kubernetes/community/tree/master/wg-security-audit/findings
Discuss the results:
Threat model findings
Controls silently fail, leading to a false sense of security
Pod Security Policies, Egress Network Rules
Audit model isn't strong enough for non-repudiation
By default, API server doesn't log user movements through system
TLS Encryption weaknesses
Most components accept cleartext HTTP
Boot strapping to add Kubelets is particularly weak
Multiple components do not check certificates and/or use self-signed certs
HTTPS isn't enforced
Certificates are long-lived, with no revocation capability
Etcd doesn't authenticate connections by default
Controllers all Bundled together
Confused Deputy: b/c lower priv controllers bundled in same binary as higher
Secrets not encrypted at rest by default
Etcd doesn't have signatures on its write-ahead log
DoS attack: you can set anti-affinity on your pods to get nothing else scheduled on their nodes
Port 10255 has an unauthenticated HTTP server for status and health checking
Vulns / Findings (not complete list, but interesting)
Hostpath pod security policy bypass via persistent volumes
TOCTOU when moving PID to manager's group
Improperly patched directory traversal in kubectl cp
Bearer tokens revealed in logs
Lots of MitM risk:
SSH not checking fingerprints: InsecureIgnoreHostKey
gRPC transport seems all set to WithInsecure()
HTTPS connections not checking certs
Some HTTPS connections are unauthenticated
Output encoding on JSON construction
This might lead to further work, as JSON can get written to logs that may be consumed elsewhere.
Non-constant time check on passwords
Lack of re-use / library-ification of code
Who will use these findings and how? Devs, google, bad guys?
Any new audit tools created from this?
Brad geesaman "Hacking and Hardening Kubernetes Clusters by Example [I] - Brad Geesaman, Symantec https://www.youtube.com/watch?v=vTgQLzeBfRU
Aaron Small:
https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster
CNCF: https://www.youtube.com/watch?v=90kZRyPcRZw
Findings:
Scope for testing:
Source code review (what languages did they have to review?)
Golang, shell, ...
Networking (discuss the networking *internal* *external*
Cryptography (TLS, data stores)
AuthN/AuthZ
RBAC (which roles were tested? Just admin/non-admin *best practice is no admin/least priv*)
Secrets
Namespace traversals
Namespace claims
Methodology:
Setup a bunch of environments?
Primarily set up a single environment IIRC
Combination of code audit and active ?fuzzing?
What does one fuzz on a K8s environment?
Tested with latest alpha or production versions?
Version 1.13 or 1.14 - version locked at whatever was current - K8S releases a new version every 3 months, so this is a challenge and means we have to keep auditing.
Tested mulitple different types of k8s implementations?
Tested primarily against kubespray (https://github.com/kubernetes-sigs/kubespray)
Bug Bounty program:
https://github.com/kubernetes/community/blob/master/contributors/guide/bug-bounty.md
Check out our Store on Teepub! https://brakesec.com/store
Join us on our #Slack Channel! Send a request to @brakesec on Twitter or email [email protected]
#Brakesec Store!:https://www.teepublic.com/user/bdspodcast
#Spotify: https://brakesec.com/spotifyBDS
#RSS: https://brakesec.com/BrakesecRSS
#Youtube Channel: http://www.youtube.com/c/BDSPodcast
#iTunes Store Link: https://brakesec.com/BDSiTunes
#Google Play Store: https://brakesec.com/BDS-GooglePlay
Our main site: https://brakesec.com/bdswebsite
#iHeartRadio App: https://brakesec.com/iHeartBrakesec
#SoundCloud: https://brakesec.com/SoundcloudBrakesec
Comments, Questions, Feedback: [email protected]
Support Brakeing Down Security Podcast by using our #Paypal: https://brakesec.com/PaypalBDS OR our #Patreon
https://brakesec.com/BDSPatreon
#Twitter: @brakesec @boettcherpwned @bryanbrake @infosystir
#Player.FM : https://brakesec.com/BDS-PlayerFM
#Stitcher Network: https://brakesec.com/BrakeSecStitcher
#TuneIn Radio App: https://brakesec.com/TuneInBrakesec