Greg Castle

it was impossible to know what was coming unless you paid even a little bit of attention

@evelyndouek.bsky.social @stamos.org I keep refreshing the moderated content podcast page....So much to cover! moderated-content.simplecast.com

A multi year epic! Thanks for seeing it through.

Volts is my favorite of all the clean energy podcasts, and I've listened to quite a few.

Getting more out of the grid we've already built The US desperately needs more electricity transmission, but new lines take forever to permit & build. Happily, there is a set of technologies that can help us get as much as 40-50% more capacity out o...

There are many cheap and proven ways to get more out of the grid we have. But the utilities only make money out of big capital projects. David Roberts has me convinced that the biggest clean energy problem we have is how utilities make money. www.volts.wtf/p/getting-mo...

This story is wild

So sorry to hear this 😢

Analyzing Volatile Memory on a Google Kubernetes Engine Node TL:DR At Spotify, we run containerized workloads in production across our entire organization in five regions where our main production workloads are in Google Kubernetes Engine (GKE) on Google Cloud ...

Spotify runs most of their production workloads on GKE and wrote this cool post on memory #forensics. They suck out memory through a privileged pod via kcore and send it to #volatility for analysis.

Love it!

After a brief hint of summer weather can confirm the PNW forests are back to their resting drip face.

Least Privilege Containers: Keeping a Bad Day from Getting Worse - Greg Castle \u0026 Vinayak Goyal Least Privilege Containers: Keeping a Bad Day from Getting Worse - Greg Castle \u0026 Vinayak Goyal, Google “Don’t run containers as root”. The K8s security community has been saying this for years. There’s tools that can detect these types of misconfigurations. But detection, and knowing you have a problem, is just the start of the journey. How do you actually fix it? What can you do if those permissions are required for the container to work? We’ve run multiple de-privileging efforts for production containers. In 2020 we focused on converting containers from running as root to running as unprivileged users. In 2021 we moved containers to minimal distroless images. For some containers the solution was as simple as removing unused permissions. But sometimes we needed to do something more drastic, like charge the design of the container to segment out powerful permissions, or split functionality out into initContainers. We’ll share how we approached these tasks, what we learned working through problems with container owners, and describe how we put checks in place to prevent new privileged containers from appearing in the future.

Talk recording: https://youtu.be/uouH9fsWVIE.

Slides: https://static.sched.com/hosted_files/cloudnativesecurityconna23/23/container_patching_cnscon_2023_castle_panther.pdf (8/8)

Converting stateless containers to non-root is relatively easy, infrastructure-heavy containers are harder.

But help is coming with Kubernetes “hostUsers” feature. Lets you run root inside the container, but the process is unprivileged on the host. (7/8)

Kubernetes should configure the ambient capability set · Issue #56374 · kubernetes/kubernetes /kind bug What happened: The following takes place on a k8s 1.8.2 cluster. I have a Docker container image that wants to listen on :80, and specifies a non-root USER. To get this running, in my pod...

Challenge 3: root capability management e.g. NET_BIND_SERVICE.

Surprise 2: adding root capabilities to non-root users via securityContext should work, but doesn’t.

Workaround: use setcap to set filesystem capabilities in Dockerfile. (6/8)

https://github.com/kubernetes/kubernetes/issues/56374

Challenge 2: host files that need to be accessed by more than one container and therefore user.

Solution: set group ownership on the host then assign groups to containers using supplementalGroups in securityContext. (5/8)

Challenge 1: containers that need access to files/sockets.

Surprise 1: fsGroup in securityContext can’t set hostpath ownership.

Workaround: set ownership in an init container as root. (4/8)

Migrating one container is not that hard, migrating a whole product or a whole company requires a strategy.

We blocked new rooty containers at code submission time, and burned through the existing ones case-by-case. (3/8)

Why non-root: so many live breakout vulns.

We found and fixed 17 exploitable breakouts in 2022 via our kCTF bugbounty program, and paid out $1.3M.

Non-root containers make breakout harder and misconfiguration less disastrous. (2/8)
https://bit.ly/45aN7ZT

Least Privilege Containers: Keeping a Bad Day from Getting Worse - Greg Castle \u0026 Vinayak Goyal Least Privilege Containers: Keeping a Bad Day from Getting Worse - Greg Castle \u0026 Vinayak Goyal, Google “Don’t run containers as root”. The K8s security community has been saying this for years. There’s tools that can detect these types of misconfigurations. But detection, and knowing you have a problem, is just the start of the journey. How do you actually fix it? What can you do if those permissions are required for the container to work? We’ve run multiple de-privileging efforts for production containers. In 2020 we focused on converting containers from running as root to running as unprivileged users. In 2021 we moved containers to minimal distroless images. For some containers the solution was as simple as removing unused permissions. But sometimes we needed to do something more drastic, like charge the design of the container to segment out powerful permissions, or split functionality out into initContainers. We’ll share how we approached these tasks, what we learned working through problems with container owners, and describe how we put checks in place to prevent new privileged containers from appearing in the future.

“Don’t run containers as root”: we’ve been saying this for a long time. Is it working?

No.

@vinayaklovespizza and I gave a talk at #KubeCon EU about our journey converting GKE system containers to non-root that explains why.

Here’s a summary… (1/8)

https://youtu.be/uouH9fsWVIE

Thanks Rory! If you were looking for a firehose to drink from, here it is.

Love it. Similar vibe: a co-worker set a SQL injection string as his official job title. Spent years having "but your title shouldn't have those characters" discussions with various internal tool authors.

I love this. And that's how bluesky learned about https://en.m.wikipedia.org/wiki/Well-known_URI

C. For loose leaf I have a single-cup thingy like this: https://a.co/d/hOhmDbA. It's easier to wash it out immediately so I wait for steep. Since I'm in the habit of using steep time as thinking time I do the same for tea bags.

Interesting! Sounds like the plan is to run CAs that only issue short lived certs and essentially opt out of revocation?

#introduction My day job is security of all things Kubernetes and containers. But if you want to have a real conversation let's talk about mountains and bikes. Or XC skiing. Or about how great boot dryers are if you live in the PNW.