The DevSecOps Talks Podcast

Our guest speaker is Anton Babenko he is DevSecOps Talks podcast fan, AWS Community Hero, Terraform fanatic, HashiCorp Ambassador and a prolific open source contributor. After listening to episode #9 Terraform in CI and #1 Infrastructure as code, Anton decided that enough is enough and volunteered to give his point of view on Terragrunt since he though that we are missing a few important points. In this episode, we are discussing the use cases of Terragrunt, a wrapper around Terraform for working with multiple environment and modules.

How do you start to implement a CI pipeline when dealing with infrastructure as code implemented via Terraform? What are the security concerns when the credentials to the whole kingdom are used in an automated process? In this episode, we discuss the various security and feasibility aspects of using Terraform in a CI pipeline. We start the episode by catching up with what we’ve been working on. Feel free to skip to 11:52 if you want to go directly to the topic. Having an automated process to deploy and manage infrastructure has advantages such as fast feedback and collaboration. The code for the infrastructure is treated like an application that is versioned, tested, and deployed. Show notes are available at https://devsecops.fm/episodes/terraform-in-ci/

Andrey tells us the story of how DevOps came into existence and took over the market. We discuss the marketing around it, its relationship with DevSecOps. We tried to shed a light on what is marketing strategy versus implementing DevOps in an organization. We also compared DevOps to SRE (Site Reliability Engineering)

In this episode, Mattias, Julien, and Andrey share tips and tricks on how to stay on top of what is going on in the industry, resources they use for continuous learning. Make sure to visit devsecops.fm to check out show notes that contain references to resources mentioned during discussion and more

This time Johan Abildskov, a Senior Consultant with Praqma/Eficode, joins us to talk about SemVer (Semantic Versioning), and we finally get to hear what Julien has to say about it. We get to explore different options regarding versioning and how it helps humans communicate. At the end of the podcast, everyone gets to share their approach and recommendations for versioning things.

We had a couple of possible topics for this episode but before getting started with them we decided to discuss what technological problems we were solving during the last two weeks. Well, turns out there was quite a lot to discuss. Tune in for tips on ssh session logging on the ssh server, preventing downloads from AWS S3 even if you got read access, credentials in Git repository 🤦, why you should (or should not) do K8S and more. Summary In this free-form early episode of DevSecOps Talks, a casual "what have you been up to" catch-up turns into a sharp exchange on the gap between security in theory and security in practice. One host discovers plaintext service account keys, database passwords, and a production SSH tunnel all committed straight into a Git repository — and the team walks through how to unwind that without breaking delivery. Julien Bisconti argues that security tooling is fundamentally failing developers because it is too hard to use under real delivery pressure. The episode also delivers strong opinions on why teams should not default to Kubernetes, the hidden complexity of S3 encryption with KMS keys, and why Google's BeyondCorp model makes VPNs look like a relic. Key Topics SSH session logging, bastion hosts, and compliance visibility The episode opens with a deep dive into SSH session logging for bastion hosts in AWS. One of the hosts explains how AWS Systems Manager Session Manager can be used to access instances without VPNs or direct inbound connectivity — the SSM agent on each instance calls home to AWS, and AWS proxies the connection back. That model is attractive for hybrid and on-prem environments because it removes networking complexity around NAT, port forwarding, and VPN setup. It also provides session logging, IAM-based access control, and command output recording. But the drawbacks surface quickly. Session Manager logs users in as a generic SSM agent user with /usr/bin as the working directory. Documentation is sparse, and Bash is launched in shell mode to support color interpretation, which pollutes session logs with escape characters. A bigger concern is that access control rests entirely on IAM credentials — in an environment with fully dynamic, short-lived credentials that is manageable, but it becomes risky anywhere static keys exist. The host describes trying to map Session Manager logins to individual users, only to find that it requires static IAM identities with specially named tags containing usernames — a non-starter for environments where everything is dynamic. That leads into alternative approaches. An AWS blog post describes forcing SSH connections through the Unix script utility to record sessions, then uploading logs to S3. But even that is fragile: logs are owned by the user, so technically the user can delete or overwrite them. A more robust path is tlog, a terminal I/O logger that writes session data in JSON format to the systemd journal, where it cannot be easily tampered with. From there, the CloudWatch agent can export journal data to S3 for long-term storage. The broader point is that command logging sounds simple in compliance conversations, but in practice it becomes a deep rabbit hole full of bypasses, noise, and design tradeoffs. Monitoring user activity without drowning in logs The hosts compare notes on monitoring shell activity. One host mentions using auditd to track user actions on bastion hosts in a previous environment, but the log volume was overwhelming — even Elasticsearch struggled to keep up with the ingestion rate. That sparks a discussion around anomaly detection and heuristics. The real challenge is not collecting logs but determining what is unusual and worth investigating. Failed SSH login alerts are mentioned as a useful signal, though another host pushes back: "Should you have SSH with the password at all? You should have a key." The point stands — without careful tuning, even sensible alerts generate noise faster than teams can act on them. The exchange captures a recurring DevSecOps reality: collecting telemetry is the easy part; turning it into something actionable is where most teams get stuck. S3 bucket security, public access controls, and KMS encryption surprises The conversation shifts to AWS S3 security. Public buckets remain a common source of breaches, but AWS now offers S3 Block Public Access — account- and bucket-level settings that prevent public access regardless of individual object ACLs. In Terraform, this is a dedicated resource block. The more nuanced insight is about encryption. The host explains the difference between S3 server-side encryption with the default AWS-managed key (SSE-S3) and encryption with a customer-managed KMS key (SSE-KMS). With SSE-S3, S3 decrypts objects transparently for any client with read access to the bucket. With a customer-managed KMS key, S3 cannot decrypt the object unless the requester also has kms:Decrypt permission on that specific key. This became a real problem in a cross-accoun

In this episode Mattias is trying to convince that running docker in k8s is more security then VM. Did he success ? listen and find out. Summary Mattias makes a bold claim: Docker containers are more secure than virtual machines. Andrey and Julien push back hard — and by the end, the three hosts explicitly agree to disagree. Along the way, they dig into why container breakouts are harder than people assume, how Lambda micro VMs can be exploited through warm TMP folders, why "containers do not contain" without extra kernel controls, and whether good monitoring matters more for security than any isolation technology. Recorded during COVID-19 lockdowns in 2020, the debate captures a moment when the container-vs-VM argument was far from settled. Key Topics Docker vs. VM security: technology vs. ways of working Mattias opens the main debate by arguing that Docker containers are more secure than VMs in practice. His reasoning: containers are smaller, more focused, and more ephemeral than traditional virtual machines, which reduces attack surface. In a typical VM, you find mail agents, host-based intrusion detection, syslog, monitoring tools, and other services all coexisting with the application. In a container, you ideally run only the application itself. Andrey pushes back immediately. He argues Mattias is comparing operational models, not technology. A well-run VM can also be immutable and minimal — you redeploy from a new image the same way you replace a container. Likewise, a badly built container can be long-lived, bloated, and full of unnecessary tools. Andrey has seen enterprises that run containers for months, SSH into them, and treat them like VMs. Mattias concedes the point but maintains that the standard approach differs: VMs are typically kept running longer with more tools, while the standard approach for containers in Kubernetes is to rotate them and keep a smaller footprint. Andrey counters that most Docker images run as root by default, giving attackers more privilege than they would have on a typical VM where processes run under limited service accounts. This is one of the sharpest exchanges in the episode — better tooling does not fix insecure defaults. The hosts eventually agree that both technologies can be secured well, but do not reach consensus on which is easier. Andrey summarizes it cleanly: containers make it "a little bit easier" to do the right thing because they narrow the focus to the application rather than the entire operating system, but it is absolutely possible to reach the same security level with VMs. Why container breakout is not as trivial as people imply Mattias challenges the common assumption that containers are unsafe because "you can break out of them." He points out that every container breakout CVE he has reviewed requires significant preconditions: either running an attacker-controlled image or running in privileged mode. You cannot take a standard Ubuntu container image, run a single command, and escape. The threat is real but requires chained attacks, not a single exploit. Julien and Andrey accept the premise but note that the comparison matters. VM isolation is fundamentally stronger at the hypervisor level. Container breakout may be hard, but it is architecturally easier than VM escape. The discussion reframes the question: runtime security is less about one isolation boundary and more about how many obstacles an attacker must pass through. Micro VMs, Firecracker, and Lambda attack vectors Andrey brings up an important middle ground between containers and VMs: micro VMs. AWS Lambda runs on Firecracker, an open-source micro VM monitor. Lambdas are ephemeral, have read-only file systems, minimal tooling, and no access to source code or settings — making them quite secure by design. But Andrey describes a real attack path researchers have demonstrated. The /tmp directory in Lambda is writable. If an attacker exploits a vulnerability to get code execution within the Lambda, and the Lambda is kept warm (invoked within 15 minutes so it stays in memory), the /tmp folder persists between invocations. An attacker can download tools incrementally across multiple Lambda runs, building up capability over time. From there, they can explore IAM permissions, exfiltrate data by encoding it in resource tags, or even override the Lambda function itself. The point is that even well-designed ephemeral environments have attack paths when defenders are not paying attention. Security depends on hardening and monitoring, not just on the isolation primitive. Containers do not contain: AppArmor, Seccomp, and policy controls Julien delivers the episode's sharpest technical point: "Containers do not contain." They are primarily Linux namespace isolation and need additional kernel controls — AppArmor profiles and Seccomp filters — to properly restrict what applications can do at runtime. Without those extra layers, a container running as root is effectively root on the host machine, and a c

Your docker images and build are be coming the base for our platform. But are they secure? In this episode we talk about how you can secure your docker images. Summary In this early DevSecOps Talks episode, Mattias, Andrey, and Julien dig into Docker security as a supply chain problem — and quickly dismantle the assumption that a signed container means you know what is inside. Julien pushes back sharply: signing only gives a "semantic guarantee" that an image is what it claims to be, not that it is safe. Mattias argues that containers were designed to be convenient, not secure by default, while Andrey points out that containerization has fundamentally changed the patching game — once the OS, web server, and application are packaged together, every security fix becomes a rebuild-and-redeploy exercise. The hosts make the case for layered scanning, slim runtime images, multi-stage builds, and continuous rebuilding as the practical path to running containers safely in production. Key Topics Container images vs. running containers The conversation starts by separating two distinct parts of container security: the image and the running container. Mattias explains that a container image can be treated much like any other file or archive — a zip or tar file sitting on disk. Because of that, teams can sign images cryptographically to verify origin and integrity, similar to how Node.js developers sign releases with their private keys. That gives consumers confidence that the image came from a known source and has not been tampered with. But Julien pushes back on a common misunderstanding: signing does not mean the contents are inherently safe. As he puts it, you get a "semantic guarantee that this image is what it's pretending to be" — but not proof that everything inside is secure. Authenticity is not the same as security. The hosts frame this as a trust problem. In a production cluster, teams often want to prevent engineers or workloads from pulling arbitrary images and running them without controls. Signed images and curated registries help, but they do not eliminate the need for careful validation. Trust, Docker Hub, and the container supply chain A major part of the episode focuses on how much trust teams should place in public images, including those from Docker Hub. Andrey raises the practical reality: if you are running four different languages, you cannot build and maintain base images for all of them. It is much easier to grab the latest Node.js, Python, Ruby, or Java images from Docker Hub and build from there. Julien and Mattias acknowledge that reality, but caution against treating "official" or branded images as automatically secure. Julien walks through the different trust levels on Docker Hub: Images from unknown individuals are the hardest to trust Organization-backed images (Red Hat, CloudBees, etc.) provide more accountability based on brand recognition Even reputable images can contain known vulnerabilities — scanning a Jenkins image from Docker Hub can reveal a surprising number of CVEs A trusted source can still introduce problems, whether by mistake or through malicious intent That leads into a broader discussion of supply chain attacks. Julien references real examples where Node.js libraries on npm were taken over by malicious parties after the original maintainer walked away. The same risk applies to container images. Julien points out that large organizations sometimes go as far as rebuilding all dependencies from source — he mentions having heard of teams that do not pull jar files from Maven Central but build their own from source to verify exactly what they are shipping. While that is not feasible for every team, the principle stands: reduce blind trust and increase verification where the environment demands it. Why container security is not just image signing The discussion then shifts from image authenticity to runtime security. Mattias explains that containers rely on Linux kernel primitives — namespaces for process isolation, along with controls for networking, memory, and disk. These low-level APIs are useful for resource sharing and scaling, but they were not originally designed as strong security boundaries. As he puts it, "the container does not contain things, it's just an abstraction." Container breakout vulnerabilities matter because an attacker who can exploit the runtime or host interface may reach beyond the container itself. This leads to one of the episode's sharpest observations from Mattias: containers became popular because they are efficient and convenient to operate — you can bin-pack them on the same hardware and run far more applications per server. But from a security perspective, "it was not designed to be secure by default, it was designed to be convenient." That gap between convenience and security is what teams must actively address through scanning, hardening, and runtime controls. CVE scanning: registries, dependencies, and source code The hosts spend a good

Gitops a new concept on devops. Whats is it and how can you use it when deploy and setup your k8s cluster. Summary GitOps sounds simple — put Kubernetes manifests in Git and let the cluster pull changes — but the episode quickly reveals the real debate is not about Git at all. Andrey argues the only genuinely novel thing about GitOps is the pull-based model where an in-cluster agent reconciles state, while Julien questions whether GitOps is good for day-2 operations or just for bootstrapping clusters. The spiciest moment: Andrey declares "life is too short to do pull requests" and advocates pushing straight to master with strong CI/CD guardrails instead. Key Topics What GitOps actually is — and what it is not Andrey frames the discussion by separating what is genuinely new about GitOps from what teams have already been doing for years. Storing deployment specifications in Git, he argues, is just version control — teams have done that for a decade. The meaningful difference is the deployment model: instead of an external CI/CD server pushing changes into Kubernetes by calling the cluster API, GitOps places an agent inside the cluster that either receives a webhook or polls a Git repository, pulls in the desired state, and applies it from within. That pull-based model is what Andrey identifies as the core innovation. It eliminates the need to expose the Kubernetes API externally — a real concern when using hosted CI services like CircleCI, which would otherwise need network access to the cluster. As Andrey puts it, exposing the API externally is risky "unless you want someone mining bitcoin on your cluster." He references the tooling landscape at the time: Weaveworks (the company that coined the term "GitOps" and created WeaveNet, a Kubernetes CNI driver), Flux, Argo, and Jenkins X. He notes that Flux and Argo were joining forces at the time of recording. He also mentions Jenkins X as a potential GitOps tool, since it runs CI/CD jobs natively in Kubernetes, but expresses skepticism about using Kubernetes for build workloads — Kubernetes is declarative about desired state, but "you cannot declare my build is successful because you have no idea how your build gonna go." Editor's note: Weaveworks, the company that originated the term "GitOps," shut down in February 2024. Flux continues as a CNCF graduated project. The GitOps principles have since been formalized by the OpenGitOps project under the CNCF. The Weaveworks definition, read straight from the source Andrey reads Weaveworks' concise GitOps definition from their blog and walks through its key points: The desired state of the whole system is described declaratively — Git is the single source of truth for every environment. All changes to desired state are Git commits — operations are driven through version control. The cluster state is observable — so teams can detect when desired and observed states have converged or diverged. A convergence mechanism brings the system back — when states diverge, the cluster automatically reconciles, either triggered immediately by a webhook or on a configurable polling interval. Rollback is simply convergence to an earlier desired state. Andrey also raises a nuance about Helm: since Helm templates can produce different output depending on input variables, true GitOps implies committing not only the Helm charts but also the rendered manifests — because the generated output is what actually represents the declarative desired state. He draws a comparison to GitHub's earlier promotion of ChatOps, noting that many of the same ideas — observable, verifiable changes driven through a central workflow — were already part of GitHub's operational philosophy, just with a different interface. Two layers: infrastructure-as-code and in-cluster GitOps Julien offers a more practical framing, splitting the problem into two distinct layers: Infrastructure as code — setting up the underlying infrastructure (VPCs, clusters, networking) GitOps — managing what runs inside the Kubernetes cluster: applications, operational tooling like monitoring (he mentions FluentD as an example), and supporting services In Julien's model, a Git repository becomes the authoritative inventory of everything that should exist in the cluster. He describes the ideal: "if anything else is running here, alert me or kill it." That gives teams confidence that the observed cluster state matches the intended one, and helps prevent configuration drift — a problem the hosts discussed in their earlier infrastructure-as-code episode. Day-2 operations: where the model gets tested While Julien appreciates GitOps for defining and bootstrapping cluster state, he is openly skeptical about its effectiveness for long-running operations. He distinguishes between two very different challenges: "setting up things" versus "running things for a long time — they're not the same." Real environments drift. People intervene manually during incidents. Urgent fixes happen outside the normal w

Are infra as code always the best way to go and if not when and where should you use it. Here we are trying to better understand when its god to use and when its not. Summary In this inaugural episode, Mattias, Andrey, and Julian discuss what infrastructure as code really means, why teams adopt it, and where it can go wrong. They explore the evolution from manual server management to declarative infrastructure, the differences between configuration management and infrastructure provisioning, the growing complexity of tools like Terraform and CloudFormation, and why culture, process, and operational discipline matter as much as the tooling itself. Key Topics What Infrastructure as Code Actually Solves The discussion starts with Mattias describing the shift from manually editing Apache configs over SSH to defining cloud environments in code. He recalls the progression: first managing individual servers by hand, then adopting configuration management tools like Puppet, Chef, and Ansible, and finally arriving at cloud-native tools like AWS CloudFormation that can provision entire environments declaratively. Andrey pushes the conversation toward first principles, arguing that it is important to separate the "what" from the "how." He explains that infrastructure as code depends on having APIs — software-defined interfaces that allow infrastructure to be created and managed programmatically. Without that kind of interface, teams are limited to SSH and the manual tools they had before. The rise of public cloud providers and platforms like OpenStack finally gave teams the APIs they needed to describe infrastructure declaratively in definition files. Configuration Management vs Infrastructure as Code A key distinction in the episode is the difference between server configuration tools and true infrastructure as code. Andrey notes that tools like Puppet, Chef, and Ansible were originally conceived as server configuration management tools — designed to automate the provisioning and configuration of servers, not to define infrastructure itself. He acknowledges this is a gray area, since tools like Ansible can now call AWS APIs and manage infrastructure directly. But historically, the configuration management era was about fighting configuration drift on existing servers, while the cloud era introduced the ability to declare entire environments as code. If you asked the vendors selling Chef, they would tell you Chef is "all about infrastructure as code" — but the original intent was different. When to Automate — and When Not To The hosts caution against automating too early. Andrey says he tends not to automate things until they genuinely need automation. If creating one cluster with a few nodes and one database is all you need, full automation may be premature. But if you know you will eventually manage hundreds or thousands, starting early makes sense. Julian reinforces this point with a memorable gym analogy: "You go to the gym, you see Arnold Schwarzenegger lifting 200 kilos from the ground and you say, he does it, I can do it. And then you pick up the little weight and find out that if you start with 200 kilos, you're gonna break your back." His point is that infrastructure as code tools get you up and running fast — that is what they are designed for — but day two operations always come knocking. The automation itself can become a burden if you are not careful about what you automate and when. Infrastructure as Documentation and Source of Truth Mattias describes one of his main reasons for using infrastructure as code: knowing what is actually running. He sees the codebase as documentation and as proof of the intended state of the environment — a way to verify that what he thinks is deployed matches what is actually in the cloud. The hosts agree with that idea, but they also point out the tension between declared state and reality. If people still make manual changes in the cloud console, the code drifts away from what is actually running. Andrey notes the problem: if undocumented manual changes are not reflected back into code, the next infrastructure deployment could recreate the original broken state — "you're back to the fire state, basically." The Terraform Complexity Problem Julian brings up Terraform as "the elephant in the room" and argues that it has become significantly more complex over time. He says the language started out as purely descriptive, but newer features in HCL2 — such as for loops, conditionals, and sequencing logic — have pushed it closer to a general-purpose programming language. His concern is that this makes infrastructure definitions harder to read and reason about. Instead of simply describing desired state, users now have to mentally execute the code to understand what it will produce. Andrey agrees there is a legitimate need for this evolution — once a declarative setup grows large enough, you genuinely want loops and conditionals — but acknowledges it creates a tension between reada