Home/Templates/See what containers are doing while they run
See what containers are doing while they run
Runtime alerts installed by the same pipeline that ships the workloads, so they cannot fall behind.
Argo CD
Why bother
Runtime security installed by hand drifts away from the cluster it is meant to watch, and disappears the next time the cluster is rebuilt. Shipping it through the same pipeline as everything else keeps them together.
Open a shell inside a pod. The alert should reach your SIEM.
Set it up
Pick your platform, then open the level you want. Each level is complete on its own. Read the comments in the files as you go. Anything with a real consequence is explained on the line where it happens.
Argo CD. Falco deployed by Argo CD, with rules kept in Git
What you need first
- A cluster where a privileged DaemonSet is permitted. Falco reads system calls, which is the whole point and also the trade.
- Kernel 5.8 or later for the eBPF driver. Older kernels need the kernel module instead.
- Somewhere for alerts to go. The Standard version expects a webhook in a Secret.
What it creates
- A falco namespace and a DaemonSet on every node
- Falcosidekick, which fans one alert out to several destinations
- A ConfigMap of local rules, synced from Git by Argo CD and mounted into every Falco pod (Standard)
The code
Quick startDraft
# k8s-runtime-security / kubernetes / t0 "Quick start"
#
# Falco watching every node, deployed by Argo CD from the official chart.
# Alerts go to the pod log, which is enough to see it working.
#
# kubectl apply -f falco.yaml
#
# Verify:
# kubectl run shell-test --rm -it --image=busybox -- sh
# kubectl logs -n falco -l app.kubernetes.io/name=falco | grep -i "shell"
# You should see a "Terminal shell in container" alert.
#
# Falco needs privileged access to each node to read system calls. That is the
# trade: it sees everything the kernel sees. Check this against your policy
# before deploying it to a cluster you share.
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: falco
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://falcosecurity.github.io/charts
chart: falco
# Pin an exact version. Check the chart repository for the current one.
# A moving target here means the cluster changes without a commit.
targetRevision: 4.0.0
helm:
values: |
driver:
# modern_ebpf needs kernel 5.8 or later and installs nothing on the
# host. Use kmod on older kernels.
kind: modern_ebpf
# Rules that ship with Falco. Start here, tune later.
falcoctl:
artifact:
install:
enabled: true
follow:
# Rule updates arriving on their own means the cluster changes
# without a commit. The Standard version turns this off and
# keeps rules in Git instead.
enabled: true
falco:
json_output: true
log_level: info
priority: notice
destination:
server: https://kubernetes.default.svc
namespace: falco
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
StandardDraft
# k8s-runtime-security / kubernetes / t1 "Standard"
#
# Falco with its settings and its local rules both kept in Git, rule updates
# turned off so nothing changes without a commit, and alerts sent to a SIEM.
#
# One Application, three sources:
# 1. the Falco chart itself, pinned
# 2. your repository, referenced as $config so the chart can read values.yaml
# 3. the rules/ folder in your repository, synced as ordinary objects
#
# The paths below assume this folder sits at the same place in your repository
# as it does in this one. Change them if it does not.
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: falco
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
annotations:
# Lands before the workloads it watches, so nothing starts on a node with
# no runtime monitoring.
argocd.argoproj.io/sync-wave: "-5"
spec:
project: platform
sources:
- repoURL: https://falcosecurity.github.io/charts
chart: falco
# Pinned. Check the chart repository for the current release and move
# this forward in a pull request, never by floating it.
targetRevision: 4.0.0
helm:
valueFiles:
- $config/outcomes/k8s-runtime-security/kubernetes/t1/values.yaml
- repoURL: https://github.com/your-org/your-cluster-config.git # change me
targetRevision: main
ref: config
- repoURL: https://github.com/your-org/your-cluster-config.git # change me
targetRevision: main
path: outcomes/k8s-runtime-security/kubernetes/t1/rules
destination:
server: https://kubernetes.default.svc
namespace: falco
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
retry:
limit: 5
backoff:
duration: 20s
factor: 2
maxDuration: 10m
# If the bootstrap template is already running, commit this folder and it is # picked up. To apply it by hand instead: # kubectl apply -k . # # Only the Application is listed. values.yaml and rules/ are read by Argo CD # from your repository, so applying them directly would create a second copy # that nothing manages. --- apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - application.yaml
# Rules that reflect decisions about this cluster, reviewed like any other code.
# Argo CD syncs this ConfigMap, and values.yaml mounts it into the Falco pods.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: falco-local-rules
namespace: falco
data:
local-rules.yaml: |-
# A shell inside a container is normal during development and suspicious in
# production. Rather than turning the rule off, this narrows it.
- list: shell_allowed_namespaces
items: [dev, staging]
- rule: Shell in production container
desc: >-
Someone opened an interactive shell inside a running container outside
the namespaces where that is expected.
condition: >
spawned_process
and container
and shell_procs
and not k8s.ns.name in (shell_allowed_namespaces)
output: >-
Shell opened in a production container
(user=%user.name container=%container.name image=%container.image.repository
namespace=%k8s.ns.name pod=%k8s.pod.name command=%proc.cmdline)
priority: WARNING
tags: [container, shell, mitre_execution]
# Reading the service account token is how a compromised pod starts moving
# sideways. Legitimate clients read it through the mounted file at startup,
# so a read by a shell or a network tool is the interesting case.
- rule: Service account token read by an unexpected process
desc: Something other than the application read the mounted service account token.
condition: >
open_read
and container
and fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount
and proc.name in (curl, wget, nc, ncat, socat, python3, perl, ruby, sh, bash)
output: >-
Service account token read by an unexpected process
(process=%proc.name command=%proc.cmdline container=%container.name
namespace=%k8s.ns.name pod=%k8s.pod.name)
priority: CRITICAL
tags: [container, credentials, mitre_credential_access]
# Falco settings. Kept in Git so a change to what the cluster watches for is a
# pull request with an author on it.
---
driver:
# Needs kernel 5.8 or later and installs nothing on the host. Use kmod on
# older kernels.
kind: modern_ebpf
# Rule updates are off on purpose. With them on, the rules running on your
# cluster change whenever upstream publishes, and "why did this alert start
# firing" stops having an answer. The rules that ship inside the Falco image
# still load.
falcoctl:
artifact:
install:
enabled: false
follow:
enabled: false
falco:
json_output: true
json_include_output_property: true
log_level: info
priority: notice
# Mount the local rules ConfigMap into the folder Falco already reads.
# Without these two lists the ConfigMap exists in the cluster and Falco never
# sees it, which fails silently: no error, and no alerts from your rules.
mounts:
volumes:
- name: local-rules
configMap:
name: falco-local-rules
volumeMounts:
- name: local-rules
mountPath: /etc/falco/rules.d/local-rules.yaml
subPath: local-rules.yaml
readOnly: true
# Where alerts go. Falcosidekick fans one event out to several destinations.
falcosidekick:
enabled: true
replicaCount: 2
config:
# The webhook URL usually carries a token, so it comes from a Secret you
# create separately, outside this file.
existingSecret: falco-alert-destinations
webui:
enabled: false
resources:
requests:
cpu: 100m
memory: 512Mi
limits:
memory: 1Gi
# Falco has to run on every node, including nodes reserved for other
# workloads. A node Falco skips is a node with no runtime visibility.
tolerations:
- operator: Exists
HardenedPlanned
A rules ConfigMap does nothing on its own. Falco reads files from a folder, so the ConfigMap has to be mounted into that folder. Skip the mount and there is no error, and no alerts from your rules either. The Standard values.yaml mounts it.
With rule updates left on, the rules your cluster alerts on change whenever upstream publishes, and the question of why an alert started firing stops having an answer. The Standard version turns them off.
How to undo it
Delete the Application. The eBPF driver installs nothing on the host, so there is nothing left behind on the nodes.
What it costs
A pod per node, roughly 100m of CPU and 512Mi of memory each. The larger cost is usually the alert volume on the receiving side, which is why the Standard rules narrow the noisy defaults and keep them running.