k8s笔记29--使用kyverno提高运维效率

介绍
原理
安装
应用场景
- 自动修正测试环境pod资源
- 强制 Pod 标签
- 限制容器镜像来源
- 禁止特权容器
- 其它潜在场景
注意事项
说明

介绍

Kyverno是一个云原生的策略引擎，它最初是为k8s构建的，现在也可以在k8s集群之外用作统一的策略语言。其允许平台工程师自动化安全、合规和关于验证的最佳实践，并为相关团队提供安全的自助服务。
在 K8S的复杂生态系统中，确保集群资源的合规性和安全性是一项极具挑战的任务。Kyverno 作为一个强大的k8s策略引擎，为我们提供了一种有效的解决方案。本文将深入探讨 Kyverno 的核心原理、经典案例以及使用过程中的注意事项。

原理

Kyverno 通过 Kubernetes 的准入控制器（Admission Controller）机制来实现策略的执行。当用户向 Kubernetes API Server 提交资源创建、更新或删除请求时，准入控制器会拦截这些请求，并将其发送给 Kyverno 进行策略检查。
Kyverno 的策略以自定义资源定义（CRD）的形式存在，用户可以通过编写 YAML 文件来定义各种策略规则。这些规则可以基于资源的类型、名称、标签选择器等属性进行匹配，并执行相应的操作，如拒绝请求、修改资源或添加注释等。
例如，我们可以定义一个策略，要求所有新建的 Pod 必须包含特定的标签。当用户尝试创建一个不包含该标签的 Pod 时，Kyverno 会根据策略规则拒绝这个请求，从而确保集群中所有 Pod 都符合我们的标签规范。

kyverno逻辑架构图：
https://kyverno.io/docs/introduction/how-kyverno-works/
在这里插入图片描述
策略和规则的运行方式：
https://kyverno.io/docs/kyverno-policies/
如下图所示，一个Policy可以包含多个Rule，每个规则必须包含一个Match(Exclude可选)和validate,mutate,generate,verifyImages等选项之一。
策略可以定义为集群层面的ClusterPolicy，也可以定义为命名空间层面的Policy。
在这里插入图片描述

安装

安装文档：https://kyverno.io/docs/installation/
可以直接通过yaml的方式安装，也可以通过helm的方式安装
yaml安装方法:

kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.13.0/install.yaml

helm安装方法:

单机版本：
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

高可用版本：
helm install kyverno kyverno/kyverno -n kyverno --create-namespace \
--set admissionController.replicas=3 \
--set backgroundController.replicas=3 \
--set cleanupController.replicas=3 \
--set reportsController.replicas=3

安装成功会有如下4个running的deploy

$ kubectl -n kyverno get deploy
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
kyverno-admission-controller    1/1     1            1           24d
kyverno-background-controller   1/1     1            1           24d
kyverno-cleanup-controller      1/1     1            1           24d
kyverno-reports-controller      1/1     1            1           24d

注意：不同版本的kyverno支持的k8s版本有限，安装前可以在官方文档查看版本支持信息

应用场景

自动添加DR的outlierDetection
火山迁移期间，新建DR的时候会默认添加outlierDetection，确保优先网格内服务的稳定性
使用一下策略后，新建DR，如果没有trafficPolicy属性则会自动补充策略中的内容。
策略内容

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: general-namespace-add-workload-dr-outlier-detection
spec:
  admission: true
  background: true
  rules:
  - exclude:
      all:
      - resources:
          namespaces:
          - quicksilver
          - appsvr
    match:
      all:
      - resources:
          kinds:
          - networking.istio.io/v1beta1/DestinationRule
    mutate:
      patchStrategicMerge:
        spec:
          +(trafficPolicy):
            outlierDetection:
              baseEjectionTime: 60s
              consecutive5xxErrors: 0
              consecutiveLocalOriginFailures: 10
              interval: 10s
              maxEjectionPercent: 80
              splitExternalLocalOriginErrors: true
    name: general-namespace-add-workload-dr-outlier-detection

自动修正测试环境pod资源

测试环境主动限制指定命名空间的服务资源request值，可以显著降低业务request过多空资源

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: ebc-default-cpu-resources
spec:
  background: true
  rules:
    - name: default-cpu-resources
      match:
        any:
        - resources:
            kinds:
              - Pod
            namespaces:
              - ebc-aladdinsite
              - ebc-bpm
      exclude:
        any:
        - resources:
            kinds:
              - Pod
            selector:
              matchLabels:
                app: nacos
        - resources:
            kinds:
              - Pod
            selector:
              matchLabels:
                app.kubernetes.io/name: ebc-ldap
      mutate:
        patchStrategicMerge:
          spec:
            containers:
              - (name): "*"
                resources:
                  requests:
                    cpu: "100m"
                    memory: "500Mi"
                  limits:
                    cpu: "2000m"
                    memory: "4000Mi"
    - name: set-resources-init-containers
      match:
        any:
        - resources:
            kinds:
              - Pod
            namespaces:
              - ebc-aladdinsite
              - ebc-bpm
      preconditions:
        all:
        - key: "{{ request.object.spec.initContainers || '' }}"
          operator: NotEquals
          value: ""
      mutate:
        patchStrategicMerge:
          spec:
            initContainers:
              - (name): "*"
                resources:
                  requests:
                    cpu: "100m"
                    memory: "200Mi"
                  limits:
                    cpu: "1000m"
                    memory: "2000Mi"

也可以通过如下方式设置资源请求和限制

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: test-kyverno-set-resource-limits
spec:
  validationFailureAction: enforce
  rules:
  - name: set-cpu-memory-limits
    match:
      resources:
        kinds:
          - Deployment
        namespaces:
          - test-kyverno 
    mutate:
      patchesJson6902:
      - op: add
        path: /spec/template/spec/containers/0/resources
        value:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

强制 Pod 标签

确保所有部署到集群中的 Pod 都带有特定的标签，用于资源管理和监控。

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: test-kyverno-enforce-pod-labels
spec:
  validationFailureAction: enforce
  rules:
  - name: check-pod-labels
    match:
      resources:
        kinds:
          - Pod
        namespaces:
          - test-kyverno
    validate:
      message: "Pod must have 'environment' and 'app' labels."
      pattern:
        metadata:
          labels:
            environment: "?*"
            app: "?*"

此时创建一个缺少label的deploy就会报错, 加上 environment 标签后就可以正常访创建了

$ vim test-nginx-with-label.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test2-deployment
  namespace: test-kyverno
  labels:
    app: test2
    # environment: prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test2
      # environment: prod
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: test2
        # environment: prod
    spec:
      containers:
        - name: test2
          image: nginx:1.23
          ports:
            - containerPort: 80
              protocol: TCP
          resources: {}
              
$ kubectl apply -f test-nginx-with-label.yaml 
The Deployment "test2-deployment" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"test2"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

限制容器镜像来源

为了保障安全，只允许从特定的镜像仓库拉取容器镜像

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: test-kyverno-restrict-image-registry
spec:
  validationFailureAction: enforce
  rules:
  - name: check-image-registry
    match:
      resources:
        kinds:
          - Pod
        namespaces:
          - test-kyverno 
    validate:
      message: "Only images from 'my-allowed-registry.com' are allowed."
      pattern:
        spec:
          containers:
          - image: my-allowed-registry.com/*

此时创建非my-allowed-registry.com 镜像的pod会报错

$ vim test-nginx-with-image-registry.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test2-deployment
  namespace: test-kyverno
  labels:
    app: test2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test2
  template:
    metadata:
      labels:
        app: test2
    spec:
      containers:
        - name: test2
          image: nginx:1.23
          # image: my-allowed-registry.com/nginx:1.23
          ports:
            - containerPort: 80
              protocol: TCP
          resources: {}

$ kubectl apply -f test-nginx-with-image-registry.yaml 
Error from server: error when creating "test-nginx-with-image-registry.yaml": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Deployment/test-kyverno/test2-deployment was blocked due to the following policies 

test-kyverno-restrict-image-registry:
  autogen-check-image-registry: 'validation error: Only images from ''my-allowed-registry.com''
    are allowed. rule autogen-check-image-registry failed at path /spec/template/spec/containers/0/image/'

禁止特权容器

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: test-kyverno-prohibit-privileged-containers
spec:
  validationFailureAction: enforce
  rules:
  - name: check-privileged
    match:
      resources:
        kinds:
          - Pod
        namespaces:
          - test-kyverno 
    validate:
      message: "Privileged containers are not allowed."
      pattern:
        spec:
          containers:
          - privileged: false

开启特权就会报错

$ vim test-nginx-with-prohibit-privileged-containers.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test2-deployment
  namespace: test-kyverno
  labels:
    app: test2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test2
  template:
    metadata:
      labels:
        app: test2
    spec:
      containers:
        - name: test2
          image: nginx:1.23
          ports:
            - containerPort: 80
              protocol: TCP
          resources: {}
          securityContext:
            privileged: true

$ kubectl apply -f test-nginx-with-prohibit-privileged-containers.yaml 
Error from server: error when creating "test-nginx-with-prohibit-privileged-containers.yaml": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Deployment/test-kyverno/test2-deployment was blocked due to the following policies 

test-kyverno-prohibit-privileged-containers:
  autogen-check-privileged: 'validation error: Privileged containers are not allowed.
    rule autogen-check-privileged failed at path /spec/template/spec/containers/0/privileged/'

其它潜在场景

新建命名空间的时候创建对应的configmap
新建命名空间的时候创建对应的resource quote
给deploy添加默认的环境变量
限制pod默认安全策略

注意事项

策略优先级
当存在多个策略时，需要注意策略的优先级设置。Kyverno 按照策略的名称顺序进行匹配和执行，因此需要合理命名策略以确保正确的执行顺序。
资源匹配规则
在编写策略时，要仔细定义资源的匹配规则，避免误判或漏判。例如，使用通配符时要确保其范围符合预期。
测试环境
在将策略应用到生产环境之前，务必在测试环境中进行全面且充分的测试，确保策略的正确性与有效性，防止因策略错误导致业务中断。
性能影响
尽管Kyverno的设计旨在实现高效运行，但大量复杂的策略仍可能对Kubernetes API Server的性能产生一定影响。因此，需要定期评估和优化策略，以保障系统的稳定运行。