Kubernetes多租户架构设计与实践

Kubernetes多租户架构设计与实践一、引言多租户是指在同一个Kubernetes集群中为多个用户或团队提供隔离的资源和环境。本文将深入探讨Kubernetes多租户架构的核心概念、实现方法和最佳实践。二、多租户架构设计2.1 多租户参考架构┌─────────────────────────────────────────────────────────────────┐ │ Kubernetes多租户架构 │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Control Plane │ │ │ │ (API Server / etcd / Scheduler / Controller Manager) │ │ │ └───────────────────────────┬─────────────────────────────┘ │ │ │ │ │ ┌─────────────────────┼─────────────────────┐ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Tenant │ │ Tenant │ │ Tenant │ │ │ │ A │ │ B │ │ C │ │ │ │ (Team A) │ │ (Team B) │ │ (Team C) │ │ │ └──────────┘ └──────────┘ └──────────┘ │ │ │ │ │ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Worker Nodes │ │ │ │ (Pods / Services / Storage) │ │ │ └─────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘2.2 多租户隔离级别隔离级别描述实现方式命名空间级别基础隔离Namespace NetworkPolicy资源配额级别资源限制ResourceQuota LimitRangeRBAC级别权限隔离Role RoleBinding网络级别网络隔离NetworkPolicy CNI存储级别存储隔离PVC StorageClass三、多租户实现实践3.1 命名空间隔离apiVersion: v1 kind: Namespace metadata: name: tenant-a labels: tenant: tenant-a environment: production3.2 资源配额配置apiVersion: v1 kind: ResourceQuota metadata: name: tenant-a-quota namespace: tenant-a spec: hard: requests.cpu: 4 requests.memory: 8Gi limits.cpu: 8 limits.memory: 16Gi pods: 20 services: 10 persistentvolumeclaims: 103.3 LimitRange配置apiVersion: v1 kind: LimitRange metadata: name: tenant-a-limits namespace: tenant-a spec: limits: - type: Pod max: cpu: 2 memory: 4Gi min: cpu: 100m memory: 128Mi - type: Container max: cpu: 1 memory: 2Gi min: cpu: 50m memory: 64Mi default: cpu: 200m memory: 256Mi defaultRequest: cpu: 100m memory: 128Mi3.4 RBAC配置apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: tenant-a-admin namespace: tenant-a rules: - apiGroups: [] resources: [pods, services, configmaps, secrets] verbs: [get, list, watch, create, update, delete] - apiGroups: [apps] resources: [deployments, statefulsets, daemonsets] verbs: [get, list, watch, create, update, delete]apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: tenant-a-admin-binding namespace: tenant-a roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: tenant-a-admin subjects: - kind: User name: tenant-a-user apiGroup: rbac.authorization.k8s.io四、多租户网络隔离4.1 NetworkPolicy配置apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: tenant-a-isolation namespace: tenant-a spec: podSelector: {} policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: tenant: tenant-a egress: - to: - podSelector: matchLabels: tenant: tenant-a - to: - namespaceSelector: matchLabels: name: kube-system4.2 网络策略全局配置apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-cross-tenant namespace: tenant-a spec: podSelector: {} policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: tenant: tenant-a五、多租户存储隔离5.1 存储类配置apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: tenant-a-storage provisioner: ebs.csi.aws.com parameters: type: gp3 iopsPerGB: 50 allowVolumeExpansion: true reclaimPolicy: Delete5.2 PVC配置apiVersion: v1 kind: PersistentVolumeClaim metadata: name: tenant-a-data namespace: tenant-a spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: tenant-a-storage六、多租户监控与计费6.1 监控隔离apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: tenant-a-monitor namespace: monitoring spec: namespaceSelector: matchNames: - tenant-a selector: matchLabels: app: tenant-a-app endpoints: - port: metrics interval: 30s6.2 资源使用统计# 租户CPU使用 sum(rate(container_cpu_usage_seconds_total{namespacetenant-a}[5m])) # 租户内存使用 sum(container_memory_working_set_bytes{namespacetenant-a}) # 租户存储使用 sum(kube_persistentvolumeclaim_resource_requests_storage_bytes{namespacetenant-a})七、多租户管理工具7.1 租户管理OperatorapiVersion: tenant.example.com/v1 kind: Tenant metadata: name: tenant-a spec: name: tenant-a description: Team A tenant quota: cpu: 4 memory: 8Gi pods: 20 roles: - name: admin users: - tenant-a-admin - name: developer users: - tenant-a-dev1 - tenant-a-dev27.2 租户创建脚本#!/bin/bash TENANT_NAME$1 # 创建命名空间 kubectl create namespace $TENANT_NAME # 创建资源配额 cat EOF | kubectl apply -f - apiVersion: v1 kind: ResourceQuota metadata: name: ${TENANT_NAME}-quota namespace: ${TENANT_NAME} spec: hard: requests.cpu: 4 requests.memory: 8Gi limits.cpu: 8 limits.memory: 16Gi pods: 20 EOF # 创建LimitRange cat EOF | kubectl apply -f - apiVersion: v1 kind: LimitRange metadata: name: ${TENANT_NAME}-limits namespace: ${TENANT_NAME} spec: limits: - type: Container max: cpu: 1 memory: 2Gi min: cpu: 50m memory: 64Mi EOF echo Tenant ${TENANT_NAME} created successfully八、多租户最佳实践8.1 租户隔离策略命名空间隔离每个租户使用独立的命名空间资源配额设置合理的资源限制网络隔离使用NetworkPolicy限制跨租户通信RBAC权限最小权限原则按需分配权限存储隔离使用专用的StorageClass8.2 租户生命周期管理apiVersion: tenant.example.com/v1 kind: Tenant metadata: name: tenant-a spec: state: active createdAt: 2024-01-15T10:00:00Z expirationDate: 2025-01-15T10:00:00Z8.3 安全审计# 查看租户资源 kubectl get all -n tenant-a # 查看RBAC绑定 kubectl get rolebindings -n tenant-a # 审计日志查询 kubectl logs -n kube-system kube-apiserver | grep tenant-a九、总结Kubernetes多租户架构是实现资源共享和隔离的关键技术。通过合理的命名空间设计、资源配额、网络策略和RBAC配置可以构建安全、高效的多租户Kubernetes集群。