Flink Kubernates Native - 入门

创建 namespace

[root@CentOSA flink-1.17.1]# kubectl create ns flink-native
[root@CentOSA flink-1.17.1]# kubectl config set-context --current --namespace=flink-native

命令空间添加资源限制

[root@CentOSA flink-1.17.1]# vim namespace-ResourceQuota.yaml

apiVersion: v1 
kind: ResourceQuota
metadata:name: mem-cpu-quotanamespace: flink-native
spec:hard:requests.cpu: "10"requests.memory: 10Gilimits.cpu: "10"limits.memory: 10Gi

[root@CentOSA flink-1.17.1]# kubectl apply -f namespace-ResourceQuota.yaml

创建用户并赋权

[root@CentOSA flink-1.17.1]# kubectl create serviceaccount ric
[root@CentOSA flink-1.17.1]# kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit --serviceaccount=flink-native:ric --namespace=flink-native

• 提前在所有的flnk节点拉取flink镜像

[root@CentOSA flink-1.17.1]# docker pull apache/flink:1.17.1-scala_2.12 
[root@CentOSA flink-1.17.1]# docker pull flink:1.17.1-scala_2.12 

SessionMode

创建Flink 会话

./bin/kubernetes-session.sh \-Dkubernetes.namespace=flink-native \-Dkubernetes.jobmanager.service-account=ric \-Dkubernetes.cluster-id=session001 \-Dtaskmanager.memory.process.size=1024m \-Dkubernetes.taskmanager.cpu=1 \-Dtaskmanager.numberOfTaskSlots=4 \-Dresourcemanager.taskmanager-timeout=3600000 \-Dkubernetes.rest-service.exposed.type=NodePort \-Dkubernetes.container.image.ref=flink:1.17.1-scala_2.12 

2023-08-28 15:50:06,802 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: taskmanager.memory.process.size, 1728m
2023-08-28 15:50:06,805 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.bind-host, localhost
2023-08-28 15:50:06,805 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: taskmanager.bind-host, localhost
2023-08-28 15:50:06,805 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: taskmanager.host, localhost
2023-08-28 15:50:06,805 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: parallelism.default, 1
2023-08-28 15:50:06,805 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2023-08-28 15:50:06,805 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.rpc.address, localhost
2023-08-28 15:50:06,806 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2023-08-28 15:50:06,806 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: rest.address, localhost
2023-08-28 15:50:06,806 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2023-08-28 15:50:06,806 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.rpc.port, 6123
2023-08-28 15:50:06,806 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: rest.bind-address, localhost
2023-08-28 15:50:06,900 INFO  org.apache.flink.client.deployment.DefaultClusterClientServiceLoader [] - Could not load factory due to missing dependencies.
2023-08-28 15:50:08,271 INFO  org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2023-08-28 15:50:08,291 INFO  org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2023-08-28 15:50:08,292 INFO  org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction network memory (57.600mb (60397978 bytes)) is less than its min value 64.000mb (67108864 bytes), min value will be used instead
2023-08-28 15:50:08,498 INFO  org.apache.flink.kubernetes.utils.KubernetesUtils            [] - Kubernetes deployment requires a fixed port. Configuration blob.server.port will be set to 6124
2023-08-28 15:50:08,498 INFO  org.apache.flink.kubernetes.utils.KubernetesUtils            [] - Kubernetes deployment requires a fixed port. Configuration taskmanager.rpc.port will be set to 6122
2023-08-28 15:50:09,989 INFO  org.apache.flink.kubernetes.KubernetesClusterDescriptor      [] - Create flink session cluster session001 successfully, JobManager Web Interface: http://192.168.42.130:32432

可以根据提示信息访问JobManager Web Interface: http://192.168.42.130:32432页面

提交任务到指定会话

运行flink session任务

 ./bin/flink run \--target kubernetes-session \-Dkubernetes.cluster-id=session001 \-Dkubernetes.namespace=flink-native \-Dkubernetes.jobmanager.service-account=ric \-Dkubernetes.rest-service.exposed.type=NodePort \./examples/streaming/TopSpeedWindowing.jar

Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
2023-08-28 15:54:02,974 INFO  org.apache.flink.kubernetes.KubernetesClusterDescriptor      [] - Retrieve flink cluster session001 successfully, JobManager Web Interface: http://192.168.42.130:32432
Job has been submitted with JobID 465dd11e73b5c9ac49c8e2945bf862b2

查看任务列表

 ./bin/flink list \--target kubernetes-session \-Dkubernetes.cluster-id=session001 \-Dkubernetes.namespace=flink-native \-Dkubernetes.jobmanager.service-account=ric \-Dkubernetes.rest-service.exposed.type=NodePort \

2023-08-28 15:57:33,654 INFO  org.apache.flink.kubernetes.KubernetesClusterDescriptor      [] - Retrieve flink cluster session001 successfully, JobManager Web Interface: http://192.168.42.130:32432
Waiting for response...
------------------ Running/Restarting Jobs -------------------
28.08.2023 15:54:03 : 465dd11e73b5c9ac49c8e2945bf862b2 : CarTopSpeedWindowingExample (RUNNING)
--------------------------------------------------------------
No scheduled jobs.

取消任务：

 ./bin/flink cancel \--target kubernetes-session \-Dkubernetes.cluster-id=session002 \-Dkubernetes.namespace=flink-native \-Dkubernetes.jobmanager.service-account=ric \-Dkubernetes.rest-service.exposed.type=NodePort \465dd11e73b5c9ac49c8e2945bf862b2

Cancelling job 465dd11e73b5c9ac49c8e2945bf862b2.
2023-08-28 16:02:43,209 INFO  org.apache.flink.kubernetes.KubernetesClusterDescriptor      [] - Retrieve flink cluster session001 successfully, JobManager Web Interface: http://192.168.42.130:32432
Cancelled job 465dd11e73b5c9ac49c8e2945bf862b2.

删除session作业：

[root@CentOSA flink-1.17.1]# kubectl delete deployment session001 -n flink-native
deployment.apps "session001" deleted

Application Model

构建docker镜像

1、编写DockerFile

[root@CentOSA flink-1.17.1] cd /export/server/app-mode
[root@CentOSA app-mode] vi Dockerfile
FROM flink
RUN mkdir -p $FLINK_HOME/usrlib
COPY TopSpeedWindowing.jar $FLINK_HOME/usrlib/TopSpeedWindowing.jar

2、构建镜像文件

[root@CentOSC app-mode]# docker build -t my-flink-job:1.0 ./
[+] Building 0.6s (8/8) FINISHED                                                                                                                                                                                      docker:default=> [internal] load .dockerignore                                                                                                                                                                                               0.1s=> => transferring context: 2B                                                                                                                                                                                                 0.0s=> [internal] load build definition from Dockerfile                                                                                                                                                                            0.1s=> => transferring dockerfile: 271B                                                                                                                                                                                            0.0s=> [internal] load metadata for docker.io/library/flink:1.17.1-scala_2.12                                                                                                                                                      0.0s=> [1/3] FROM docker.io/library/flink:1.17.1-scala_2.12                                                                                                                                                                        0.1s=> [internal] load build context                                                                                                                                                                                               0.0s=> => transferring context: 17.04kB                                                                                                                                                                                            0.0s=> [2/3] RUN mkdir -p /opt/flink/usrlib                                                                                                                                                                                        0.3s=> [3/3] COPY TopSpeedWindowing.jar  /opt/flink/usrlib/TopSpeedWindowing.jar                                                                                                                                                   0.0s=> exporting to image                                                                                                                                                                                                          0.0s=> => exporting layers                                                                                                                                                                                                         0.0s=> => writing image sha256:a017db16aaa8086045dc1b28315e6e2fe47113aa41e1a0f547b58c1b191dac4a                                                                                                                                    0.0s=> => naming to docker.io/library/my-flink-job:1.0                                                                                                                                                                             0.0s
[root@CentOSC appMode]# docker images
REPOSITORY                                           TAG                 IMAGE ID       CREATED              SIZE
my-flink-job                                         1.0                 a017db16aaa8   About a minute ago   782MB
flannel/flannel                                      v0.22.2             d73868a08083   10 days ago          70.2MB
flink                                                1.17.1-scala_2.12   efd18635797b   11 days ago          782MB
flannel/flannel-cni-plugin                           v1.2.0              a55d1bad692b   5 weeks ago          8.04MB
apache/flink                                         1.17.1-scala_2.12   fef5f770f8b3   3 months ago         780MB
registry.aliyuncs.com/google_containers/kube-proxy   v1.20.5             5384b1650507   2 years ago          118MB
registry.aliyuncs.com/google_containers/pause        3.2                 80d28bedfe5d   3 years ago          683kB

启动任务

 ./bin/flink run-application \--target kubernetes-application \-Dkubernetes.cluster-id=my-first-application-cluster \-Dkubernetes.container.image=my-flink-job:1.0 \-Dkubernetes.namespace=flink-native \-Dkubernetes.jobmanager.service-account=ric \-Dkubernetes.rest-service.exposed.type=NodePort \local:/opt/flink/usrlib/TopSpeedWindowing.jar 

2023-08-28 16:07:45,180 INFO  org.apache.flink.kubernetes.utils.KubernetesUtils            [] - Kubernetes deployment requires a fixed port. Configuration blob.server.port will be set to 6124
2023-08-28 16:07:45,181 INFO  org.apache.flink.kubernetes.utils.KubernetesUtils            [] - Kubernetes deployment requires a fixed port. Configuration taskmanager.rpc.port will be set to 6122
2023-08-28 16:07:46,742 INFO  org.apache.flink.kubernetes.KubernetesClusterDescriptor      [] - Create flink application cluster my-first-application-cluster successfully, JobManager Web Interface: http://192.168.42.130:32003

查看任务列表

./bin/flink list \--target kubernetes-application \-Dkubernetes.cluster-id=my-first-application-cluster \-Dkubernetes.namespace=flink-native \-Dkubernetes.jobmanager.service-account=ric \-Dkubernetes.rest-service.exposed.type=NodePort 

2023-08-28 16:10:00,093 INFO  org.apache.flink.kubernetes.KubernetesClusterDescriptor      [] - Retrieve flink cluster my-first-application-cluster successfully, JobManager Web Interface: http://192.168.42.130:32003
Waiting for response...
------------------ Running/Restarting Jobs -------------------
28.08.2023 16:07:59 : c9ec57a2003a808ef5df04219fc7aacb : CarTopSpeedWindowingExample (RUNNING)
--------------------------------------------------------------
No scheduled jobs.

删除任务

 ./bin/flink cancel \--target kubernetes-application \-Dkubernetes.cluster-id=my-first-application-cluster \-Dkubernetes.namespace=flink-native \-Dkubernetes.jobmanager.service-account=ric \-Dkubernetes.rest-service.exposed.type=NodePort \c9ec57a2003a808ef5df04219fc7aacb

2023-08-28 16:11:16,963 INFO  org.apache.flink.kubernetes.KubernetesClusterDescriptor      [] - Retrieve flink cluster my-first-application-cluster successfully, JobManager Web Interface: http://192.168.42.130:32003
Cancelled job c9ec57a2003a808ef5df04219fc7aacb.