aws(学习笔记第三十一课) aws cdk深入学习（batch-arm64-instance-type）

aws(学习笔记第三十一课)

• aws cdk深入学习

学习内容：

• 深入练习aws cdk下部署batch-arm64-instance-type

---

1. 深入练习aws cdk下部署batch-arm64-instance-type

1. 代码链接

   • 代码链接
     代码链接 -> batch-arm64-instance-type
   • 之前代码学习
     之前学习代码链接 -> aws(学习笔记第十八课) 使用aws cdk(python)进行部署

2. batch-arm64-instance-type代码测试

   • setup环境
     • setup virtual environment

       python -m venv .venv
       

     • 激活python virtual environment

       source .venv/Scripts/activate #这里使用的是git bash
       

     • 安装python的依赖包

        pip install -r requirements.txt
       

     • 代码中cdk.json的python3进行替换
       这里的phthon3没有配置，所以不能执行python3，讲这里改成了python。
       

3. 代码执行

   • 预先执行cdk bootstrap
cdk提供了bootstrap命令来创建cdk执行需要的前提条件。

     CDKToolkit: creating CloudFormation changeset...
     CDKToolkit |  0/12 | 21:22:46 | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack | CDKToolkit User Initiated
     CDKToolkit |  0/12 | 21:22:52 | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | CDKToolkit User Initiated
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | FilePublishingRole
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | CloudFormationExecutionRole
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::ECR::Repository    | ContainerAssetsRepository
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::S3::Bucket         | StagingBucket
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | ImagePublishingRole
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | LookupRole
     CDKToolkit |  0/12 | 21:22:56 | CREATE_IN_PROGRESS   | AWS::SSM::Parameter     | CdkBootstrapVersion
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::SSM::Parameter     | CdkBootstrapVersion Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | FilePublishingRole Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::ECR::Repository    | ContainerAssetsRepository Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | CloudFormationExecutionRole Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | ImagePublishingRole Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::S3::Bucket         | StagingBucket Resource creation Initiated
     CDKToolkit |  0/12 | 21:22:57 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | LookupRole Resource creation Initiated
     CDKToolkit |  1/12 | 21:22:57 | CREATE_COMPLETE      | AWS::SSM::Parameter     | CdkBootstrapVersion
     CDKToolkit |  2/12 | 21:22:58 | CREATE_COMPLETE      | AWS::ECR::Repository    | ContainerAssetsRepository
     CDKToolkit |  3/12 | 21:23:12 | CREATE_COMPLETE      | AWS::S3::Bucket         | StagingBucket
     CDKToolkit |  3/12 | 21:23:14 | CREATE_IN_PROGRESS   | AWS::S3::BucketPolicy   | StagingBucketPolicy
     CDKToolkit |  3/12 | 21:23:15 | CREATE_IN_PROGRESS   | AWS::S3::BucketPolicy   | StagingBucketPolicy Resource creation Initiated
     CDKToolkit |  4/12 | 21:23:15 | CREATE_COMPLETE      | AWS::S3::BucketPolicy   | StagingBucketPolicy
     CDKToolkit |  5/12 | 21:23:16 | CREATE_COMPLETE      | AWS::IAM::Role          | ImagePublishingRole
     CDKToolkit |  6/12 | 21:23:16 | CREATE_COMPLETE      | AWS::IAM::Role          | FilePublishingRole
     CDKToolkit |  7/12 | 21:23:16 | CREATE_COMPLETE      | AWS::IAM::Role          | CloudFormationExecutionRole
     CDKToolkit |  7/12 | 21:23:16 | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | FilePublishingRoleDefaultPolicy
     CDKToolkit |  8/12 | 21:23:16 | CREATE_COMPLETE      | AWS::IAM::Role          | LookupRole
     CDKToolkit |  8/12 | 21:23:17 | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | ImagePublishingRoleDefaultPolicy
     CDKToolkit |  8/12 | 21:23:17 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | DeploymentActionRole
     CDKToolkit |  8/12 | 21:23:18 | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | FilePublishingRoleDefaultPolicy Resource creation Initiated
     CDKToolkit |  8/12 | 21:23:18 | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | ImagePublishingRoleDefaultPolicy Resource creation Initiated
     CDKToolkit |  8/12 | 21:23:19 | CREATE_IN_PROGRESS   | AWS::IAM::Role          | DeploymentActionRole Resource creation Initiated
     CDKToolkit |  9/12 | 21:23:34 | CREATE_COMPLETE      | AWS::IAM::Policy        | FilePublishingRoleDefaultPolicy
     CDKToolkit | 10/12 | 21:23:34 | CREATE_COMPLETE      | AWS::IAM::Policy        | ImagePublishingRoleDefaultPolicy
     CDKToolkit | 11/12 | 21:23:38 | CREATE_COMPLETE      | AWS::IAM::Role          | DeploymentActionRole
     CDKToolkit | 12/12 | 21:23:39 | CREATE_COMPLETE      | AWS::CloudFormation::Stack | CDKToolkit✅  Environment aws://081353481087/ap-northeast-1 bootstrapped.
     

     这里，cdk bootstrap命令执行完之后，会生成一个CDKToolkit名字的cloudformation。
     
     如果由于平时的误删除之类的操作，会使CDKToolkit的创建的cdk deploy需要的前提条件遭到破坏。这时如果重新执行cdk bootstrap会导致没有差分执行出来。这时候需要将CDKToolkit这个cloudformation删除掉，之后执行cdk bootstrap命令，这样就会重新生成cdk deploy执行的所有条件。

   • 执行cdk --require-approval never deploy
     如果执行不加上--require-approval never，会导致错误，所以这里加上该选项。

   • Trouble Shouting
     如果存在默认的vpc，这里会出现如下错误。
     原因在于:
     Fn:GetAZs这个函数调用的话，会只返回default VPC的Available Zone的数量，从而影响cdk的执行。
     对策：
     所以要删除default VPC，不过不用担心，default VPC可以重新创建，大可放心。

4. 代码解析

   • 全体架构
     

   • 作成VPC

      # This resource alone will create a private/public subnet in each AZ as well as nat/internet gateway(s)
     vpc = ec2.Vpc(self, "VPC")
     

     创建一下service组件：

     • 一个VPC
     • 一个public subnet
     • 一个private subnet
     • 一个Internet Gateway
     • 一个NAT Gateway Compute Environment通过这个NAT Gateway访问internet
       

   • 创建Job Queue

     # Create AWS Batch Job Queue
     self.batch_queue = batch.JobQueue(self, "JobQueueArm64")
     

     

   • 创建Compute Environment，并将每个Compute Environment分配给JobQueue

             # For loop to create Batch Compute Environmentsfor i in range(count):name = "MyBatchARM64Env" + str(i)batch_environment = batch.ManagedEc2EcsComputeEnvironment(self, name,spot=True,spot_bid_percentage=75,instance_types=[ec2.InstanceType("c7g.medium"),ec2.InstanceType("c7g.large")],use_optimal_instance_classes=False,vpc_subnets=ec2.SubnetSelection(subnet_type=ec2.SubnetType.PRIVATE_WITH_NAT),vpc=vpc)self.batch_queue.add_compute_environment(batch_environment, i)
     

     

   • 创建Job Defintion

            # Create ECS Job Definition to submit job in batch job queue.batch_jobDef = batch.EcsJobDefinition(self, "MyJobDefArm64",container=batch.EcsEc2ContainerDefinition(self, "CDKJobDefArm64",image=ecs.ContainerImage.from_registry("public.ecr.aws/amazonlinux/amazonlinux:latest"),command=["sleep", "60"],memory=Size.mebibytes(512),cpu=1))
     

     

5. 执行job

   • 投入新的Job
     
   • Job执行成功