AWS

AWS GWLB对访问NLB流量做安全检测

Liu Qianglong

Nov 7, 2022 • 22 min read

B站视频

一、架构图

这个设计的核心在于路由表的设计，可以根据下面的架构图示，理解路由流量走向。（我发现公众号对图片压缩特别厉害，暂时没有找到上传原图的办法。如果你和我一样有强迫症，可以找我要高清图片）

AWS-GWLB-Scenario-1-Inspection-Ingress-Traffic-with-NLB

实验环境一共有两个VPC，左边的是业务VPC，里面有两个APP模拟HTTP的业务，EC2放在私有子网里面，在不同的AZ。

有一个面向互联网的Network Load Balancer，向公网发布了这个HTTP的服务，这样互联网上的用户，就可以通过NLB访问后面的业务。我们要做的是把这个流量，引导到防火墙上去，做安全检测。这里使用Linux的iptables来模拟防火墙。

另外，APP可以通过NAT GW上网，这个APP主动访问互联网的流量，也需要送到防火墙上去，做安全检测。所以，APP有两种流量都需要送到防火墙上去做安全检测，下面看一下APP的两种流量路径。

一、来自于互联网对NLB访问的流量路径。

首先，互联网上的用户，对NLB的公有DNS发起请求，这个DNS请求，会解析到NLB的两个公网IP地址，然后流量通过ISP路由，到达AWS的IGW上。
因为IGW关联了一个Ingress Route Table，发现去往NLB所在子网的路由指向GWLB endpoint1或GWLB endpoint2，这里假设现在解析的主IP是Endpoint2所在的网段，这样路由会送到GWLB Endpoint2。
Endpoint2收到流量之后，会通过Private Link，把流量发送到GWLB上。
GWLB会通过GENEVE封装报文，把流量发送到防火墙。
防火墙做完安全检测之后，又会把流量送回给GWLB，然后通过Private Link，送到GWLB Endpoint2。
注意，到达GWLB Endpoint2之后，查询的是GWLB Endpoint2关联的Public Route Table，因为目的地址是NLB subnet2网段，所以匹配到local路由，将流量送给NLB。
流量到NLB之后，因为侦听组关联了APP实例，所以NLB将流量发送到实例的主接口，这里查询的是NLB所在子网的local路由。
最后，流量抵达了APP。以上就是来自互联网，对NLB发起的请求，流量经过防火墙之后，再抵达APP的过程。

继续看一下回包流程。

注意，对于来自NLB的流量，EC2并不查询所在子网的路由表，而是直接返回给NLB（这个点很有意思，有机会单独再聊NLB的行为）。
NLB收到报文之后，查询所在子网路由表，因为回包是互联网上的地址，所以会匹配到默认路由，将报文发送到GWLB Endpoint2。
接下来又是一样的流程，流量会经过Private Link到防火墙绕一圈再回来。
流量回到GWLB Endpoint2上之后，会匹配到默认路由，将流量通过IGW发送到互联网上。

以上就是完整的从互联网对NLB发起请求和回包的流程。

二、APP1主动访问互联网的流量路径

APP1对互联网地址发起请求，APP1查询子网关联的路由表，匹配到默认路由，流量送到NAT GW1。
NAT GW1收到流量后，查询路由表，默认路由把流量送给GWLB Endpoint1，发送时会对源IP地址做NAT转换。
流量到GWLB Endpoint1之后，通过Private Link把流量送到防火墙检测，然后再发回来。
GWLB Endpoint1收到流量之后，默认路由将流量送往IGW到达互联网，源地址是NAT GW1的公网IP地址。

继续看回包流程。

互联网上的主机收到报文以后，源地址是NAT GW1的公网IP地址，报文通过ISP路由，流量到达AWS的IGW。
到达IGW之后，会查询Ingress Route Table，去往NAT GW1网段的路由送到GWLB Endpoint1。
流量到GWLB Endpoint1之后，通过Private Link把流量送到防火墙检测，然后再发回来。
GWLB Endpoint1收到流量之后，匹配到local路由，将流量发送到NAT GW1。
NAT GW1收到报文之后，查询NAT转换表项，目的地址匹配到local路由表，将流量发送到APP1。

二、创建实验环境

通过CloudFormation创建实验环境，堆栈需要7分钟左右的时间创建完成。

上传堆栈文件。

编辑堆栈名称，修改实例密钥。

允许创建IAM资源。


Parameters:
  EC2InstanceAmiId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'
  Environment:
    Type: String
    AllowedValues:
      - dev
      - prod
    Default: dev
  MyKeyPair:
    Description: Amazon EC2 Key Pair
    Type: AWS::EC2::KeyPair::KeyName
    Default: Global_Tokyo_KeyPair
  WebServerPort:
    Description: Apache Http Server Port
    Type: String
    Default: 8443
    AllowedValues:
      - 8443
      - 8888
      - 8088

Resources:

  BastionSsmRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /

  BastionSsmPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: PrivatelianceInstanceAccess
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - ssm:DescribeAssociation
              - ssm:GetDeployablePatchSnapshotForInstance
              - ssm:GetDocument
              - ssm:DescribeDocument
              - ssm:GetManifest
              - ssm:GetParameter
              - ssm:GetParameters
              - ssm:ListAssociations
              - ssm:ListInstanceAssociations
              - ssm:PutInventory
              - ssm:PutComplianceItems
              - ssm:PutConfigurePackageResult
              - ssm:UpdateAssociationStatus
              - ssm:UpdateInstanceAssociationStatus
              - ssm:UpdateInstanceInformation
            Resource: "*"
          - Effect: Allow
            Action:
              - ssmmessages:CreateControlChannel
              - ssmmessages:CreateDataChannel
              - ssmmessages:OpenControlChannel
              - ssmmessages:OpenDataChannel
            Resource: "*"
          - Effect: Allow
            Action:
              - ec2messages:AcknowledgeMessage
              - ec2messages:DeleteMessage
              - ec2messages:FailMessage
              - ec2messages:GetEndpoint
              - ec2messages:GetMessages
              - ec2messages:SendReply
            Resource: "*"
      Roles:
        - !Ref BastionSsmRole

  BastionSsmProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref BastionSsmRole

#=========================================SecVpc========================================#
# 创建SecVpc
  SecVpc:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.100.10.0/16
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'
      Tags:
       - Key: Name
         Value: !Sub ${AWS::StackName}-SecVpc

# 创建IGW并且关联到VPC
  SecVpcIGW:
    Type: "AWS::EC2::InternetGateway"
    Properties:
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpcIGW

  SecVpcAttachIgw:
    Type: "AWS::EC2::VPCGatewayAttachment"
    Properties:
      VpcId: !Ref SecVpc
      InternetGatewayId: !Ref SecVpcIGW

#---------------------------SecVpc创建6个子网-------------------------------------#

# SecVpc AZ1内创建公有子网
  SecVpcAz1PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.10.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ1-Public-Subnet

# SecVpc AZ2内创建公有子网
  SecVpcAz2PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.20.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ2-Public-Subnet

# SecVpc AZ1内创建私有子网
  SecVpcAz1PrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.30.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ1-Private-Subnet

# SecVpc AZ2内创建私有子网
  SecVpcAz2PrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.40.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ2-Private-Subnet


# SecVpc AZ1内创建TGW子网
  SecVpcAz1TgwSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.50.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ1-TGW-Subnet

# SecVpc AZ2内创建TGW子网
  SecVpcAz2TgwSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.60.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ2-TGW-Subnet

#---------------------------SecVpc创建路由表-------------------------------------#

# 公有子网路由表及关联
  SecVpcAz1PublicRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ1-Public-RouteTable

  SecVpcAz1PublicRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz1PublicRouteTable
      SubnetId: !Ref SecVpcAz1PublicSubnet

  SecVpcAz2PublicRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ2-Public-RouteTable

  SecVpcAz2PublicRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz2PublicRouteTable
      SubnetId: !Ref SecVpcAz2PublicSubnet

# Private子网路由表及关联
  SecVpcAz1PrivateRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ1-Private-RouteTable

  SecVpcAz1PrivateRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz1PrivateRouteTable
      SubnetId: !Ref SecVpcAz1PrivateSubnet

  SecVpcAz2PrivateRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ2-Private-RouteTable

  SecVpcAz2PrivateRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz2PrivateRouteTable
      SubnetId: !Ref SecVpcAz2PrivateSubnet


# Tgw路由表及关联
  SecVpcAz1TgwRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ1-Tgw-RouteTable

  SecVpcAz1TgwRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz1TgwRouteTable
      SubnetId: !Ref SecVpcAz1TgwSubnet

  SecVpcAz2TgwRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ2-Tgw-RouteTable

  SecVpcAz2TgwRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz2TgwRouteTable
      SubnetId: !Ref SecVpcAz2TgwSubnet

#---------------------------NAT Gateway------------------------------------#

# AZ1 NAT GW
  SecVpcAz1NatGatewayEIP:
     Type: AWS::EC2::EIP
     Properties:
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ1-NatGateway-EIP

  SecVpcAz1NatGateway:
     Type: AWS::EC2::NatGateway
     Properties:
        AllocationId: !GetAtt SecVpcAz1NatGatewayEIP.AllocationId
        SubnetId: !Ref SecVpcAz1PublicSubnet
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ1-NatGateway

# AZ2 NAT GW
  SecVpcAz2NatGatewayEIP:
     Type: AWS::EC2::EIP
     Properties:
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ2-NatGateway-EIP

  SecVpcAz2NatGateway:
     Type: AWS::EC2::NatGateway
     Properties:
        AllocationId: !GetAtt SecVpcAz2NatGatewayEIP.AllocationId
        SubnetId: !Ref SecVpcAz2PublicSubnet
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ2-NatGateway

  SecVpcAz1PrivateSubnetToInternetRoute:
     DependsOn: SecVpcAz1NatGateway
     Type: AWS::EC2::Route
     Properties:
        RouteTableId: !Ref SecVpcAz1PrivateRouteTable
        DestinationCidrBlock: '0.0.0.0/0'
        NatGatewayId: !Ref SecVpcAz1NatGateway

  SecVpcAz2PrivateSubnetToInternetRoute:
     DependsOn: SecVpcAz2NatGateway
     Type: AWS::EC2::Route
     Properties:
        RouteTableId: !Ref SecVpcAz2PrivateRouteTable
        DestinationCidrBlock: '0.0.0.0/0'
        NatGatewayId: !Ref SecVpcAz2NatGateway

#---------------------------添加路由------------------------------------#

# 公有子网添加默认路由去往IGW
  SecVpcAz1PublicSubnetToInternetRoute:
    Type: "AWS::EC2::Route"
    DependsOn: SecVpcIGW
    Properties:
     RouteTableId: !Ref SecVpcAz1PublicRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     GatewayId: !Ref SecVpcIGW

  SecVpcAz2PublicSubnetToInternetRoute:
    Type: "AWS::EC2::Route"
    DependsOn: SecVpcIGW
    Properties:
     RouteTableId: !Ref SecVpcAz2PublicRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     GatewayId: !Ref SecVpcIGW

#---------------------------SecVpc创建安全组------------------------------------#
  SecVpcSg:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: SG to test ping
      VpcId: !Ref SecVpc
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 22
        ToPort: 22
        CidrIp: 0.0.0.0/0
      - IpProtocol: icmp
        FromPort: -1
        ToPort: -1
        CidrIp: 0.0.0.0/0
      - IpProtocol: -1
        FromPort: -1
        ToPort: -1
        CidrIp: 10.100.0.0/16
      - IpProtocol: tcp
        FromPort: 3389
        ToPort: 3389
        CidrIp: 0.0.0.0/0
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpcSg


#---------------------------SecVpc创建EC2实例------------------------------------#

#--------------------------IAM Instance Role and Profile------------------------------------#
  ApplianceRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${AWS::StackName}-appliance-role"
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /

  AppliancePolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: AppServer
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - ec2:DescribeNetworkInterfaces
            Resource: '*'
      Roles:
        - !Ref ApplianceRole

  ApplianceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref ApplianceRole

#---------------------------SecVpc创建GWLB------------------------------------#

# Gateway Load Balancer (GWLB), Target Group, Listener
  Gwlb:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      LoadBalancerAttributes:
        - Key: load_balancing.cross_zone.enabled
          Value: true
      Name: gwlb1
      Type: gateway
      Subnets:
        - !Ref SecVpcAz1PrivateSubnet
        - !Ref SecVpcAz2PrivateSubnet
      Tags:
      - Key: Name
        Value: !Sub "${AWS::StackName}-gwlb-1"

  # Target Group:
  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: tg1
      Port: 6081
      Protocol: GENEVE
      TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
        Value: "20"
      VpcId: !Ref SecVpc
      HealthCheckPort: 80
      HealthCheckProtocol: HTTP
      TargetType: instance
      Targets:
        - Id: !Ref Appliance1
        - Id: !Ref Appliance2
      Tags:
      - Key: Name
        Value: !Sub "${AWS::StackName}-tg-1"

  # Listener:
  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
      - Type: forward
        TargetGroupArn: !Ref TargetGroup
      LoadBalancerArn: !Ref Gwlb

#---------------------------SecVpc创建EC2实例------------------------------------#


# EC2 Instances (Appliances acting as target for GWLB):
  Appliance1:
    DependsOn: [Gwlb, SecVpcAz1PrivateSubnetToInternetRoute]
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t2.micro
      IamInstanceProfile: !Ref ApplianceProfile
      SecurityGroupIds:
        - !Ref SecVpcSg
      SubnetId: !Ref SecVpcAz1PrivateSubnet
      Tags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-appliance-1"
      UserData:
        Fn::Base64: |
          #!/bin/bash -ex

          # Install packages:
          yum update -y;
          yum install jq -y;
          yum install httpd -y;
          yum install htop -y;
          sudo yum install iptables-services -y;

          # Enable IP Forwarding:
          echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/00-defaults.conf;
          sysctl -p /etc/sysctl.d/00-defaults.conf;

          # Configure hostname:
          hostnamectl set-hostname gwlb-target-1;

          # Configure SSH client alive interval for ssh session timeout:
          echo 'ClientAliveInterval 60' | sudo tee --append /etc/ssh/sshd_config;
          service sshd restart;

          # Set dark background for vim:
          touch /home/ec2-user/.vimrc;
          echo "set background=dark" >> /home/ec2-user/.vimrc;

          # Define variables:
          curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document > /home/ec2-user/iid;
          export instance_interface=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/);
          export instance_vpcid=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/$instance_interface/vpc-id);
          export instance_az=$(cat /home/ec2-user/iid |grep 'availability' | awk -F': ' '{print $2}' | awk -F',' '{print $1}');
          export instance_ip=$(cat /home/ec2-user/iid |grep 'privateIp' | awk -F': ' '{print $2}' | awk -F',' '{print $1}' | awk -F'"' '{print$2}');
          export instance_region=$(cat /home/ec2-user/iid |grep 'region' | awk -F': ' '{print $2}' | awk -F',' '{print $1}' | awk -F'"' '{print$2}');
          export local_az_gwlb_ip=$(aws --region $instance_region ec2 describe-network-interfaces --filters Name=vpc-id,Values=$instance_vpcid | jq ' .NetworkInterfaces[] | select(.AvailabilityZone=='$instance_az') | select(.InterfaceType=="gateway_load_balancer") |.PrivateIpAddress' -r);
          export remote_az_gwlb_ip=$(aws --region $instance_region ec2 describe-network-interfaces --filters Name=vpc-id,Values=$instance_vpcid | jq ' .NetworkInterfaces[] | select(.AvailabilityZone!='$instance_az') | select(.InterfaceType=="gateway_load_balancer") |.PrivateIpAddress' -r);


          # Start http and configure index.html:
          systemctl enable httpd;
          systemctl start httpd;
          touch /var/www/html/index.html;
          echo "<html>" >> /var/www/html/index.html
          echo "  <head>" >> /var/www/html/index.html
          echo "    <title>Gateway Load Balancer POC</title>" >> /var/www/html/index.html
          echo "    <meta http-equiv='Content-Type' content='text/html; charset=ISO-8859-1'>" >> /var/www/html/index.html
          echo "  </head>" >> /var/www/html/index.html
          echo "  <body>" >> /var/www/html/index.html
          echo "    <h1>Welcome to Gateway Load Balancer POC:</h1>" >> /var/www/html/index.html
          echo "    <h2>This is appliance running in $instance_az. Happy testing!</h2>" >> /var/www/html/index.html
          echo "  </body>" >> /var/www/html/index.html
          echo "</html>" >> /var/www/html/index.html

          # Start and configure iptables:
          systemctl enable iptables;
          systemctl start iptables;

          # Configuration below allows allows all traffic:
          # Set the default policies for each of the built-in chains to ACCEPT:
          iptables -P INPUT ACCEPT;
          iptables -P FORWARD ACCEPT;
          iptables -P OUTPUT ACCEPT;

          # Flush the nat and mangle tables, flush all chains (-F), and delete all non-default chains (-X):
          iptables -t nat -F;
          iptables -t mangle -F;
          iptables -F;
          iptables -X;

          # Configure nat table to hairpin traffic back to GWLB:
          iptables -t nat -A PREROUTING -p udp -s $local_az_gwlb_ip -d $instance_ip -i eth0 -j DNAT --to-destination $local_az_gwlb_ip:6081;
          iptables -t nat -A POSTROUTING -p udp --dport 6081 -s $local_az_gwlb_ip -d $local_az_gwlb_ip -o eth0 -j MASQUERADE;

          iptables -t nat -A PREROUTING -p udp -s $remote_az_gwlb_ip -d $instance_ip -i eth0 -j DNAT --to-destination $remote_az_gwlb_ip:6081;
          iptables -t nat -A POSTROUTING -p udp --dport 6081 -s $remote_az_gwlb_ip -d $remote_az_gwlb_ip -o eth0 -j MASQUERADE;

          # Save iptables:
          service iptables save;

  Appliance2:
    DependsOn: [ Gwlb, SecVpcAz2PrivateSubnetToInternetRoute ]
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t2.micro
      IamInstanceProfile: !Ref ApplianceProfile
      SecurityGroupIds:
        - !Ref SecVpcSg
      SubnetId: !Ref SecVpcAz2PrivateSubnet
      Tags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-appliance-2"
      UserData:
        Fn::Base64: |
          #!/bin/bash -ex

          # Install packages:
          yum update -y;
          yum install jq -y;
          yum install httpd -y;
          yum install htop -y;
          sudo yum install iptables-services -y;

          # Enable IP Forwarding:
          echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/00-defaults.conf;
          sysctl -p /etc/sysctl.d/00-defaults.conf;

          # Configure hostname:
          hostnamectl set-hostname gwlb-target-2;

          # Configure SSH client alive interval for ssh session timeout:
          echo 'ClientAliveInterval 60' | sudo tee --append /etc/ssh/sshd_config;
          service sshd restart;

          # Set dark background for vim:
          touch /home/ec2-user/.vimrc;
          echo "set background=dark" >> /home/ec2-user/.vimrc;

          # Define variables:
          curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document > /home/ec2-user/iid;
          export instance_interface=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/);
          export instance_vpcid=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/$instance_interface/vpc-id);
          export instance_az=$(cat /home/ec2-user/iid |grep 'availability' | awk -F': ' '{print $2}' | awk -F',' '{print $1}');
          export instance_ip=$(cat /home/ec2-user/iid |grep 'privateIp' | awk -F': ' '{print $2}' | awk -F',' '{print $1}' | awk -F'"' '{print$2}');
          export instance_region=$(cat /home/ec2-user/iid |grep 'region' | awk -F': ' '{print $2}' | awk -F',' '{print $1}' | awk -F'"' '{print$2}');
          export local_az_gwlb_ip=$(aws --region $instance_region ec2 describe-network-interfaces --filters Name=vpc-id,Values=$instance_vpcid | jq ' .NetworkInterfaces[] | select(.AvailabilityZone=='$instance_az') | select(.InterfaceType=="gateway_load_balancer") |.PrivateIpAddress' -r);
          export remote_az_gwlb_ip=$(aws --region $instance_region ec2 describe-network-interfaces --filters Name=vpc-id,Values=$instance_vpcid | jq ' .NetworkInterfaces[] | select(.AvailabilityZone!='$instance_az') | select(.InterfaceType=="gateway_load_balancer") |.PrivateIpAddress' -r);


          # Start http and configure index.html:
          systemctl enable httpd;
          systemctl start httpd;
          touch /var/www/html/index.html;
          echo "<html>" >> /var/www/html/index.html
          echo "  <head>" >> /var/www/html/index.html
          echo "    <title>Gateway Load Balancer POC</title>" >> /var/www/html/index.html
          echo "    <meta http-equiv='Content-Type' content='text/html; charset=ISO-8859-1'>" >> /var/www/html/index.html
          echo "  </head>" >> /var/www/html/index.html
          echo "  <body>" >> /var/www/html/index.html
          echo "    <h1>Welcome to Gateway Load Balancer POC:</h1>" >> /var/www/html/index.html
          echo "    <h2>This is appliance running in $instance_az. Happy testing!</h2>" >> /var/www/html/index.html
          echo "  </body>" >> /var/www/html/index.html
          echo "</html>" >> /var/www/html/index.html

          # Start and configure iptables:
          systemctl enable iptables;
          systemctl start iptables;

          # Configuration below allows allows all traffic:
          # Set the default policies for each of the built-in chains to ACCEPT:
          iptables -P INPUT ACCEPT;
          iptables -P FORWARD ACCEPT;
          iptables -P OUTPUT ACCEPT;

          # Flush the nat and mangle tables, flush all chains (-F), and delete all non-default chains (-X):
          iptables -t nat -F;
          iptables -t mangle -F;
          iptables -F;
          iptables -X;

          # Configure nat table to hairpin traffic back to GWLB:
          iptables -t nat -A PREROUTING -p udp -s $local_az_gwlb_ip -d $instance_ip -i eth0 -j DNAT --to-destination $local_az_gwlb_ip:6081;
          iptables -t nat -A POSTROUTING -p udp --dport 6081 -s $local_az_gwlb_ip -d $local_az_gwlb_ip -o eth0 -j MASQUERADE;

          iptables -t nat -A PREROUTING -p udp -s $remote_az_gwlb_ip -d $instance_ip -i eth0 -j DNAT --to-destination $remote_az_gwlb_ip:6081;
          iptables -t nat -A POSTROUTING -p udp --dport 6081 -s $remote_az_gwlb_ip -d $remote_az_gwlb_ip -o eth0 -j MASQUERADE;

          # Save iptables:
          service iptables save;


  SecVpcBastionLinux:
    Type: AWS::EC2::Instance
    Properties:
      IamInstanceProfile: !Ref BastionSsmProfile
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t3.nano
      NetworkInterfaces:
        - AssociatePublicIpAddress: true
          DeviceIndex: 0
          GroupSet:
            - Ref: SecVpcSg
          SubnetId: !Ref SecVpcAz1PublicSubnet
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-Bastion-Linux

# Create VPC Endpoint Service:

  VpcEndpointService:
    Type: AWS::EC2::VPCEndpointService
    Properties:
      GatewayLoadBalancerArns:
        - !Ref Gwlb
      AcceptanceRequired: false

# Create Lambda Custom Resource to retrieve VPC Endpoint Service Name:

  VpceServiceLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: arn:aws:logs:*:*:*
              - Effect: Allow
                Action:
                  - ec2:DescribeVpcEndpointServiceConfigurations
                  - ec2:DescribeVpcEndpointServicePermissions
                  - ec2:DescribeVpcEndpointServices
                Resource: "*"


  VpceServiceLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
        LogGroupName: !Sub /aws/lambda/${AWS::StackName}-service
        RetentionInDays: 1

  VpceServiceName:
    Type: AWS::Lambda::Function
    DependsOn: VpceServiceLogGroup
    Properties:
      FunctionName: !Sub ${AWS::StackName}-service
      Handler: "index.handler"
      Role: !GetAtt VpceServiceLambdaExecutionRole.Arn
      Code:
        ZipFile: |
          import json
          import logging
          import time

          import boto3
          import cfnresponse
          from botocore.exceptions import ClientError

          try:
              ec2 = boto3.client('ec2')
          except ClientError as e:
              logger.error(f"ERROR: failed to connect to EC2 client: {e}")
              sys.exit(1)

          def handler(event, context):
              logger = logging.getLogger()
              logger.setLevel(logging.INFO)
              logger.info('Received event: {}'.format(json.dumps(event)))

              responseData = {}
              responseStatus = cfnresponse.FAILED

              try:
                  serviceid = event["ResourceProperties"]["VpceServiceId"]
              except Exception as e:
                  logger.info('Attribute retrival failure: {}'.format(e))

              try:
                  if event["RequestType"] == "Delete":
                      responseStatus = cfnresponse.SUCCESS
                      cfnresponse.send(event, context, responseStatus, responseData)
              except Exception:
                  logger.exception("Signaling failure to CloudFormation.")
                  cfnresponse.send(event, context, cfnresponse.FAILED, {})

              if event["RequestType"] == "Create":
                  logger.info("Retrieving VPC Endpoint Service Name:")
                  try:
                      response = ec2.describe_vpc_endpoint_service_configurations(
                          Filters=[
                              {
                                  'Name': 'service-id',
                                  'Values': [serviceid]
                              }
                          ]
                      )
                  except Exception as e:
                      logger.info('ec2.describe_vpc_endpoint_service_configurations failure: {}'.format(e))

                  service_name = response['ServiceConfigurations'][0]['ServiceName']

                  time.sleep(120)

                  responseData['ServiceName'] = service_name
                  responseStatus = cfnresponse.SUCCESS
                  cfnresponse.send(event, context, responseStatus, responseData)
      Runtime: python3.7
      Timeout: 150

  RetrieveVpceServiceName:
    Type: Custom::RetrieveAttributes
    Properties:
      ServiceToken: !GetAtt VpceServiceName.Arn
      VpceServiceId: !Ref VpcEndpointService

# Create Gateway Load Balancer Endpoint:

  GwlbEndpoint1:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref SecVpc
      ServiceName: !GetAtt RetrieveVpceServiceName.ServiceName
      VpcEndpointType: GatewayLoadBalancer
      SubnetIds:
        - !Ref SecVpcAz1PrivateSubnet

  GwlbEndpoint2:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref SecVpc
      ServiceName: !GetAtt RetrieveVpceServiceName.ServiceName
      VpcEndpointType: GatewayLoadBalancer
      SubnetIds:
        - !Ref SecVpcAz2PrivateSubnet


#=========================================VpcA========================================#

  VpcA:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.110.10.0/16
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'
      Tags:
       - Key: Name
         Value: !Sub ${AWS::StackName}-VpcA

# 创建IGW并且关联到VPC
  VpcAIGW:
    Type: "AWS::EC2::InternetGateway"
    Properties:
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcAIGW

  VpcAAttachIgw:
    Type: "AWS::EC2::VPCGatewayAttachment"
    Properties:
      VpcId: !Ref VpcA
      InternetGatewayId: !Ref VpcAIGW

#---------------------------VpcA创建8个子网-------------------------------------#

# VpcA AZ1内创建公有子网
  VpcAAz1PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcA
      CidrBlock: 10.110.10.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcA-AZ1-Public-Subnet

# VpcA AZ2内创建公有子网
  VpcAAz2PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcA
      CidrBlock: 10.110.20.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcA-AZ2-Public-Subnet

# VpcA AZ1内创建APP子网
  VpcAAz1NlbSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcA
      CidrBlock: 10.110.30.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcA-AZ1-Nlb-Subnet

# VpcA AZ2内创建APP子网
  VpcAAz2NlbSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcA
      CidrBlock: 10.110.40.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcA-AZ2-Nlb-Subnet

# VpcA AZ1内创建GWLB子网
  VpcAAz1AppSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcA
      CidrBlock: 10.110.50.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcA-AZ1-App-Subnet

# VpcA AZ2内创建GWLB子网
  VpcAAz2AppSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcA
      CidrBlock: 10.110.60.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcA-AZ2-App-Subnet

# VpcA AZ1内创建TGW子网
  VpcAAz1TgwSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcA
      CidrBlock: 10.110.70.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcA-AZ1-TGW-Subnet

# VpcA AZ2内创建TGW子网
  VpcAAz2TgwSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcA
      CidrBlock: 10.110.80.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcA-AZ2-TGW-Subnet

#---------------------------VpcA创建路由表-------------------------------------#

# 公有子网路由表及关联
  VpcAAz1PublicRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcA
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ1-Public-RouteTable

  VpcAAz1PublicRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcAAz1PublicRouteTable
      SubnetId: !Ref VpcAAz1PublicSubnet

# IGW路由表及关联
  VpcAIgwRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcA
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-Ingress-RouteTable

  VpcAIgwRouteTableAssociation:
    Type: "AWS::EC2::GatewayRouteTableAssociation"
    Properties:
      GatewayId: !Ref VpcAIGW
      RouteTableId: !Ref VpcAIgwRouteTable

  VpcAAz2PublicRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcAAz1PublicRouteTable
      SubnetId: !Ref VpcAAz2PublicSubnet

# Nlb子网路由表及关联
  VpcAAz1NlbRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcA
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ1-Nlb-RouteTable

  VpcAAz1NlbRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcAAz1NlbRouteTable
      SubnetId: !Ref VpcAAz1NlbSubnet

  VpcAAz2NlbRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcA
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ2-Nlb-RouteTable

  VpcAAz2NlbRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcAAz2NlbRouteTable
      SubnetId: !Ref VpcAAz2NlbSubnet

# App子网路由表及关联
  VpcAAz1AppRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcA
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ1-App-RouteTable

  VpcAAz1AppRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcAAz1AppRouteTable
      SubnetId: !Ref VpcAAz1AppSubnet

  VpcAAz2AppRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcA
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ2-App-RouteTable

  VpcAAz2AppRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcAAz2AppRouteTable
      SubnetId: !Ref VpcAAz2AppSubnet

# Tgw路由表及关联
  VpcAAz1TgwRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcA
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ1-Tgw-RouteTable

  VpcAAz1TgwRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcAAz1TgwRouteTable
      SubnetId: !Ref VpcAAz1TgwSubnet

  VpcAAz2TgwRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcA
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ2-Tgw-RouteTable

  VpcAAz2TgwRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcAAz2TgwRouteTable
      SubnetId: !Ref VpcAAz2TgwSubnet

#---------------------------NAT Gateway------------------------------------#

# AZ1 NAT GW
  VpcAAz1NatGatewayEIP:
     Type: AWS::EC2::EIP
     Properties:
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ1-NatGateway-EIP

  VpcAAz1NatGateway:
     Type: AWS::EC2::NatGateway
     Properties:
        AllocationId: !GetAtt VpcAAz1NatGatewayEIP.AllocationId
        SubnetId: !Ref VpcAAz1NlbSubnet
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ1-NatGateway

# AZ2 NAT GW
  VpcAAz2NatGatewayEIP:
     Type: AWS::EC2::EIP
     Properties:
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ2-NatGateway-EIP

  VpcAAz2NatGateway:
     Type: AWS::EC2::NatGateway
     Properties:
        AllocationId: !GetAtt VpcAAz2NatGatewayEIP.AllocationId
        SubnetId: !Ref VpcAAz2NlbSubnet
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-AZ2-NatGateway

# VpcA 创建Endpoint
  VpcAEndpoint1:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref VpcA
      ServiceName: !GetAtt RetrieveVpceServiceName.ServiceName
      VpcEndpointType: GatewayLoadBalancer
      SubnetIds:
        - !Ref VpcAAz1PublicSubnet

  VpcAEndpoint2:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref VpcA
      ServiceName: !GetAtt RetrieveVpceServiceName.ServiceName
      VpcEndpointType: GatewayLoadBalancer
      SubnetIds:
        - !Ref VpcAAz2PublicSubnet


#---------------------------添加路由------------------------------------#

# IGW 路由表
  VpcAIgwRoute1:
    DependsOn: VpcAEndpoint1
    Type: AWS::EC2::Route
    Properties:
      DestinationCidrBlock: '10.110.30.0/24'
      VpcEndpointId: !Ref VpcAEndpoint1
      RouteTableId: !Ref VpcAIgwRouteTable

  VpcAIgwRoute2:
    DependsOn: VpcAEndpoint2
    Type: AWS::EC2::Route
    Properties:
      DestinationCidrBlock: '10.110.40.0/24'
      VpcEndpointId: !Ref VpcAEndpoint2
      RouteTableId: !Ref VpcAIgwRouteTable


# 公有子网添加默认路由去往IGW
  VpcAPublicSubnetToInternetRoute:
    Type: "AWS::EC2::Route"
    DependsOn: VpcAIGW
    Properties:
     RouteTableId: !Ref VpcAAz1PublicRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     GatewayId: !Ref VpcAIGW

# Nlb 添加默认路由去往App
  VpcANlbAz1Route1:
    DependsOn: VpcAEndpoint1
    Type: AWS::EC2::Route
    Properties:
      DestinationCidrBlock: '0.0.0.0/0'
      VpcEndpointId: !Ref VpcAEndpoint1
      RouteTableId: !Ref VpcAAz1NlbRouteTable

  VpcANlbAz2Route1:
    DependsOn: VpcAEndpoint2
    Type: AWS::EC2::Route
    Properties:
      DestinationCidrBlock: '0.0.0.0/0'
      VpcEndpointId: !Ref VpcAEndpoint2
      RouteTableId: !Ref VpcAAz2NlbRouteTable

# App添加默认路由去往IGW
  VpcAAz1AppSubnetToInternetRoute:
    Type: "AWS::EC2::Route"
    DependsOn: VpcAIGW
    Properties:
     RouteTableId: !Ref VpcAAz1AppRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     NatGatewayId: !Ref VpcAAz1NatGateway

  VpcAAz2AppSubnetToInternetRoute:
    Type: "AWS::EC2::Route"
    DependsOn: VpcAIGW
    Properties:
     RouteTableId: !Ref VpcAAz2AppRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     NatGatewayId: !Ref VpcAAz2NatGateway

#---------------------------VpcA创建安全组------------------------------------#
  VpcABastionVpcSg:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: SG to test ping
      VpcId: !Ref VpcA
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 22
        ToPort: 22
        CidrIp: 0.0.0.0/0
      - IpProtocol: icmp
        FromPort: -1
        ToPort: -1
        CidrIp: 0.0.0.0/0
      - IpProtocol: tcp
        FromPort: !Ref WebServerPort
        ToPort: !Ref WebServerPort
        CidrIp: 10.110.0.0/16
#      - IpProtocol: -1
#        FromPort: -1
#        ToPort: -1
#        SourceSecurityGroupId: !Ref VpcANlbSg
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcASg

#---------------------------VpcA创建EC2实例------------------------------------#

  VpcABastionLinux:
    Type: AWS::EC2::Instance
    Properties:
      IamInstanceProfile: !Ref BastionSsmProfile
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t3.nano
      NetworkInterfaces:
        - AssociatePublicIpAddress: true
          DeviceIndex: 0
          GroupSet:
            - Ref: VpcABastionVpcSg
          SubnetId: !Ref VpcAAz1PublicSubnet
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-Bastion-Linux

  VpcAApp1Linux:
    Type: AWS::EC2::Instance
    DependsOn: [VpcAEndpoint1, VpcANlbAz1Route1]
    Properties:
      IamInstanceProfile: !Ref BastionSsmProfile
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t3.nano
      NetworkInterfaces:
        - DeviceIndex: 0
          GroupSet:
            - Ref: VpcABastionVpcSg
          SubnetId: !Ref VpcAAz1AppSubnet
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-App1-Linux
      UserData:
        Fn::Base64:
          !Sub |
          #!/bin/bash
          yum update -y
          yum install -y httpd
          sed -i.bak 's/Listen 80/Listen ${WebServerPort}/g' /etc/httpd/conf/httpd.conf
          echo "<h2>Hello World from $(hostname -f)</h2>" > /var/www/html/index.html
          systemctl start httpd.service
          systemctl enable httpd.service

  VpcAApp2Linux:
    Type: AWS::EC2::Instance
    DependsOn: [VpcAEndpoint2, VpcANlbAz2Route1]
    Properties:
      IamInstanceProfile: !Ref BastionSsmProfile
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t3.nano
      NetworkInterfaces:
        - DeviceIndex: 0
          GroupSet:
            - Ref: VpcABastionVpcSg
          SubnetId: !Ref VpcAAz2AppSubnet
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcA-App2-Linux
      UserData:
        Fn::Base64:
          !Sub |
          #!/bin/bash
          yum update -y
          yum install -y httpd
          sed -i.bak 's/Listen 80/Listen ${WebServerPort}/g' /etc/httpd/conf/httpd.conf
          echo "<h2>Hello World from $(hostname -f)</h2>" > /var/www/html/index.html
          systemctl start httpd.service
          systemctl enable httpd.service

# -----------创建NLB-----------
  VpcAEc2TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 10
      HealthCheckProtocol: TCP
      HealthCheckTimeoutSeconds: 10
      HealthyThresholdCount: 3
      Name: VpcAEc2TargetGroup
      Port: !Ref WebServerPort
      Protocol: TCP
      TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
        Value: '20'
      - Key: preserve_client_ip.enabled
        Value: false
      Targets:
      - Id: !Ref VpcAApp1Linux
        Port: !Ref WebServerPort
      - Id: !Ref VpcAApp2Linux
        Port: !Ref WebServerPort
      UnhealthyThresholdCount: 3
      VpcId: !Ref VpcA
      Tags:
        - Key: Name
          Value: VpcA-Target-Group

  VpcANlbListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref VpcAEc2TargetGroup
      LoadBalancerArn: !Ref VpcANetworkLoadBalancer
      Port: !Ref WebServerPort
      Protocol: TCP

  VpcANetworkLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Sub ${AWS::StackName}-VpcA-NLB
      Type: network
      Scheme: internet-facing
      Subnets:
      - !Ref VpcAAz1NlbSubnet
      - !Ref VpcAAz2NlbSubnet

三、测试

3.1 访问NLB测试

复制NLB的DNS名称。

访问NLB的DNS名称测试业务连通性。

3.2 App抓包测试

连接App1

通过Session Manger连接。

在App上抓取TCP 8443的报文，因为Target Group关闭了Preserve client IP addresses属性，所以App会看到NLB的私有IP地址，安全组也可以只放行来自NLB子网的流量。

另外可以通过curl ifconfig.me查看App访问公网的IP地址，是NAT GW1的公网IP地址。

sudo -i
tcpdump -n port 8443

3.3 appliance抓包测试

在App1上ping公网地址。

在appliance上抓取GENEVE的报文，并使用grep对ICMP报文进行过滤。

root@gwlb-target-1 ~]# tcpdump -n -i eth0 port 6081 | grep ICMP

可以使用grep -i过滤特定主机报文

root@gwlb-target-1 ~]# tcpdump -n -i eth0 port 6081 | grep -i 8.8.8.8

抓包结果可以使用grep的\|进行“或”过滤。

root@gwlb-target-1 ~]# tcpdump -n -i eth0 port 6081 | grep "ICMP\|https"

3.4 防火墙高可用测试

在App1上发起对公网地址的访问，同时在appliance1和appliance2上开启抓包，确认流量目前通过appliance1转发。

Stop正在转发流量的appliance实例。

等待大概1分钟左右（依据是否为新建会话，还有协议切换时间不同），流量切换到另外一台防火墙。因为GWLB开启了Cross-zone load balancing，所以流量可以跨AZ，通过appliance2转发流量。