Securely join Kafka purchasers operating exterior AWS to Amazon MSK with IAM Roles Wherever

Kafka purchasers which can be operating exterior of AWS (on-premises atmosphere or different clouds) would require an IAM person with a long-lived entry key to be offered as a part of their codebase or of their server configuration. From a safety perspective, there may be an extra threat if anybody features entry to these long-term credentials, as they’d have entry to that AWS account.

On this put up, we exhibit tips on how to use AWS IAM Roles Wherever to request momentary AWS safety credentials, utilizing x.509 certificates for shopper functions which allows safe interactions with an Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster. The answer described on this put up is appropriate with each Amazon MSK Provisioned and Serverless clusters.

Introduction to AWS IAM Roles Wherever

AWS Id and Entry Administration (IAM) Roles Wherever lets you get hold of momentary safety credentials in IAM for workloads operating exterior of AWS, similar to servers, containers, and functions.

Through the use of IAM Roles Wherever, your workloads can make the most of the identical IAM insurance policies and roles utilized by AWS functions to entry AWS sources. This eliminates the necessity to handle long-term credentials for kafka purchasers operating exterior AWS. By associating a number of roles with a profile and enabling IAM Roles Wherever to imagine these roles, your functions can make use of the shopper certificates issued by your Certificates Authorities (CAs) to securely provoke requests to AWS. Consequently, your functions get hold of momentary credentials, granting them entry to the AWS atmosphere.

IAM entry management for Amazon MSK lets you handle each authentication and authorization in your Amazon MSK cluster at no further value. This eliminates the need of utilizing separate mechanisms for authentication and authorization. We advocate Amazon MSK prospects use IAM Entry Management except they’ve a selected want for utilizing mutual TLS or SASL/SCRAM authN/Z.

Within the following sections, we present you tips on how to implement a safe Kafka shopper machine with an in depth step-by-step tutorial utilizing an AWS IAM Roles Wherever to attach with a MSK Cluster.

Answer overview

The next diagram illustrates the answer structure.

The stream of the structure is as follows:

1. The session token question out of your shopper machine is directed to an AWS IAM Roles Wherever endpoint, facilitated by the trade of X.509 certificates.
2. IAM Roles Wherever validates the certificates and retrieves a brief session token from STS, which is then returned to the shopper machine.
3. In Amazon MSK Provisioned, the MSK shopper machine connects to the AWS Transit Gateway or AWS Community Load Balancer in your VPC over AWS VPN or AWS Direct Join. For extra info, confer with Safe connectivity patterns to entry Amazon MSK.
4. In Amazon MSK Serverless, the MSK shopper machine connects to the interface VPC endpoint in your VPC over AWS VPN or AWS Direct Join. For extra info, confer with Hook up with Amazon MSK Serverless out of your on-premises community.
5. In Amazon MSK Serverless , the interface endpoint is a group of a number of elastic community interfaces with a non-public IP tackle inside your account. It serves because the entry level for site visitors directed in direction of a MSK Serverless service.

Conditions

The directions offered on this put up assume that you’re already acquainted with the method of making an MSK serverless cluster and a shopper machine. Moreover, it’s presumed that you’ve got efficiently completed the next duties:

1. Create an Amazon MSK serverless cluster or Create an Amazon MSK Provisioned Cluster
2. Create a MSK shopper machine in your on-prem knowledge middle or a VPC from one other AWS account.
3. Set up community connectivity between on premises and the Amazon MSK Serverless Cluster or Set up community connectivity between on premises and the Amazon MSK Provisioned Cluster

Configure AWS IAM Roles Wherever

To allow IAM Roles Wherever in your on-premises Kafka shopper machine, you will need to configure two important parts in AWS Roles Wherever: the belief anchor and the profile. The belief anchor establishes the belief relationship between Roles Wherever and your certificates authority. This belief is utilized for authenticating certificates to acquire credentials for an IAM position. Profiles are predefined units of permissions which can be utilized as soon as profitable authentication with Roles Wherever has been achieved.

Step 1: Generate a CA

An X.509 certificates performs an vital position in facilitating communication between the shopper machine and Roles Wherever. You should use Public Key Infrastructure (PKI) platform of your alternative to determine a certificates authority (CA).

Should you favor to generate your individual X.509 shopper certificates, you may confer with the directions outlined in IAM Roles Wherever with an exterior certificates authority to information you thru the method.

For simplicity of this instance, we use an AWS Non-public CA:

Navigate to the AWS Non-public CA console.

Create a Root CA

1. Select Root as CA kind choice and put your group title and group unit title.
2. Select default RSA 2048 key algorithm.
3. Select Create CA button to generate a non-public the CA and set up the certificates.

Create a Subordinate CA

1. Select Subordinate as CA kind choice.
2. Select default RSA 2048 key algorithm.
3. Select Create CA button.
   
   
4. Acquire the CSR from the subordinate CA and have it signed by the basis CA.
   
   

This CA can be used for issuing certificates to IAM Roles Wherever.

For producing a extra secured and auto-renewed AWS personal CA, confer with Process for making a CA and Tips on how to construct a CA hierarchy.

Step 2: Configure anchor

1. Go to Roles Wherever console and open the Create a belief anchor web page.
2. Present a reputation in your belief anchor and choose the personal CA that we created in step 1. Should you favor to make use of your individual exterior CA, select the Exterior certificates bundle choice and supply the mandatory certificates bundle.
3. Select create a belief anchor button to complete the method.

Step 3: Create and configure a job that trusts IAM Roles Wherever

Now we create a job that you really want your on-premises Kafka shopper machine to imagine after authenticating to IAM Roles Wherever.

1. The belief coverage of the position ought to comprise the next:

   {
     "Model": "2012-10-17",
     "Assertion": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "rolesanywhere.amazonaws.com"
         },
         "Action": [
           "sts:AssumeRole",
           "sts:SetSourceIdentity",
           "sts:TagSession"
         ],
         "Situation": {
           "StringEquals": {
             "aws:PrincipalTag/x509Subject/CN": "specific-certificate-common-name"
           }
         }
       }
     ]
   }

2. For this demo, create the next coverage and fix it to the position:

   {
       "Model":"2012-10-17",		 	 	 
       "Assertion": [
           {
               "Effect": "Allow",
               "Action": [
                   "kafka-cluster:Connect",
                   "kafka-cluster:AlterCluster",
                   "kafka-cluster:DescribeCluster"
               ],
               "Useful resource": [
                   "arn:aws:kafka:::cluster//"
               ]
           },
           {
               "Impact": "Enable",
               "Motion": [
                   "kafka-cluster:CreateTopic",
                   "kafka-cluster:DescribeTopic",
                   "kafka-cluster:WriteData",
                   "kafka-cluster:ReadData",
                   "kafka-cluster:AlterGroup",
                   "kafka-cluster:DescribeGroup"
               ],
               "Useful resource": [
                   "arn:aws:kafka:::cluster/ /",
                   "arn:aws:kafka:::topic/msk-// ",
                   "arn:aws:kafka:::group///"
               ]
           }
       ]
   }

Step 4: Setup profile

1. Navigate again to the Roles Wherever console.
2. Underneath Profiles, select Create a profile.
3. Enter a reputation for the profile.
4. Choose the position we created in Step 3 and create the Roles Wherever profile.

Step 5: Take a look at the shopper machine

Now that we have now efficiently arrange Roles Wherever by making a belief anchor and a profile, the subsequent step is to check the shopper machine’s communication with Roles Wherever. This entails retrieving a session token and establishing communication with the MSK dealer.

1. Request a non-public certificates from the CA we created in Step 1 and export the shopper certificates for use within the shopper machine.
2. Create a .pem file and duplicate all of the certificates contents into this .pem file(e.g. private_key. pem) and run under command to generate a decrypted model of certificates.

   openssl rsa -in private_key.pem -out decrypted_private_key.pem

   

3. Obtain the credential helper and use this signing helper instrument to check and make sure the performance out of your shopper machine. We provide the ARNs of the belief anchor and profile of Roles Wherever, and the position we created in IAM.

   ./aws_signing_helper credential-process 
   --certificate /path/to/certificates.pem 
   --private-key /path/to/decrypted_private_key.pem 
   --trust-anchor-arn  
   --profile-arn  
   --role-arn  
   --region 

   You need to obtain the session credentials efficiently from IAM Roles Wherever.

   

4. After verifying the profitable setup, proceed to replace or create the ~/.aws/config file. Add the signing helper as a credential_process on this file to allow unattended entry for the on-premises server.
   
   [default]

   credential_process = ./aws_signing_helper credential-process 
   --certificate /path/to/certificates.pem 
   --private-key /path/to/decrypted_private_key.pem 
   --trust-anchor-arn  
   --profile-arn  
   --role-arn 
   --region 

As soon as all steps are performed, it is best to be capable of see the Kafka shopper speaking to the MSK dealer.

./kafka-topics.sh --create 
--bootstrap-server  
--command-config  
--replication-factor  
--partitions  
--topic 

Clear up

To cease incurring prices, it’s endorsed to manually delete the IAM Function, Profile, Belief Anchor, Insurance policies, requested certificates in ACM and created certificates in AWS Non-public CA.

aws delete-role --role-name 

aws delete-profile --profile-id 

aws delete-trust-anchor --trust-anchor-id 

aws acm delete-certificate --certificate-arn 

aws acm-pca revoke-certificate --certificate-authority-arn  --certificate-serial  --revocation-reason  

aws acm-pca delete-certificate --certificate-authority-arn  --certificate-serial 

Conclusion

On this put up, we confirmed you tips on how to make the most of AWS IAM Roles Wherever to generate momentary session tokens for accessing MSK brokers from shopper machines exterior of AWS. By implementing this strategy, the safety posture of Kafka purchasers connecting to MSK from exterior of AWS are enhanced, permitting prospects with stringent safety necessities to confidently undertake MSK.

When you have any questions, you can begin a brand new thread on AWS re:Put up or attain out to AWS Assist.

In regards to the authors