Automate Package Deployment via AWS System Manager

The task

Recently we got a task from the Company’s security team, to install 2 security agents which will be used to perform centralized security scans for all active AWS EC2 instances. here I will see how to use AWS Systems Manager for software distribution and installation for multiple AWS accounts and infrastructure at scale.

SSM features will be used :

Session Manager: ensure the EC2 instance has the SSM Agent installed and running and The instances need an IAM role with at least the AmazonSSMManagedInstanceCore policy attached
Run Command: send command and execute security agent software package installation scripts and command to varify post-installation status on remote instances for task automation

Prerequisites:

AWSCLI: programatically manage all the operation bellow.
SSM Agent: Ensure the SSM Agent is installed and running on all EC2 instances. Most Amazon Machine Images (AMIs) have the SSM Agent pre-installed.
IAM Role: Attach an IAM role to each instance with the AmazonSSMManagedInstanceCore policy.

Create and attach IAM role to EC2 instance for SSM to be able to perform action:

[cloudshell-user@ip-10-134-56-72 ~]$ vim trust-policy.json

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "Service": "ec2.amazonaws.com"
 },
 "Action": "sts:AssumeRole"
 }
 ]
}

[cloudshell-user@ip-10-134-56-72 ~]$ aws iam create-role --role-name SSMAccessRole --assume-role-policy-document file://trust-policy.json
{
 "Role": {
 "Path": "/",
 "RoleName": "SSMAccessRole",
 "RoleId": "AROA4MTWLTSHKK3QL3NOU",
 "Arn": "arn:aws:iam::85xxxxxx1342:role/SSMAccessRole",
 "CreateDate": "2024-06-25T00:29:07+00:00",
 "AssumeRolePolicyDocument": {
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "Service": "ec2.amazonaws.com"
 },
 "Action": "sts:AssumeRole"
 }
 ]
 }
 }
}

[cloudshell-user@ip-10-134-56-72 ~]$ aws iam attach-role-policy --role-name SSMAccessRole --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

[cloudshell-user@ip-10-134-56-72 ~]$ aws iam create-instance-profile --instance-profile-name SSMInstanceProfile
{
 "InstanceProfile": {
 "Path": "/",
 "InstanceProfileName": "SSMInstanceProfile",
 "InstanceProfileId": "AIPA4MTWLTSHF7KJRFLYC",
 "Arn": "arn:aws:iam::851xxxxxxx42:instance-profile/SSMInstanceProfile",
 "CreateDate": "2024-06-25T00:31:25+00:00",
 "Roles": []
 }
}

[cloudshell-user@ip-10-134-56-72 ~]$ INSTANCE_IDS=$(aws ec2 describe-instances --query "Reservations[*].Instances[*].InstanceId" --output text)
for INSTANCE_ID in $INSTANCE_IDS; do
> aws ec2 associate-iam-instance-profile --instance-id $INSTANCE_ID --iam-instance-profile Name=SSMInstanceProfile
> done

{
 "IamInstanceProfileAssociation": {
 "AssociationId": "iip-assoc-04e81626bf6bcd9b5",
 "InstanceId": "i-0762xxxxxxxcf2",
 "IamInstanceProfile": {
 "Arn": "arn:aws:iam::8517xxxxxxx42:instance-profile/SSMInstanceProfile",
 "Id": "AIPA4MTWLTSHF7KJRFLYC"
 },
 "State": "associating"
 }
}
{
 "IamInstanceProfileAssociation": {
 "AssociationId": "iip-assoc-09233165b3a53ca68",
 "InstanceId": "i-01a2xxxxxx048c",
 "IamInstanceProfile": {
 "Arn": "arn:aws:iam::851xxxx42:instance-profile/SSMInstanceProfile",
 "Id": "AIPA4MTWLTSHF7KJRFLYC"
 },
 "State": "associating"
 }
}
{
 "IamInstanceProfileAssociation": {
 "AssociationId": "iip-assoc-09236a8ee456e39cd",
 "InstanceId": "i-07axxxxxxb823",
 "IamInstanceProfile": {
 "Arn": "arn:aws:iam::85xxxxxxx42:instance-profile/SSMInstanceProfile",
 "Id": "AIPA4MTWLTSHF7KJRFLYC"
 },
 "State": "associating"
 }
}

[cloudshell-user@ip-10-134-56-72 ~]$ aws ssm describe-instance-information

Now verify from session manager to see if instances are there:

image tooltip here

Tag Instances

Tag EC2 instances to identify which instances need the security agent to be installed:

Key=InstallSecurityAgent, Value=True.

# List all instance IDs #: 
INSTANCE_IDS=$(aws ec2 describe-instances --query "Reservations[*].Instances[*].InstanceId" --output text)

# Tag all instances with Key=InstallSecurityAgent and Value=True
for INSTANCE_ID in $INSTANCE_IDS; do
 aws ec2 create-tags --resources $INSTANCE_ID --tags Key=InstallSecurityAgent,Value=True
done

SSM Run Command to Create a custom SSM document that contains the script to install the security agent

[cloudshell-user@ip-10-134-56-72 ~]$ vim install_security_agent.json
{
 "schemaVersion": "2.2",
 "description": "Install Security Agent",
 "mainSteps": [
 {
 "action": "aws:runShellScript",
 "name": "installSecurityAgent",
 "inputs": {
 "runCommand": [
 "curl -o /tmp/security-agent-installer.sh https://github.com/ZackZhouHB/zack-gitops-project/blob/editing/Python_scripts/security-agent-installer.sh",
 "chmod +x /tmp/security-agent-installer.sh",
 "/tmp/security-agent-installer.sh"
 ]
 }
 }
 ]
}

Create Document to execute the installation script on all tagged instances.

[cloudshell-user@ip-10-134-56-72 ~]$ aws ssm create-document \
 --name "InstallSecurityAgent" \
 --document-type "Command" \
 --content file://install_security_agent.json

{
 "DocumentDescription": {
 "Hash": "9e17a699d2d987134eb05f6b49a7c837161320b0ed42635b07928acc557970b5",
 "HashType": "Sha256",
 "Name": "InstallSecurityAgent",
 "Owner": "851725491342",
 "CreatedDate": "2024-06-25T01:51:36.666000+00:00",
 "Status": "Creating",
 "DocumentVersion": "1",
 "Description": "Install Security Agent",
 "PlatformTypes": [
 "Linux",
 "MacOS"
 ],
 "DocumentType": "Command",
 "SchemaVersion": "2.2",
 "LatestVersion": "1",
 "DefaultVersion": "1",
 "DocumentFormat": "JSON",
 "Tags": []
 }
}

[cloudshell-user@ip-10-132-90-150 ~]$ aws ssm send-command \
> --document-name "InstallSecurityAgent" \
> --targets "Key=tag:InstallSecurityAgent,Values=True" \
> --comment "Installing security agent on all instances with the specified tag" \
> --max-concurrency "50" \
> --max-errors "0" \
> --region ap-southeast-2
{
 "Command": {
 "CommandId": "edfc9e9b-5e74-4660-8335-a98eb48251f7",
 "DocumentName": "InstallSecurityAgent",
 "DocumentVersion": "$DEFAULT",
 "Comment": "Installing security agent on all instances with the specified tag",
 "ExpiresAfter": "2024-06-25T04:03:44.665000+00:00",
 "Parameters": {},
 "InstanceIds": [],
 "Targets": [
 {
 "Key": "tag:InstallSecurityAgent",
 "Values": [
 "True"
 ]
 }
 ],
 "RequestedDateTime": "2024-06-25T02:03:44.665000+00:00",
 "Status": "Pending",
 "StatusDetails": "Pending",
 "OutputS3Region": "ap-southeast-2",
 "OutputS3BucketName": "",
 "OutputS3KeyPrefix": "",
 "MaxConcurrency": "50",
 "MaxErrors": "0",
 "TargetCount": 0,
 "CompletedCount": 0,
 "ErrorCount": 0,
 "DeliveryTimedOutCount": 0,
 "ServiceRole": "",
 "NotificationConfig": {
 "NotificationArn": "",
 "NotificationEvents": [],
 "NotificationType": ""
 },
 "CloudWatchOutputConfig": {
 "CloudWatchLogGroupName": "",
 "CloudWatchOutputEnabled": false
 },
 "TimeoutSeconds": 3600,
 "AlarmConfiguration": {
 "IgnorePollAlarmFailure": false,
 "Alarms": []
 },
 "TriggeredAlarms": []
 }
}
(END)

Validate from Run Command console for the installation: image tooltip here

Create an SSM Document to Check the package installation status:

[cloudshell-user@ip-10-134-56-72 ~]$ vim verify.json

{
 "schemaVersion": "2.2",
 "description": "Install Security Agent",
 "mainSteps": [
 {
 "action": "aws:runShellScript",
 "name": "installSecurityAgent",
 "inputs": {
 "runCommand": [
 "apt list --installed | grep nfs-common",
 "apt list --installed | grep lrzsz"
 ]
 }
 }
 ]
}

Create this document using the AWS CLI:

[cloudshell-user@ip-10-134-56-72 ~]$ aws ssm create-document \
 --name "VerifyPackageInstallation" \
 --document-type "Command" \
 --content file://verify.json

[cloudshell-user@ip-10-134-56-72 ~]$ aws ssm send-command \
 --document-name "VerifyPackageInstallation" \
 --targets "Key=tag:InstallSecurityAgent,Values=True" \
 --comment "Check if Packages installed on all instances" \
 --max-concurrency "50" \
 --max-errors "0" \
 --region ap-southeast-2

Verify both “nfs-common” and “lrzsz”, we have 3 machines with “nfs-common” installed, and 2 instances with Ubuntu24.04 which did not get “lrzsz” installed.

image tooltip here

Conclusion

So now we can use AWS CLI and AWS System Manager to automate software deployment and verify the installation status.