While I was in an interview last week, the company heavily using Ansible to provision and configure their k8s cluster on AWS EC2, aim to have the most control and flexibility in case they need to move to another cloud provider.
Here I will use both Terraform and Ansible to automate K8S cluster creation and configuration on a few EC2 instances.
The tools and workflow
Major tools will include Ansible, terraform, together with a shell script.
Use terraform to create S3 as backend state file, create security groups and 4 ec2 instances ( 1 bastion, 1 master, 2 workers)
As Instances are created in AWS environment dynamically, Ansible ec2-dynamic-inventory approach is best way to manage them.
Using terraform “connection”, to ssh to the bastion host after creation, use provisoners “file” to upload shell script to bootstrap bastion host as a ansbile master, configure ec2-dynamic-inventory to fetch the other 3 instances for K8S.
Continue with terraform “file” and “remote-exec” to upload playbooks and with “inline” to execute playbooks to init k8s master node and join the worker node.
Finally another playbook to configure bastion to run kubectl
Create SGs and EC2s, with Bastion bootstrap via terraform
SSH to bastion host and run playbooks to init and join k8s nodes
Conclusion
Now we are able to spin up a 3-nodes K8S cluster on EC2 instances in about 20 minutes.
For lab purpose, running K8S on EC2 instances with default VPC can be cheaper and flexible option compare to EKS, as AWS managed EKS require additional VPC and unable to change node instance type unless delete the existing node group.