Overview
Well, nothing is really timeless, but hopefully this article is still more revelant a year from now, compared to some of the other resouces for getting to grips with EKS! There are plenty of resources out there for managing AWS EKS with Terraform, which is great, it shows the level of interest in running Kubernetes on AWS, but it has its pitfalls too. Primarily because the Terraform modules and resources for use with EKS are changing so fast that it can be difficult to actually find something that works still effectively, and is current. You can literally spend hours going through Youtube videos just to find that the video you were watched was created before an update to a module and no longer works.
So what can you do? The best thing is to have an understanding of the EKS/Kubernetes fundamentals first. That way you can then go back to the source (the Terraform registry) and plug the modules into place, regardless of how much they may have changed.
By the way, the incredible rate of change is a good thing, but it just makes it hard to get started at times.
Components
So how do we start? We will start with a VPC, and then build up from there. So we will need the following:
- The VPC and subnets
- The EKS Cluster
- The management interface for EKS
- An EKS deployment (such as Nginx)
Those are fairly easy size chunks to treat relatively separately.
The VPC and Subnets
For that, you can build your VPC from scratch. So that would be doing something like this:
provider "aws" {
region = "eu-west-2"
}
resource "aws_vpc" "greg-vpc-1" {
cidr_block = "192.168.0.0/20"
assign_generated_ipv6_cidr_block = true
tags = {
Name = "greg-vpc-1",
Terraform = "true",
TF-Module = "base",
VPC = "greg-vpc-1"
}
}
# Subnets
variable "public-subnets" {
default = {
greg-public-1 = {
cidr_block = "192.168.0.0/24"
availability_zone = "eu-west-2a"
map_public_ip_on_launch = true
}
greg-public-2 = {
cidr_block = "192.168.1.0/24"
availability_zone = "eu-west-2b"
map_public_ip_on_launch = true
}
}
}
variable "private-subnets" {
default = {
greg-private-1 = {
cidr_block = "192.168.2.0/24"
availability_zone = "eu-west-2a"
map_public_ip_on_launch = false
}
greg-private-2 = {
cidr_block = "192.168.3.0/24"
availability_zone = "eu-west-2b"
map_public_ip_on_launch = false
}
}
}
resource "aws_subnet" "public-subnet" {
for_each = var.public-subnets
vpc_id = aws_vpc.greg-vpc-1.id
cidr_block = each.value.cidr_block
availability_zone = each.value.availability_zone
map_public_ip_on_launch = each.value.map_public_ip_on_launch
tags = {
Name = join(" - ",tolist(["public", each.value.cidr_block, each.value.availability_zone]))
}
}
resource "aws_subnet" "private-subnet" {
for_each = var.private-subnets
vpc_id = aws_vpc.greg-vpc-1.id
cidr_block = each.value.cidr_block
availability_zone = each.value.availability_zone
map_public_ip_on_launch = each.value.map_public_ip_on_launch
tags = {
Name = join(" - ",tolist(["private", each.value.cidr_block, each.value.availability_zone]))
}
}
# Internet Gateway
resource "aws_internet_gateway" "greg-igw" {
vpc_id = aws_vpc.greg-vpc-1.id
tags = {
Name = "greg-igw",
Terraform = "true",
TF-Module = "base",
VPC = "greg-vpc-1"
}
}
# NAT Gateway
resource "aws_eip" "eip_ngw" {
vpc = true
depends_on = [aws_internet_gateway.greg-igw]
}
resource "aws_nat_gateway" "greg-natgw" {
depends_on = [aws_internet_gateway.greg-igw]
allocation_id = aws_eip.eip_ngw.id
subnet_id = aws_subnet.public-subnet["greg-public-1"].id
tags = {
Zone = "Private"
Terraform = "True"
TF-Module = "Base"
VPC = "greg-vpc-1"
}
}
# Route tables
resource "aws_route_table" "greg-Public-rt" {
vpc_id = aws_vpc.greg-vpc-1.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.greg-igw.id
}
tags = {
Name = "greg-vpc-1-Public-rt",
Terraform = "true",
TF-Module = "base",
VPC = "greg-vpc-1"
}
}
# Route table association
resource "aws_route_table_association" "public_rt_assocation-1" {
subnet_id = aws_subnet.public-subnet["greg-public-1"].id
route_table_id = aws_route_table.greg-Public-rt.id
}
resource "aws_route_table_association" "public_rt_assocation-2" {
subnet_id = aws_subnet.public-subnet["greg-public-2"].id
route_table_id = aws_route_table.greg-Public-rt.id
}
That is going to give you a basic VPC, with public and private subnets, route tables, a NAT GW, and an IGW. None of that should be a mystery, and it is all fairly fundemntal. You can hopefully put all of that together yourself fairly easily.
Alternatively, you could decide that you want to use a module. This code will do something similar, and is much shorter. It is creating a couple of private/public subnets, and creating a NAT gateway.
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "3.18.1"
name = "greg-vpc"
cidr = "10.0.0.0/16"
azs = data.aws_availability_zones.available.names
private_subnets = ["10.0.0.0/19", "10.0.32.0/19"]
public_subnets = ["10.0.64.0/19", "10.0.96.0/19"]
enable_nat_gateway = true
single_nat_gateway = true
one_nat_gateway_per_az = false
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Environment = "Staging"
}
}
It will abstract a lot of the underlying code away from you, which is great, as long as you understand the fundamentals. Once you do, use the module, it is much less typing. The problem is that if the module changes, and you get errors, an understanding of what the VPC looks like will help you immensely as you try and figure out what is wrong, or what is missing. So modules are fantastic, but having a good idea of what the module is actually doing is also very important.
You should save the TF code in a file, and use Terraform to apply it.
The EKS Cluster
So when you have have an account with an VPC, you can then look at creating an EKS cluster. You can create your EKS cluster with Terraform, and you then have a lot of options about how you manage the cluster itself. In other words, you can use TF to create, and then the standard Kubernetes tools to manage it. So you can use kubectl, you can use kubectl with Terraform (as a provider), or you can use the Kubernetes provider with Terraform. You want to separate management of the cluster from creation of the cluster though to make things easy for yourself
So when you create your cluster with Terraform, do the create first, before you dive too deep into management of it.
There is a Terraform EKS module, which is a big help here. You can jump in and use the native Terraform resources, but the module is going to simplify so much of this.
Here is an example of some code that uses the Terraform module to create an EKS cluster.
module "eks" {
version = "19.3.1"
source = "terraform-aws-modules/eks/aws"
cluster_name = var.cluster_name
cluster_version = "1.23"
cluster_endpoint_private_access = true
cluster_endpoint_public_access = true
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
enable_irsa = true}
eks_managed_node_groups = {
#blue = {}
green = {
instance_types = ["t3.small"]
min_size = 1
max_size = 1
desired_size = 1
labels = {
role = "general"
}
}
}
}
It is referring to the VPC module used previously to get the VPC and the subnets, so if you don't want to use the VPC module, you can simply replace the values here with the VPC and subnets of your choice. Note too, that it is creating a Managed Node Group called "green". You might also want to create one called "blue", and if you did, you could simply comment out that line. In that case though, the "blue" group would just use all of the default values. In the definition for "green", we are overriding some of the default values to use a particular instance size, and desired size.
One thing to note is that this module in particular changes a lot. Much more than the VPC module. That means that there is a significant chance that if you are looking at this later, the code above may not work because some of the settings might change.
Do not dispair! The whole point of trying to separate this out is to simplify the amount of change that you need to tackle when trying to get the hang of something like EKS. So, if the EKS module has changed and you get errors running the above code, go to the EKS module website and take a look. You can drop in code from whatever example is currently on the site without affecting anything you have done up to this point (with the VPC), and hopefully anything after!
One thing to check though is that you are using the correct version. The definition above for 19.3.1 should always work for 19.3.1, so just make sure you include that line. Of course you might want or need to use a newer version!
Don't be afraid to look at various newer versions of the module though You just need the code to create the EKS cluster.
You can have this code in the same folder (same file or different) as your code for your VPC. Doing so will mean that the one terraform apply command will do both the VPC and EKS. After the apply command, you should see the resources created.
You will then be able to see your cluster in the console.
The management interface for EKS
With management of your cluster, if you are going to use Terraform to create a provider, you will want a separate folder structure for this, for the simple reason that when you configure a kubernetes provider like we will do here, it will reach out to the EKS cluster endpoint. If you try and do all that with the same apply it will fail, so it is easier in a separate folder.
We are going to do this using the kubernetes provider in Terraform.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.20.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = ">= 2.0.1"
}
}
}
data "terraform_remote_state" "eks" {
backend = "local"
config = {
path = "../EKS/terraform.tfstate"
}
}
# Retrieve EKS cluster information
provider "aws" {
region = data.terraform_remote_state.eks.outputs.region
}
data "aws_eks_cluster" "cluster" {
name = data.terraform_remote_state.eks.outputs.cluster_name
}
provider "kubernetes" {
host = data.aws_eks_cluster.cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
args = [
"eks",
"get-token",
"--cluster-name",
data.aws_eks_cluster.cluster.name
]
}
}
You will need to edit the code for the path to point to the location where the terraform state file for your cluster is. That is the code you created above for EKS.
There are other ways that you can connect to your cluster. Besides the Terraform provided kubernetes resource, you can also use kubectl. There is also kustomize, but if you use Terraform, the kubernetes provider is probably the way to go.
One good thing is that however you created your Kubernetes cluster on EKS, you can choose a different method for management. The code above is obviously going to read an existing TF state to get the information about the cluster, but even if your cluster wasn't created in Terraform, you can still use the kubernetes provider. What you will need to do though, is change the host line to grab the endpoint for your cluster another way, and the same with the certificate.
An EKS deployment
A deployment in Kubernetes is basically a desired state of a pods, stored in a declarative configuration. As an example, if you are hosting a webplatform, your deployment could be your pods running your webserver software, in a defined configuration. Kubernetes uses YAML for this configuration normally. There is a page here from the Kubernetes website that explains it.
The example on that page deploys a simple nginx deployment, three pods running nginx. We can do the same on EKS. The configuration for the deployment is basically the same, but we wrap it in a Terraform resources.
resource "kubernetes_deployment" "nginx" {
metadata {
name = "scalable-nginx-example"
labels = {
App = "ScalableNginxExample"
}
}
spec {
replicas = 2
selector {
match_labels = {
App = "ScalableNginxExample"
}
}
template {
metadata {
labels = {
App = "ScalableNginxExample"
}
}
spec {
container {
image = "nginx:1.7.8"
name = "example"
port {
container_port = 80
}
resources {
limits = {
cpu = "0.5"
memory = "512Mi"
}
requests = {
cpu = "250m"
memory = "50Mi"
}
}
}
}
}
}
}
Save that to a file and apply it.
You should be able to see pods using kubectl, although you will need to run aws eks update-kubeconfig first.
You now need a service, which will expose your nginx deployment. This can be done with a Kubernetes loadbalancer. Again, there are plenty of ways we can do this, but we can of course use Terraform.
resource "kubernetes_service" "nginx" {
metadata {
name = "nginx-example"
}
spec {
selector = {
App = kubernetes_deployment.nginx.spec.0.template.0.metadata[0].labels.App
}
port {
port = 80
target_port = 80
}
type = "LoadBalancer"
}
}
You can also define an output. This can of course go into the same file as your deployment, or you might want an outputs.tf (in the same folder)
output "lb_ip" {
value = kubernetes_service.nginx.status.0.load_balancer.0.ingress.0.hostname
}
Once you have applied those configurations with Terraform, you can run terraform apply and you will get the hostname of the loadbalancer, that you can then use to connect to. Give it a minute or so, and then conect using HTTP (not HTTPS).
Summary
So hopefully you have seen that you can mix and match how you create your EKS cluster, how you manage it, and how you create a deployment (and a service). If you are trying to learn EKS (or Kubernetes), it can be really helpful when going through guides and tutorials to try and understand exactly what commands/instructions (whether TF, kubectl, CloudFormation, or whatever) are being used for each specific purpose. Understanding how they then fit together can also help digest large streams of information quicker.
You can download the files I used here. First, the TF files for the creation of the EKS cluster (which I put in a folder called EKS, and that path is referenced in the provider). These are the files that I used for the management and the deployment (including the provider), which I placed in a separate folder.
You will need to run terraform init for each folder before attempting terraform apply or terraform plan.
Note that it costs money to run an EKS cluster, as well as for the resources you spin up.
comments powered by Disqus