Overivew
I recently came across a problem where someone was trying to use Jenkins, but it was constantly getting attacked, and vulnerabiliities were being exploited. Once exploited, the Jenkins instance was being used for crypto-mining, which not only is bad for the devlopers trying to use the platform, but bad for your bottom line.
So I spent a bit of time going over some of the options to secure what they have, and we explored how you can use AWS services and features to keep your application more secure than (in most cases) the same system on-premise.
AWS has an enormous number of options for securing your workloads, and we were able to recommend using just a few of those to provide much more secure system, without affecting accessibility too much. This made the platform significantly more secure, and allowed much more confidence in detecting and responding to these incidents in future.
The Existing Environment
Jenkins was running as a container on docker, on an EC2 instance running Ubuntu, using an AWS provided AMI. This was in a public security group, and had access allowed for both SSH and HTTP. They were storing the Jenkins image in ECR.
Users, and admins, were connecting from all around the world.
Connectivity
There were two main types of connections into the environment. They had the Jenkins users, who were connecting to the system via the web interface to running jobs, and they had admins, who were using SSH to connect to the instance that Jenkins was running on.
It wasn't possible to restrict either of these users geographically, and the user of a VPN wasn't considered acceptable for this use case. All connections had to come from the Internet.
Image Security
The images were being stored in ECR, but there was no scanning done, either within the registry, or once the images had been deployed. This was a significant hole, as it meant that that any exploit was only visible when the users noticed that the password for the Jenkins admin account had been changed. This was obviously a terrible position to be in.
Changes
The changes to be applied focused on several areas.
Firstly, network security had to be improved. The Jenkins instance needed to be moved away from a Public Subnet to a Private Subnet, and Security Groups used to significantly limit the access between subnets. Additionally, use of the Route 53 Resolver DNS Firewall would allow only filtered access to the Internet. This would mean that even if the instance was accessed again via an exploit, there was only specific DNS addresses on the Internet that would be resolvable.
Secondly, access to Jenkins had to be changed. The use of an Application Load Balancer, would allow HTTP requests to be done using HTTPS that terminated on the ALB (with certificates stored in AWS Certificate Manager), and WAF functionality could also be enabled on the ALB. No HTTP would be allowed for the Jenkins users.
For the admins, who were previously accessing the instance via SSH exposed to the Internet, this would be replaced with AWS Systems Manager Session Manager. Admins would establish connections via a web session to AWS Systems Manager, and then open a console via the browser to get access to the instance. This would allow them console access if they required it, but would not allow direct SSH from the Internet. This would significantly reduce the attack surface of the system.
Thirdly, additional AWS services were enabled to improve security posture. Amazon Inspector was enabled, with agents placed on the instance and container to monitor for any suspicious acticity. The admins could then be alerted to any findings, rather than having to wait for instrusion events to be discovered. AWS GuardDuty would also be enabled to provide account level monitoring of Cloudtrail logs and VPC Flow logs amongst other sources. Any findings here would provided an even greater level of visibility.
Lastly, ECR Image Scanning could be enabled to help ensure that images are secure, even before they are deployed. If anything was added to an image during the build process, or any vulnerabilities existed in vendor provided images, ECR Image Scanning could provide visibility. Enchanged mode also integrated directly with Amazon Inspector.
Conclusion and Takeaways
These changes gave the customer a much more secure platform, without compromising user access, or having significant increases in costs. They were quick to implement, and provided building blocks that can then be used with other systems and workloads.
When deploying anything in AWS, it is always important to make security the priority from day one, and the job never finishes. There are always further improvements that can be made, and there should be regular reviews to ensure this happens.
When putting in any system, not just Jenkins, as a start, consider the following:
- Consider the security of the application itself (users/passwords, etc.)
- Plan for defence in depth, so not just at the perimeter
- Limit traffic to only the required source/destination, and only on the required ports
- Ensure you are protecting data in transit (so use HTTPS not HTTP) as well as at rest
- Ensure tracability - lots of logging at the application layer, VPC flow logs, load balancers, etc.
- Have a way to detect issues ASAP - there are plenty of AWS tools such as Inspector and Guardduty to help!
- Automate your configuration and as many responses as you can
For more information, a good place to start is the AWS Well Architected Security Pillar, or the AWS Security White Paper. Also for your criticsl applications, consider talking to AWS about the SIP program.
comments powered by Disqus