My last post looked at how I use GitLab CI/CD pipelines to automatically deploy my blog to AWS. One thing I mentioned was the use of GitLab environment variables to pass AWS keys to the runner so it had the relevant credentials to execute my job. This removed the need to hard code any sensitive credentials into places they shouldn’t be (i.e. repo files), but also has a few shortcomings.
Within GitLab, maintainer and owner roles can view project variables, something which your security team may not want from a least privilege/separation of duties point of view. What if a developer leaves the organisation and takes knowledge of the AWS keys with them? They may have their GitLab access revoked as part of an off-boarding process, but they could still potentially do some damage to your AWS environment directly using known API keys.
Of course, hopefully you are frequently (and automatically) rotating keys, but I’m sure there are many places that are not. So it got me wondering - what if I could push changes to my AWS environment without the need for any credentials being stored in GitLab at all? Obviously overkill for a single personal blog, but for an enterprise environment, potentially very useful.
Previously, I talked about using GitLab runners to build HTML (using Hugo) and deploy the resulting files to an S3 bucket. GitLab runners are basically servers with some software on that are capable of running some code as part of a pipeline job. That code could shell commands, SSH commands, docker images etc.
GitLab Runners come in two flavours - “shared runners” (an autoscaling platform that GitLab provides on top of GCP) and “specific runners” (software that you install on a server of your choosing). Shared runners are great for a wide variety of uses, but there are times you will need a specific runner.
As an example, let’s say you are moving to an Infrastructure as Code approach (IaC), and you’d like to do the same for your on-premise network as well (such as switches, routers, firewalls). In this instance you would need a specific runner so you could deploy it in a network that has reachability to those devices. You could then use software on that server such as Ansible to deploy changes to the network, whilst securely maintaining your code in a GitLab versioning system.
So thinking back to the original question, could I use a specific runner within AWS to securely publish code to S3 without the need for API credentials being passed? (Hint: the answer is yes!)
AWS IAM Roles
Think of an IAM role as being a bit like an IAM user - but without any credentials. Existing users (who have permissions to do so) can “assume” a role and access resources with a different set of permissions via temporary credentials. Roles can also be assigned directly to certain AWS services, such as an EC2 instance. Attaching a role in this way allows the code running on that box to access other AWS resources, without the need for credentials, whilst applying fine-grained control of permissions using IAM policies.
Roles can obviously be a good thing and a bad thing, depending on the use-case. It prevents the need for API credentials which can help prevent any old Joe Bloggs (who managed to get hold of your AWS keys) from being able to just access resources across the Internet. But if the box that uses that role gets compromised, then it is going to have access to other resources without the need for authentication. So clearly - it is vitally important you harden these boxes, restrict access to them and heavily monitor them.
The High-Level Workflow
The high-level workflow we are going to look at in this post will operate as follows:
- I commit a change to my repository as normal
- This triggers the GitLab CI/CD pipeline process, except this time I will use tags to force it to run on a specific runner located on an EC2 instance in AWS
- The EC2 runner picks up the job (via outbound polling of GitLab over HTTPS) and runs it as normal, this time without the environment variables
- The EC2 instance will have an appropriate role assigned that allows it to run the relevant commands without credentials
Testing It Out
All sounds good in practice, but let’s see it in action. The following sections will run through some of the key points and how I deploy the relevant settings to my environment.
NOTE: THE FOLLOWING SETUP WILL NOT BE BEST PRACTICE! DO NOT REPLICATE IT!
I’m going to deploy a relatively simple setup just to prove the concept and keep costs minimal, however in a production environment this is not how you would want to architect it.
For example, I’m going to associate a public IP directly with my EC2 instance so I can SSH straight to it. In real-life you’d want to avoid this and put the host in an isolated private subnet with no direct Internet access. Management of that device would then be via a bastion host of sorts or at a minimum over a secured VPN/private link with source IP restrictions. You’d want to be throwing some 2FA in there for good measure as well, not to mention server hardening, patching etc. You get my point.
Building The Environment In Code
To aid with my studies and to make this re-usable for demos in future, I’m going to setup the bulk of the AWS environment using a CloudFormation template which I’ve written from scratch. CloudFormation, if you’re not familiar with it, is a tool for deploying AWS infrastructure using an “Infrastructure as Code” (IaC) approach. I can’t recommend IaC enough (for both on-premise and cloud). Having your environment documented as code and checked into a version control system (such as GitLab) brings with it so many benefits - perhaps another post for future.
Whilst I’ve used CloudFormation in this example, it is AWS specific, so you may want to consider looking at something such as Terraform instead for production purposes (it is a much broader IaC tool). This is particularly true if you’re deploying your applications and services into multiple cloud environments.
Back to the setup… I’m going to deploy all of the settings I need with a single CloudFormation template and a load of hardcoded variables. Again, not representative of what you would want to be doing in real life, but it allows me to re-use this easily in future. My template will create the following test environment under my AWS account:
- A single VPC with an Internet gateway
- A single subnet and a route table associated (the default route pointing to an Internet gateway)
- An IAM role and policy permitting the relevant access to my S3 bucket and CloudFront
- A single EC2 instance (based on a t2.micro image of Amazon Linux 2) with the relevant role attached. I’ll use a basic provisioning script (user data) as part of the EC2 build to install the GitLab runner software
- Security group configuration to allow outbound Internet access outbound on HTTP/HTTPS and DNS (against the VPC DNS service)
- Security group configuration to allow inbound Internet access on SSH
ONCE AGAIN… this isn’t representative of how you’d look to deploy this setup in real-life. Don’t go exposing sensitive hosts like this directly to the Internet, and don’t give them unfettered Internet access.
For those curious people amongst you, the CloudFormation code can be found here.
Deploying The Template
Now I’ll deploy the template. To make it easier for those who aren’t familiar with the AWS CLI I will run through this in the AWS web console. After navigating to CloudFormation > Create Stack. I’ll select “Template is ready” and upload from my desktop with the “Upload a template file”. Note, when you choose this option it actually uploads the file to an automatically created S3 bucket and then references that URL - so keep an eye on that.
Next you’ll be asked to provide a Stack name and any parameters that need to be input. You’ll notice this is blank for me because I’ve hardcoded all the details I need, but this is how you would allow for the use of a single template, take inputs from the person/process deploying it and change the environment accordingly.
On the next page you can assign the usual AWS tags to the Stack, select a different role to run the job as and tweak some advanced options. I’m leaving all these as default for now.
On the last page you are provided an opportunity to review your settings. One thing to note with my template is the capability I have to acknowledge at the bottom. This is a protection mechanism aimed at preventing people downloading random templates and deploying them without knowing it is making changes to their IAM configuration.
It is flagging here that my template will create a new IAM role and that I need to acknowledge this. This is fine in this instance (for me), as it is intended, but if I had downloaded it from the Internet, I may want to be thinking twice about what it is doing and review further.
Once you acknowledge the capability and click create stack, CloudFormation will do its thing! Assuming you have no errors (which you absolutely will when you first play around with CloudFormation templates!) it will go through the tasks creating the relevant resources before confirming once it is finished.
One of the great things about using stacks is that once I’m finished, I can just delete it and it will tidy up the whole environment. Great for demoing consistent environments without leaving them running all the time.
The EC2 Instance And Role
There isn’t anything particularly special to call out about the EC2 instance. As long as the OS can support the GitLab runner software and can call home (https://gitlab.com), then it can be of your choosing and sized according to you needs. I’ve gone for a t2.micro instance running the Amazon Linux 2 AMI.
I’m not going to go into the details of the whole CloudFormation template, but some of the main bits to call out in respect to the EC2 instance and the role creation/attachment are shown below. The code snippet below shows the IAM role, the IAM policy and the profile which is then attached to the EC2 instance. These all work together to define the permissions, restrict the ability to “assume” the role to the EC2 service, and then specifically attach this role permission to the created instance.
TestRunnerRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - "sts:AssumeRole" Description: IAM role to provide GitLab runer relevant permissions RoleName: TestRunnerRole Tags: - Key: Name Value: TestRunnerRole TestRunnerIAMPolicy: DependsOn: TestRunnerRole Type: AWS::IAM::ManagedPolicy Properties: Description: IAM policy for GitLab CI/CD pipeline PolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Action: - s3:PutObject - s3:GetObject - s3:DeleteObject Resource: "arn:aws:s3:::mikeguy.co.uk/*" - Effect: "Allow" Action: "cloudfront:CreateInvalidation" Resource: "arn:aws:cloudfront::<ACCOUNT_ID_OMITTED>:distribution/<DISTRIBUTION_ID_OMITTED>" - Effect: "Allow" Action: s3:ListBucket Resource: "arn:aws:s3:::mikeguy.co.uk" ManagedPolicyName: TestRunnerIAMPolicy Roles: - Ref: TestRunnerRole TestGitLabRunnerProfile: DependsOn: TestRunnerRole Type: AWS::IAM::InstanceProfile Properties: InstanceProfileName: TestGitLabRunnerProfile Roles: - Ref: TestRunnerRole TestGitLabRunner: DependsOn: - TestSecurityGroup - TestSubnet - TestGitLabRunnerProfile Type: "AWS::EC2::Instance" Properties: IamInstanceProfile: !Ref TestGitLabRunnerProfile <REST OMITTED>
Registering The Runner
Now that my environment is built, I’m going to SSH to my EC2 runner and register the host with GitLab. I’m going to set the runner up to just accept shell commands and not worry about more advanced executors such as docker containers.
The command you need to register the instance is “sudo /usr/local/bin/gitlab-runner register” and the registration token can be found in GitLab under the project settings > CI/CD > Runners > Set up a specific runner automatically. Don’t think about trying to re-use my registration key - I’ve changed it before posting!
As you can see from the screenshot you have to provide a few values - the URL, the registration token, a description, tag(s) and the “executor” that the software should run (shell, docker etc.) Once you’ve done this, you’ll notice your GitLab specific runners now have an active entry.
Adjusting The Pipeline Job
Almost there! The only thing left to do is to update our .gitlab-ci.yml file to take into account the new changes and remove the previous environment variables from GitLab settings.
I’m going to leave the build process on the shared runners still, but then adjust my deploy phase to use the new specific runner. I’ll remove the image setting (as I’m not going to use a docker container), remove the pip install command (aws CLI is already installed on the base EC2) and add a “tags” section. The tag will tell GitLab to run it on a runner with that tag - the only one being available of course is my EC2 instance.
The section of the .gitlab-ci.yml file now looks as follows:
deploy: stage: deploy script: - echo $AWS_ACCESS_KEY_ID - echo $AWS_SECRET_ACCESS_KEY - aws s3 sync public_html s3://mikeguy.co.uk --delete - aws cloudfront create-invalidation --distribution-id <OMITTED> --paths /posts/ /posts/* tags: - ec2-runner only: - master
I’ve added the echo commands purely to show the environment variables are not set as they were previously. Now let’s add and commit our changes to GitLab and kick off a pipeline job.
No environment vars…
S3 Sync and CloudFront invalidation successful…
Et Voila! As you can see from above, the build and deploy phases have executed successfully without any AWS environment variables. Success!
As you can see, using a specific runner is a great way to achieve what I set out to do. I could have done something similar using the runner whereby instead of pushing straight to S3 with a role, I retrieved credentials from AWS Secrets Manager (again using a role). These credentials could then be used for whatever build/test/deploy purpose I needed. Handy if I wanted/needed to keep using API keys but wanted to rotate them frequently and prevent the need for putting them into GitLab project variables. The credentials would never actually leave the AWS environment meaning your developers could be restricted from having access to them directly still.
Like a lot of things with AWS, there are multiple ways of achieving a desired outcome.
Hopefully this article was of some use! It was more for my own benefit of playing around and getting more and more familiar with AWS, GitLab and DevOps styles of approaching projects, but may give you some food for thought.
Any questions or comments then as always I’d be delighted to hear from you!
2019-11-06 00:00 +0000