Low-Cost Public Jenkins Server using Terraform and DigitalOcean

Love it or hate it, Jenkins isn't going anywhere. Jenkins is the leading open source automation server and supports building, deploying and automating projects.

My goal is to automate the infrastructure and setup of Jenkins so my time is spent on using Jenkins rather than setting it up. I also want this tutorial to be as affordable as possible so everyone can get started with little financial investment.

By using Terraform, I can build and destroy the entire stack with a single command.

What you'll end up with

• DigitalOcean Droplet
  • nginx reverse proxy
  • Jenkins server
• No Jenkins setup wizard
• Automated configuration (via Jenkins Configuration as Code)
  • plugins (via script)
  • pipelines
• A custom domain with HTTPS (using Let's Encrypt)

Tools and infrastructure used

Terraform

Terraform is a utility developed by HashiCorp that enables you to treat Infrastructure as Code (IaC). Feed it config files and it will manage any cloud, infrastructure, or service.

Digital Ocean

Digital Ocean is a cloud provider with flat-rate billing. This tutorial can be adapted to use any cloud provider but I enjoy the simple billing DigitalOcean offers.

Before you begin

Before running Terraform, you need to setup a few things. The good news is, once complete, you'll never have to do it again.

Buy a domain

For HTTPS certificates, you need a fully qualified domain name (FQDN) that is publicly available.

Use your favorite domain registrar to purchase a domain. For this tutorial I'm using Namecheap.

Use DigitalOcean's nameservers

By using DigitalOcean for DNS there is no need to configure multiple providers in Terraform. Update the nameservers in Namecheap to use DigitalOcean for DNS.

Under Manage domains in Namecheap:

If you're following along, use these exact nameservers:

ns1.digitalocean.com
ns2.digitalocean.com
ns3.digitalocean.com

DNS propagation takes up to 24 hours. However, in my testing it usually is faster. To validate the nameserver records are public, check the domain at https://whois.net/

Export the domain name

export DOMAIN="myjenkinsserver.com"

Create DigitalOcean API access token

Navigate to https://cloud.digitalocean.com/account/api/tokens and click Generate New Token.

Give the token write access.

Export DigitalOcean API access token

Once created, copy the token for safe keeping and export it as a variable.

# DigitalOcean personal access token
export DO_PAT=b4fec39662e1543fc9ac76b6ca9bba9ba6b9ab9bc7b9ab0a

Install Terraform

Check for later versions: https://releases.hashicorp.com/terraform/

# set version
export terraformVersion="0.12.24"

# curl the binary
curl -Lo terraform.zip https://releases.hashicorp.com/terraform/${terraformVersion}/terraform_${terraformVersion}_linux_amd64.zip

# unpack and make executable
unzip terraform.zip
rm -rf terraform.zip
chmod +x ./terraform
sudo mv ./terraform /usr/local/bin/terraform

# validate
terraform version

Export your SSH fingerprint

Your SSH private and public keys are used to authenticate into the server for provisioning. Terraform needs your SSH fingerprint.

# the cut command will remove "MD5:" from the output
export SSH_FINGERPRINT="$(ssh-keygen -E md5 -lf ~/.ssh/id_rsa.pub | awk '{print $2}' | cut -c 5-)"

Create the infrastructure

Validate the following variables are set (check with echo $VARIABLE_NAME):

• DO_PAT
• SSH_FINGERPRINT
• DOMAIN

1. Export Terraform variables

   By adding TF_VAR_ to variables referenced in the Terraform configuration, there is no need to supply it on the command line.

   export TF_VAR_do_token=${DO_PAT}
   export TF_VAR_pub_key=$HOME/.ssh/id_rsa.pub
   export TF_VAR_pvt_key=$HOME/.ssh/id_rsa
   export TF_VAR_ssh_fingerprint=${SSH_FINGERPRINT}
   export TF_VAR_domain=${DOMAIN}
   

2. Clone Terraform files from GitHub

   git clone https://github.com/jimangel/terraform-jenkins.git
   cd terraform-jenkins
   

3. Terraform init downloads the needed Terraform providers

   terraform init
   

4. Dry-run with plan

   terraform plan
   

5. Deploy with apply

   This part takes ~10 minutes don't be alarmed if it hangs occasionally

   terraform apply
   

Generate SSL certs and nginx config

Automating SSL with Terraform is a “chicken and egg” problem.

• Terraform can't automatically generate certs on the server without DNS existing
• Terraform can't update DNS before creating the server

There is a way to generate certificates with Terraform using a DNS TXT record challenge, but then I can't use the awesome certbot command to automatically renew, configure and update my nginx config. One day I might revisit this.

Until then, you need to run a few commands to generate certs:

# ssh into the newly created server
ssh root@$(terraform output ip)

# set domain variable
export DOMAIN="myjenkinsserver.com"

# set email
export EMAIL="email@gmail.com"

# generate certs and automatically configure nginx
sudo certbot --nginx --manual-public-ip-logging-ok --no-eff-email --agree-tos --rsa-key-size 4096 --email ${EMAIL} --redirect -d "${DOMAIN}" -d "www.${DOMAIN}"

While you're logged into the server, grab the admin password:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

5a298e2v1r23erbt0wreb

That's it! Let's check it out

The credentials for initial login are:

• User: admin
• Password: /var/lib/jenkins/secrets/initialAdminPassword

Clean up with destroy

terraform destroy

terraform apply deep dive

When terraform apply is ran, it checks the required variables and providers from provider.tf. Next it uses www-jenkins.tf & domain.tf to build the server and update DNS respectively.

├── domain.tf
├── install-jenkins.sh
├── install-plugins.sh
├── jenkins.yaml
├── nginx.tpl
├── outputs.tf
├── provider.tf
├── versions.tf
└── www-jenkins.tf

www-jenkins.tf specifies the DigitalOcean droplet parameters and imports our SSH keys.

www-jenkins.tf also copies the install-jenkins.sh, install-plugins.sh, and jenkins.yaml files to the newly created server.

install-jenkins.sh is a bash script that does all of the configuration on the server.

• updates packages
• installs Docker (for pipelines)
• installs Java
• installs Jenkins
• installs nginx
  • disables TLS v1 & v1.1
  • enables TLS v1.2 & v1.3
• installs certbot
• enables swap
• installs plugins via install-plugins.sh
• force skips the Jenkins setup wizard
• moves jenkins.yaml JCasC to be used
• restarts Jenkins

install-plugins.sh was copied from a gist I found. You can automate the configuration of a Jenkins server with the Jenkins Configuration as Code plugin; however JCasC is not present by default.

jenkins.yaml is the heart of our configuration used by Jenkins Configuration as Code. If you want to configure Jenkins or add pipelines, do it here.

nginx.tpl generates a nginx.conf file based on your domain variable. This is needed to support custom domains.

Terraform uses outputs.tf to capture information that is referenced later such as the server public IP (terraform output ip).

versions.tf validates your version of Terraform before continuing (>= 0.12)

Costs

My total out-of-pocket cost is $9.06 for the domain and $5/mo for my sever.

DigitalOcean offers incremental billing at $0.007/hr which is good to know if you want to destroy everything afterwards.

I can also spin up the server, test, and spin it down when done.

Other thoughts

Configuration Management

Using install-jenkins.sh to configure our Jenkins server is kind of a hack. The problem is, Terraform can only validate that our infrastructure exists but not if our script completed successfully.

If install-jenkins.sh is modified, and terraform apply is ran, Terraform says there is nothing to change; since all of the infrastructure technically exists. Our only option is to run install-jenkins.sh by hand in the server or terraform destroy and terraform apply.

A better option would be to use Terraform to build our infrastructure and then use Ansible for provisioning Jenkins / nginx. Geerling's Ansible role would be perfect for something like this.

Docker

I mentioned earlier that install-plugins.sh is needed since the JCasC plugin is not present by default.

If I deployed Jenkins using docker instead of the Ubuntu package, I could have built it with the needed plugins.

I purposely chose to not use Docker given the small server size and, to be honest, I didn't want to support building custom docker images for this.

Setup wizard

To skip the initial Jenkins setup wizard I run a sed 's/NEW/RUNNING/g' command to trick Jenkins into thinking it's already provisioned.

I'm not sure what impact this will have on upgrades / installs.

HTTPS

I really don't like having to manually run the HTTPS portion. One option might be to use Terraform and a TXT DNS challenge to get the initial certificates and then use an Ansible playbook to implement Let's Encrypt.

Existing offerings

If you're not trying to nerd out on managing infrastructure, there is a managed CloudBees Jenkins Distribution on DigitalOcean. Although I believe the droplet size needs to be larger which leads to a larger cost.

Shout out

Shout out to AutomatingGuy for such a great tutorial on using JCasC. Most of my Jenkins config is borrowed from his post.