CI/CD Pipeline Tools and AWS CodePipeline Tutorial

·9 min read

CI/CD Pipeline Tools and AWS CodePipeline Tutorial

What is a CI/CD pipeline?

CI/CD stands for continuous integration/continuous deployment and refers to creating a code pipeline where code changes are committed, tested, and then deployed without any manual interference. Basically, a developer can add a new feature to their codebase and push updates in an efficient, streamlined process. 

 

Why is CI/CD important?

Say you create a Django web application on your local device that you deploy on an AWS Elastic Beanstalk.  If you have previously done this yourself or followed this tutorial, you probably know that it's a pretty straightforward process.  Once AWS CLI is setup and your application is finished, you run eb init then eb deploy to launch.  If you need to make any changes, you edit the necessary files and run eb deploy again.  However, if your application is attracting users throughout the day, a new code change might cause a bug and users could get a 404 or 500 error. To alleviate this issue, developers setup a CI/CD pipelines where any code change is tested before automatically being added to the production environment.  This also allows developers at larger startups to submit hundreds of changes a day without having to manually check the same things.  A CI/CD pipeline will allow you to ship code faster and more effectively -- a crucial process for scalability.

 

Who uses CI/CD?

Any software company or startup likely already has a CI/CD pipeline setup in place. In fact, any new software position usually requires a prospective developer to understand how to use a code pipeline.  Just check out any "careers" tab for a given software company and see what pipeline tools they recommend to know.  You might see one of the tools listed below.

 

What are some common CI/CD tools?

  • Buddy
  • Travis CI/CD
  • Jenkins
  • CircleCI
  • Gitlab
  • Codeship
  • AWS Codepipeline

 

Despite all of the different tools, each one is fairly similar.  They all involve pushing code using git to a repository that is connected to a given CI/CD tool.  Then the changes go through tests and deploy without having to do any additional work.  Of course there are some unique features with each one, but we're going to stick with AWS Codepipeline since we're going to deploy on an ElasticBeanstalk and we've covered other AWS topics here. . When selecting a CI/CD tool you might consider factors such as pricing, ease of implementation, and even popularity since other developers will likely be familiar with the software. 

 

CI/CD Tutorial - AWS Codepipeline

Don't worry, deploying your own code pipeline should not be that difficult; it can be broken down into the following steps: Create a pipeline, specify where your project's source code is hosted (most likely Github), add a build stage to run tests in an isolated environment, then select where you're going to deploy your code.  To hopefully make this a little simpler, we'll go through the steps to deploy to ElasticBeanstalk first.  That way you'll know where we're going to deploy from the start.

1. Setup your Django Application and deploy on an AWS ElasticBeanstalk

If you've never deployed a Django web application to ElasticBeanstalk, here's a brief breakdown. For detailed instructions check out Django Quick Start and Deploying a Django Web App to AWS ElasticBeanstalk.

  • From command prompt/terminal, create a virtual environment:  python3 -m venv env (Mac)      py -m venv env (Windows)
  • Activate the virtual environment:  source bin/activate (Mac)    Scripts\activate (Windows)
  • CD into the virtual environment and install Django: pip install Django
  • create a new django project with: django-admin startproject mysite
  • CD into mysite and run the server: python3 manage.py runserver (Mac)      py manage.py runserver(Windows)

You should see the following screen when you visit http://127.0.0.1:8000/.  

django rocketship runserver setup

  • Next, in the mysite project directory, save your dependencies in a requierements.txt file:  pip freeze > requirements.txt
  • Then create an ebextensions folder: mkdir .ebextensions
  • Inside the ebextensions folder create a file named django.config with the following contents:
    option_settings:
      aws:elasticbeanstalk:container:python:
        WSGIPath: mysite/wsgi.py
    ​

     

  • Awesome, there's two steps left, creating a git repository for a our project and deploying to Elasticbeanstalk either through the AWS command line interface in the article linked above or by uploading our zipped project to the Elasticbeanstalk console.   Regardless you'll need to run git init next to initialize an empty repository.  You can use this guide to install git: https://gist.github.com/derhuerst/1b15ff4652a867391f03. As expected, you'll need a Github account to continue.
  • After initializing your repository, visit github.com and create a new remote repository that you will connect to your local repository.  After creating the new repository on Github, connect the two repositories by entering the following line in your command prompt/terminal where you just ran git init:  git remote add origin <remote_url>
  • Replace <remote_url> with the url from your newly created remote repository:
github remote repository url
  • Next add and commit your files then push to the remote repository.  Afterwards you should see your project in the remote repository
    • git add .
    • git commit -m "first commit"
    • git push origin
  • Alright you're ready to deploy your web app.  I've already listed a few options above so you can choose how you want to deploy, either through the AWS CLI or the AWS ElasticBeanstalk console.  Remember if you choose to deploy through the AWS CLI, the last commit you made will be deployed NOT the current files in your project.  Once deployed, continue the steps below.
  • 2. Create a Codepipeline

    create pipeline

     

    Now let's get started with the actual CI/CD integration.  If you're not already, login to your AWS account then visit AWS Codepipeline and click "Create pipeline".  Give your pipeline a name. Select "New service role" and click "Next".  

     

    3. Add source stage

    add source stage

    Currently we should have the source code for our demo project hosted on Github, so let's select Github (Version 2).  Version 2 manages authentication in more secure way while Version 1 uses OAuth tokens and is no longer recommended according to the official docs.  Select "Connect to Github" and create a connection by naming your connection and then installing a new Github app.  You'll be asked to authorize permissions.

    add github authentication

    After creating the connection, you should see "Ready to connect" in a green box if successful.  Select the repository and the branch name you'd like to use.  Then select "CodePipeline default" instead of cloning the entire repository and hit "Next".

    4. Adding a build stage

    adding a build stage

    Now we'll need to setup a build environment.  Basically, we can select AWS CodeBuild or Jenkins to build our project on a separate server to run tests in an isolated environment before deployment.  If following along, select AWS CodeBuild then click the "Create project" box.  A new window will open.  Give your build project a name.  For environment image, we'll select "Managed image",  for operating system, Amazon Linux 2, for runtime, standard, and for image version, select the newest version.  We'll leave the rest of the configuration options as is. 

     

    complete build stage

     

    Once finished, you should see a green box noting the success of your build creation.  Environment variables should be added, however, we typically use python-decouple to setup env variables within our project and add an .env file to the private repository; so we'll skip this step. Select "single build" and hit "Next".

     

    5. Add deploy stage and Review:

    aws-provider

    (Note: the image above will look a little different from the first time you enter this step.  This is an image after setting up the pipeline and then going back and editing this step.)  Since we deployed our application on Elastic Beanstalk in the first step, we'll select this option as our deploy provider.  Then choose the right application and environment that you created earlier. Click "Next" and review all of the options we configured.  This should give you a pretty good idea of the entire process. 

     

    6. Deploy Again With a BuildSpec

    Here's the fun part. First add a buildspec.yml to the mysite project folder:

    version: 0.2
    
    phases:
      install:
        runtime-versions:
          python: 3.7
        commands:
          # Install dependencies needed for running tests
          - echo Installing dependencies
          - pip install -r requirements.txt
    
      build:
        commands:
          - echo Build started on `date`
          - echo Running tests
          - python3 manage.py test
    
      post_build:
        commands:
          - echo Build completed on `date`
    
    artifacts:
      files:
        - '**/*'
    

    Next, change your Django project by replacing the debug page with your own homepage.  Then add, commit, and push the changes.  After pushing your changes to the master branch, your code pipeline will start! You'll see your code going through the different stages; from being retrieved from Github, to being tested in AWS CodeBuild, and ultimately, deployed on your ElasticBeanstalk environment.   If it all goes successfully, then you should see a green checkbox next to each stage.

    aws code pipeline example
    Otherwise, you'll get an error at one of the stages and need to debug using the provided logs. 

     

    Conclusion

    That concludes this CI/CD overview and tutorial.  Let me know in the comments if I should expand on certain points or if you're a little confused on a given topic.  This tutorial was meant for a simple Django project so I didn't discuss how to handle different databases.  For example, if you're using AWS RDS, you might have the following in your settings.py file: 

        DATABASES = {
         'default': {
            'ENGINE': 'django.db.backends.postgresql',
            'NAME': config('DB_NAME'),
            'USER': config('DB_USER'),
            'PASSWORD': config('DB_PASSWORD'),
            'HOST': config('DB_HOST'),
            'PORT': config('DB_PORT'),
            }
        }

     

    This actually creates a problem at the AWS CodeBuild stage.  Our build environment does not have permission to connect to the remote database.  However, there's a few solutions.

    Also I never touched on what's happening in that buildspec.yml file.  We're specifying the commands to run during specific phases of the build stage to setup our dependencies and tests.  If you wanted to test something, run py manage.py createapp main to create an app for your project.  Here's some tests to check if a robots.txt file has been correctly added to your site.  This is helpful for connecting your Django project to Google Search Console

    from django.test import TestCase
    from http import HTTPStatus
    
    # Create your tests here.
    
    class RobotsTxtTests(TestCase):
        def test_get(self):
    
            response = self.client.get("/robots.txt")
            self.assertEqual(response.status_code, HTTPStatus.OK)
            self.assertEqual(response["content-type"], "text/plain")
            lines = response.content.decode().splitlines()
            self.assertEqual(lines[0], "User-Agent: *")
    
        def test_post_disallowed(self):
    
            response = self.client.post("/robots.txt")
            self.assertEqual(response.status_code, HTTPStatus.METHOD_NOT_ALLOWED)

     

     

    Hope you enjoyed!