Creating CI tests with the Checks API

Introduction

This guide will introduce you to GitHub Apps and the Checks API, which you'll use to build a continuous integration (CI) server that runs tests.

CI is a software practice that requires frequently committing code to a shared repository. Committing code more often raises errors sooner and reduces the amount of code a developer needs to debug when finding the source of an error. Frequent code updates also make it easier to merge changes from different members of a software development team. This is great for developers, who can spend more time writing code and less time debugging errors or resolving merge conflicts. 🙌

A CI server hosts code that runs CI tests such as code linters (which check style formatting), security checks, code coverage, and other checks against new code commits in a repository. CI servers can even build and deploy code to staging or production servers. For some examples of the types of CI tests you can create with a GitHub App, check out the continuous integration apps available in GitHub Marketplace.

Checks API overview

The Checks API allows you to set up CI tests that are automatically run against each code commit in a repository. The Checks API reports detailed information about each check on GitHub in the pull request's Checks tab. With the Checks API, you can create annotations with additional details for specific lines of code. Annotations are visible in the Checks tab. When you create an annotation for a file that is part of the pull request, the annotations are also shown in the Files changed tab.

A check suite is a group of check runs (individual CI tests). Both the suite and the runs contain statuses that are visible in a pull request on GitHub. You can use statuses to determine when a code commit introduces errors. Using these statuses with protected branches can prevent people from merging pull requests prematurely. See "About protected branches" for more details.

The Checks API sends the check_suite webhook event to all GitHub Apps installed on a repository each time new code is pushed to the repository. To receive all Checks API event actions, the app must have the checks:write permission. GitHub automatically creates check_suite events for new code commits in a repository using the default flow, although Update repository preferences for check suites if you'd like. Here's how the default flow works:

1. Whenever someone pushes code to the repository, GitHub sends the check_suite event with an action of requested to all GitHub Apps installed on the repository that have the checks:write permission. This event lets the apps know that code was pushed and that GitHub has automatically created a new check suite.
2. When your app receives this event, it can add check runs to that suite.
3. Your check runs can include annotations that are displayed on specific lines of code.

In this guide, you’ll learn how to:

• Part 1: Set up the framework for a CI server using the Checks API.
  • Configure a GitHub App as a server that receives Checks API events.
  • Create new check runs for CI tests when a repository receives newly pushed commits.
  • Re-run check runs when a user requests that action on GitHub.
• Part 2: Build on the CI server framework you created by adding a linter CI test.
  • Update a check run with a status, conclusion, and output details.
  • Create annotations on lines of code that GitHub displays in the Checks and Files Changed tab of a pull request.
  • Automatically fix linter recommendations by exposing a "Fix this" button in the Checks tab of the pull request.

To get an idea of what your Checks API CI server will do when you've completed this quickstart, check out the demo below:

Prerequisites

Before you get started, you may want to familiarize yourself with GitHub Apps, Webhooks, and the Checks API, if you're not already. You'll find more APIs in the REST API docs. The Checks API is also available to use in GraphQL, but this quickstart focuses on REST. See the GraphQL Checks Suite and Check Run objects for more details.

You'll use the Ruby programming language, the Smee webhook payload delivery service, the Octokit.rb Ruby library for the GitHub REST API, and the Sinatra web framework to create your Checks API CI server app.

You don't need to be an expert in any of these tools or concepts to complete this project. This guide will walk you through all the required steps. Before you begin creating CI tests with the Checks API, you'll need to do the following:

1. Clone the Creating CI tests with the Checks API repository.

   $ git clone https://github.com/github-developer/creating-ci-tests-with-the-checks-api.git

   Inside the directory, you'll find a template_server.rb file with the template code you'll use in this quickstart and a server.rb file with the completed project code.

2. Follow the steps in the "Setting up your development environment" quickstart to configure and run the app server. Note: Instead of cloning the GitHub App template repository, use the template_server.rb file in the repository you cloned in the previous step in this quickstart.

   If you've completed a GitHub App quickstart before, make sure to register a new GitHub App and start a new Smee channel to use with this quickstart.

   See the troubleshooting section if you are running into problems setting up your template GitHub App.

Part 1. Creating the Checks API interface

In this part, you will add the code necessary to receive check_suite webhook events and create and update check runs. You'll also learn how to create check runs when a check was re-requested on GitHub. At the end of this section, you'll be able to view the check run you created in a GitHub pull request.

Your check run will not be performing any checks on the code in this section. You'll add that functionality in Part 2: Creating the Octo RuboCop CI test.

You should already have a Smee channel configured that is forwarding webhook payloads to your local server. Your server should be running and connected to the GitHub App you registered and installed on a test repository. If you haven't completed the steps in "Setting up your development environment," you'll need to do that before you can continue.

Let's get started! These are the steps you'll complete in Part 1:

1. Updating app permissions
2. Adding event handling
3. Creating a check run
4. Updating a check run

Step 1.1. Updating app permissions

When you first registered your app, you accepted the default permissions, which means your app doesn't have access to most resources. For this example, your app will need permission to read and write checks.

To update your app's permissions:

1. Select your app from the app settings page and click Permissions & Webhooks in the sidebar.
2. In the "Permissions" section, find "Checks", and select Read & write in the Access dropdown next to it.
3. In the "Subscribe to events" section, select Check suite and Check run to subscribe to these events.
4. Click Save changes at the bottom of the page.
5. If you've installed the app on your account, check your email and follow the link to accept the new permissions. Any time you change your app's permissions or webhooks, users who have installed the app (including yourself) will need to accept the new permissions before the changes take effect. You can also accept the new permissions by navigating to your installations page and clicking on "Configure" next to your app. You'll see a banner at the top of the page letting you know that the app is requesting different permissions. Click "Details" and click "Accept new permissions."

Great! Your app has permission to do the tasks you want it to do. Now you can add the code to handle the events.

Step 1.2. Adding event handling

Now that your app is subscribed to the Check suite and Check run events, it will start receiving the check_suite and check_run webhooks. GitHub sends webhook payloads as POST requests. Because you forwarded your Smee webhook payloads to http://localhost/event_handler:3000, your server will receive the POST request payloads at the post '/event_handler' route.

An empty post '/event_handler' route is already included in the template_server.rb file, which you downloaded in the prerequisites section. The empty route looks like this:

  post '/event_handler' do

    # # # # # # # # # # # #
    # ADD YOUR CODE HERE  #
    # # # # # # # # # # # #

    200 # success status
  end

Use this route to handle the check_suite event by adding the following code:

# Get the event type from the HTTP_X_GITHUB_EVENT header
case request.env['HTTP_X_GITHUB_EVENT']
when 'check_suite'
  # A new check_suite has been created. Create a new check run with status queued
  if @payload['action'] == 'requested' || @payload['action'] == 'rerequested'
    create_check_run
  end
end

Every event that GitHub sends includes a request header called HTTP_X_GITHUB_EVENT, which indicates the type of event in the POST request. Right now, you're only interested in events of type check_suite, which are emitted when a new check suite is created. Each event has an additional action field that indicates the type of action that triggered the events. For check_suite, the action field can be requested, rerequested, or completed.

The requested action requests a check run each time code is pushed to the repository, while the rerequested action requests that you re-run a check for code that already exists in the repository. Because both the requested and rerequested actions require creating a check run, you'll call a helper called create_check_run. Let's write that method now.

Step 1.3. Creating a check run

You'll add this new method as a Sinatra helper in case you want other routes to use it too. Under helpers do, add this create_check_run method: