Forcing a failure in GitHub Actions based on a condition

Last night I got tweeted at asking me how one could halt a CI workflow in GitHub Actions on a condition.  This particular condition was if the code coverage tests failed a certain coverage threshold.  I’m not a regular user of code coverage tools like Coverlet but I went Googling for some answers and oddly did not find the obvious answer that was pointed out to me this morning.  Regardless the journey to discover an alternate means was interesting to me so I’ll share what I did that I feel is super hacky, but works and is a similar method I used for passing some version information in other workflows.

First, the simple solution for if you are using Coverlet and want to fail a build and thus a CI workflow is to use the MSBuild integration option and then you can simply use:

dotnet test /p:CollectCoverage=true /p:Threshold=80

I honestly felt embarrassed that I didn’t find this simple option, but oh well, it is there and is definitely the simplest option if you can use this option.  But there you have it.  When used in an Actions workflow if the threshold isn’t met, this will fail that step and you are done.

Creating your condition to inspect

But let’s say you need to fail for a different reason or in this example here, you couldn’t use the MSBuild integration and instead are just using the VSTest integration with a collector.  Well, we’ll use this code coverage scenario as an example but the key step here is focusing on how to fail a step.  Your condition may be anything but I suspect it is usually based on some previous step’s output or value.  Well first, if you are relying on previous steps values, be sure you understand the power of using outputs.  This is I think the best way to kind of ‘set state’ of certain things in steps.  A step can do some things and either in the Action itself set an Output value, or in the workflow YAML you can do this as well using a shell command and calling the ::set-output method.  Let’s look at an example…first the initial step (again using our code coverage scenario):

- name: Test
  run: dotnet test XUnit.Coverlet.Collector/XUnit.Coverlet.Collector.csproj --collect:"XPlat Code Coverage"

This basically will produce an XML output ‘report’ that contains the values we want to extract.  Namely it’s in this snippet:

<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="0.85999999999" branch-rate="1" version="1.9" timestamp="1619804172" lines-covered="15" lines-valid="15" branches-covered="8" branches-valid="8">
  <sources>
    <source>D:\</source>
  </sources>
  <packages>
    <package name="Numbers" line-rate="1" branch-rate="1" complexity="8">
      <classes>

I want the line-rate value (line 2) in this XML to be my condition…so I’m going to create a new Actions step to extract the value by parsing the XML using a PowerShell cmdlet.  Once I have that I will set the value as the output of this step for later use:

- name: Get Line Rate from output
  id: get_line_rate
  shell: pwsh  
  run: |
    $covreport = get-childitem -Filter coverage.cobertura.xml -Recurse | Sort-Object -Descending -Property LastWriteTime -Top 1
    Write-Output $covreport.FullName
    [xml]$covxml = Get-Content -Path $covreport.FullName
    $lineRate = $covxml.coverage.'line-rate'
    Write-Output "::set-output name=lineRate::$lineRate"

As you can see in lines 2 and 9 I have set a specific ID for my step and then used the set-output method to write a value to an output of the step named ‘lineRate’ that can be later used.  So now let’s use it!

Evaluating your condition and failing the step manually

Now that we have our condition, we want to fail the run if the condition evaluates a certain way…in our case if the code coverage line rate isn’t meeting our threshold.  To do this we’re going to use a specific GitHub Action called actions/github-script which allows you to run some of the GitHub API directly in a script.  This is great as it allows us to use the core library which has a set of methods for success and failure!  Let’s take a look at how we combine the condition with the setting:

- name: Check coverage tolerance
  if: ${{ steps.get_line_rate.outputs.lineRate < 0.9 }}
  uses: actions/github-script@v3
  with:
    script: |
        core.setFailed('Coverage test below tolerance')

Okay, so we did a few things here.  First we are defining this step as executing the core.setFailed() method…that’s specifically what this step will do, that’s it…it will fail the run with a message we put in there.  *BUT* we have put a condition on the step itself using the if condition checking.  In line 6 we are executing the setFailed function with our custom message that will show in the runner log.  On line 2 we have set the condition for if this step even runs at all.  Notice we are using the ID of a previous step (get_line_rate) and the output parameter (lineRate) and then doing a quick math check.  If this condition is met, then this step will run.  If the condition is NOT met, this step will not run, but also doesn’t fail and the run can continue.  Observe that if the condition is met, our step will fail and the run fails:

If the condition is NOT met the step is ignored, the run continues:

Boom, that’s it! 

Summary

This was just one scenario but the key here is if you need to manually control a fail condition or otherwise evaluate conditions, using the actions/github-script Action is a simple way to do a quick insertion to control your run based on a condition.  It’s quick and effective for some scenarios where your steps may not have natural success/fail exit codes that would otherwise fail your CI run.  What do you think? Is there a better/easier way that I missed when you don’t have clear exit codes?

Hope this helps someone!