How We Integrated End-to-End API Testing into CI/CD

An API can respond successfully to a request and still leave an engineering team uncertain about the next deployment. The critical question is whether the behaviour that external consumers depend on will continue to work when the application, its configuration or its environment changes.

At Parser, we worked alongside the development team to improve how we validated changes to an API used by external consumers. Our goal was to move beyond manual checks and make API testing a repeatable part of the delivery process. To achieve this, we built an end-to-end API testing framework and integrated its execution into our GitLab and Octopus workflows.

After a merge into the development branch, GitLab handles the deployment process through Octopus Deploy. The E2E API tests run once the application has been deployed to the integration testing environment, with separate jobs for Azure and GCP.

The integration risks we needed to address

Our starting point was an external API receiving client data used by downstream processes. Its validation relied mainly on manual checks and isolated test executions. While useful for investigating individual changes, they could not provide a consistent view of the deployed application after each delivery. As the API became more central to the integration flow, the gap between checking a request and validating a deployment became impossible to ignore.

Working on this project at Parser, we observed how several risks could overlap:

• A payload change could affect consumers
• An endpoint could behave differently depending on a feature flag
• The same application could run with different configurations in Azure and GCP.

A successful response from one endpoint in a single environment could not tell us whether the critical scenarios still worked across both deployment targets.

We needed repeatable checks to validate those specific risks consistently.

Designing the framework around critical scenarios

From a QA perspective, we approached the initiative by identifying high-risk endpoints and core business scenarios. That gave the framework a clear purpose: validate the behaviour on which external integrations relied and make the results available during delivery. Returning an expected HTTP status code was part of that validation, alongside verifying the response structure, payload handling, and expected behaviour in both valid and negative scenarios.

This focus on selected critical scenarios is consistent with Ham Vocke's discussion The Practical Test Pyramid. He describes end-to-end tests as a useful but costly layer that should be used selectively within a bigger testing strategy.

To keep the framework maintainable, we separated test scenarios from the code responsible for executing them.

We used SpecFlow and Gherkin to describe the API scenarios in Given, When and Then steps. These steps are connected to C# bindings, which delegate the actual work to driver classes. The drivers handle operations such as loading request payloads, sending requests, polling asynchronous processes and checking responses.

For HTTP communication, we used Playwright's APIRequestContext through a reusable client. Test data is stored in JSON fixtures, while environment-specific settings and GitLab variables allow the same framework to run against Azure and GCP without duplicating the scenarios.

The diagram below shows how these components work together.

Feature flags also had to be reflected in the expected behaviour: an endpoint deliberately disabled in a particular configuration should not be assessed as though it were enabled. Pete Hodgson's Feature Toggles article explains how flags introduce additional testing considerations and why teams need to make deliberate choices about the configurations they validate. In our case, supporting flag-dependent scenarios was part of making the suite representative of the deployed API.

Bringing API validation into the delivery workflow

Once the framework could express those scenarios clearly, we connected it to the team's delivery workflow.

1. Developers could run the tests locally against the relevant environment before merging, and attach execution evidence to a merge request when a change affected the API. This gave QA and development a shared basis for reviewing the change and made feedback available before the automatic post-deployment run.
2. After merging into the development branch, GitLab runs the delivery pipeline and deploys the application through Octopus Deploy. We configured the Azure and GCP test jobs to depend on the integration environment deployment job, so the tests run against the deployed application rather than an earlier build. Each job uses its own environment variables to select the target API.

The following diagram shows how the E2E API tests fit into the delivery pipeline, from merging a change to collecting the test results for each cloud environment.

The timing was important. We wanted to test the application that had actually been deployed, including its environment configuration, because some integration problems only become visible when those pieces are running together.

Running the suite against both Azure and GCP addressed another limitation of our previous approach. Although the application code was shared, configuration, routing, secrets, certificates or dependent services could differ between the two platforms. Using the same critical scenarios with separate target configuration gave us evidence about behaviour in each cloud. It also highlighted why passing the suite in one platform did not establish that the other platform behaved correctly.

Introducing a non-blocking quality signal

The initiative is still in an early adoption phase. We have therefore kept allow_failure: true enabled for the relevant GitLab test jobs. In GitLab's CI/CD, this setting allows subsequent jobs to continue when the job fails. At this stage, the suite provides a post-deployment quality signal, while a failing test job does not itself fail the pipeline or block its progression.

Below is a shortened version of our Azure job. I've left out the dependency setup, build and authentication steps, but kept the deployment dependency, test execution and reporting configuration. The GCP job follows the same structure.

aatest-e2e-smoke-azure:
  stage: aatest-e2e-smoke

  needs:
    - job: octopus-deploy-aatest

  allow_failure: true

  variables:
    TEST_ENVIRONMENT: AZURE_AATEST
    E2E_BASEURL: $AATEST_AZURE_E2E_BASEURL
    E2E_RUN_TAG: smoke

  script:
    # Restore, build and authentication setup omitted
    - dotnet test --no-build -c Release --logger "trx;LogFileName=${CI_PROJECT_DIR}/artifacts/aatest-e2e-smoke-azure.trx"

  artifacts:
    paths:
      - artifacts

That distinction changes how the result must be read. A successful pipeline status alone does not establish that the API tests passed when failures are allowed. The test job's own result and execution evidence remain necessary to understand what happened.

Moving to a blocking quality gate would be a separate decision. Before doing so, we would want consistent execution and a clear process for investigating failures, including distinguishing application defects from test or environment problems.

What changed and what comes next

The main change is that API testing is now part of the deployment workflow rather than something we have to run separately. GitLab executes the tests against Azure and GCP after deployment, and the results are available alongside the other pipeline jobs.

We also have a framework that we can extend without rebuilding the request handling or environment configuration for each new scenario. The BDD structure makes it easier to see what a test is checking, while the separate cloud jobs let us investigate failures in each deployment target.

We haven't yet measured how much manual testing time this has saved or how often the suite catches regressions. For now, the result we can point to is a repeatable process for running API checks after deployment.

For me, the work illustrates the role QA can play in shaping automation around delivery risk. Selecting meaningful scenarios was as important as implementing requests and assertions. Deciding when the suite should run, how configuration should be handled and what a failure should mean connected the tests to the decisions the team needed to make.

Our experience delivering this integration showed the value of bringing deployed API behaviour into regular delivery feedback. The framework gives us a consistent way to ask whether critical integrations still work after a change, across both cloud platforms. As the initiative matures, that evidence can inform the decision about whether particular checks are ready to become blocking quality gates.

References

• Ham Vocke. The Practical Test Pyramid. 26 February 2018. https://martinfowler.com/articles/practical-test-pyramid.html
• Pete Hodgson. Feature Toggles aka Feature Flags. https://martinfowler.com/articles/feature-toggles.html
• GitLab. CI/CD YAML syntax reference. allow_failure. https://docs.gitlab.com/ci/yaml/#allow_failure