Testing Infrastructure

Relevant source files

• .gitignore
• CONTRIBUTING.md
• Makefile
• api/Conduit.postman_collection.json
• api/README.md
• api/openapi.yml
• api/run-api-tests.sh
• apps/api/server/routes/api/articles/index.get.ts

Purpose and Scope

This document describes the comprehensive testing infrastructure used to validate RealWorld API implementations against the OpenAPI 3.0.1 specification. The testing system uses Postman collections executed via Newman CLI to perform automated API validation, ensuring that any backend implementation adheres to the contract defined in openapi.yml().

The testing infrastructure serves two primary purposes:

1. Validation - Verify that the reference implementation in apps/api/ correctly implements all endpoints
2. Compliance Testing - Provide a standardized test suite that any RealWorld backend implementation can use to validate conformance

For information about the OpenAPI specification that these tests validate, see API Specification. For details on running the reference implementation itself, see Reference Implementation. For CI/CD pipeline configuration, see CI/CD Pipelines.

Sources: api/Conduit.postman_collection.json1-10 api/openapi.yml1-20 api/README.md1-12

---

Test Architecture Overview

The testing system consists of three interconnected components that work together to validate API implementations:

Test Execution Flow Diagram

The testing architecture implements a three-layer validation approach. The OpenAPI specification openapi.yml1-919 serves as the source of truth, defining all 15 API endpoints, request/response schemas, and authentication requirements. The Postman collection Conduit.postman_collection.json1-2411 translates these specifications into executable test cases with embedded JavaScript assertions. The execution layer, coordinated by run-api-tests.sh1-20 uses Newman to run the collection and validate responses.

Sources: api/Conduit.postman_collection.json1-10 api/openapi.yml1-20 api/run-api-tests.sh1-20 Makefile66-77

---

Test Execution Methods

Environment Variables

The test suite is configurable through environment variables that control the test target and user credentials:

Variable	Default	Description
APIURL	https://api.realworld.io/api	Base URL of the API to test
USERNAME	u{timestamp}	Username for test user registration
EMAIL	{USERNAME}@mail.com	Email for test user
PASSWORD	password	Password for test user
DELAY_REQUEST	500	Milliseconds to wait between requests

These variables are processed by run-api-tests.sh6-11 and passed to Newman as global variables via the --global-var flag run-api-tests.sh15-18

Sources: api/run-api-tests.sh6-11

Running Tests Locally

Option 1: Test Against Already-Running Server

This method is defined in Makefile63-64 and directly invokes the test script with APIURL=http://localhost:3000/api.

Option 2: Automated Server Lifecycle

This target Makefile66-77 performs the complete test lifecycle:

1. Starts the API server in the background with JWT_SECRET set
2. Captures the server process ID (SERVER_PID)
3. Waits 0.3 seconds for server initialization
4. Verifies the server process is running
5. Executes the test suite
6. Cleans up the server process regardless of test outcome
7. Reports test results with colored output

Sources: Makefile63-77 api/run-api-tests.sh1-20

Newman CLI Invocation

The actual test execution uses Newman via bun x newman:

This command run-api-tests.sh13-19:

• Uses bun x to execute Newman without requiring global installation
• Runs the Postman collection JSON file
• Sets request delay to avoid overwhelming the API
• Injects environment variables as Postman global variables
• Accepts additional arguments via "$@"

Sources: api/run-api-tests.sh13-19

---

Postman Collection Architecture

Collection Structure and Organization

Postman Collection Folder Organization

The collection Conduit.postman_collection.json8-2411 is organized into seven top-level folders that group related functionality:

1. Auth lines 10-343 - User registration, login, authentication, and profile updates
2. Articles lines 345-654 - Read-only article queries (by author, tag, favorites)
3. Articles, Favorite, Comments lines 656-1865 - Full CRUD operations with stateful workflows
4. Profiles lines 1867-2043 - Profile viewing and follow/unfollow operations
5. Comments lines 2045-2202 - Comment creation, retrieval, and deletion
6. Favorites lines 2204-2342 - Article favoriting and unfavoriting
7. Tags lines 2344-2409 - Tag listing

Sources: api/Conduit.postman_collection.json8-2411

Stateful Test Execution

The collection uses Postman's global variables to maintain state across requests, enabling sequential test workflows:

State Management in Test Flow

Key state variables stored across requests:

Variable	Set By	Used By	Purpose
token	Login and Remember Token lines 141	All authenticated endpoints	JWT authentication
slug	Create Article line 676	Article CRUD operations	Target specific article
slug2	Create Second Article line 744	Pagination test cleanup	Second article for testing
commentId	Create Comment lines 2103-2104	Delete Comment	Target specific comment

Example token storage script Conduit.postman_collection.json140-141:

Example token usage in Authorization header Conduit.postman_collection.json216:

Sources: api/Conduit.postman_collection.json120-148 api/Conduit.postman_collection.json660-693 api/Conduit.postman_collection.json728-747

Test Assertion Patterns

Each request contains embedded JavaScript test scripts that validate response structure and content. The collection uses two assertion styles:

Legacy Style (using tests object):

Modern Style (using pm.test):

Example comprehensive property validation Conduit.postman_collection.json23-31:

Example ISO 8601 timestamp validation Conduit.postman_collection.json373:

Sources: api/Conduit.postman_collection.json19-34 api/Conduit.postman_collection.json326-340 api/Conduit.postman_collection.json356-389

---

Test Coverage and Validation

Endpoint Coverage

The test suite provides comprehensive coverage of all 15 API endpoints defined in the OpenAPI specification:

Category	Endpoint	Method	Test Requests	Authentication
User/Auth	/users	POST	1	No
	/users/login	POST	2	No
	/user	GET	2	Yes
	/user	PUT	2	Yes
Profiles	/profiles/{username}	GET	2	Optional
	/profiles/{username}/follow	POST	1	Yes
	/profiles/{username}/follow	DELETE	1	Yes
Articles	/articles	GET	7	Optional
	/articles	POST	2	Yes
	/articles/feed	GET	1	Yes
	/articles/{slug}	GET	2	Optional
	/articles/{slug}	PUT	1	Yes
	/articles/{slug}	DELETE	2	Yes
Comments	/articles/{slug}/comments	GET	1	Optional
	/articles/{slug}/comments	POST	1	Yes
	/articles/{slug}/comments/{id}	DELETE	1	Yes
Favorites	/articles/{slug}/favorite	POST	2	Yes
	/articles/{slug}/favorite	DELETE	2	Yes
Tags	/tags	GET	1	No

Total: 37 test requests across 15 unique endpoints

Sources: api/Conduit.postman_collection.json8-2411 api/openapi.yml21-453

Validation Categories

The test suite validates multiple aspects of API behavior:

1. Response Structure Validation

Every endpoint test validates that responses contain the expected properties and nested objects:

2. Data Type Validation

Tests verify that properties have the correct data types:

3. Pagination Behavior

Specific tests validate that articlesCount represents total count, not page size:

The collection creates a second article lines 728-779 specifically to test pagination, then deletes it lines 877-911 to clean up.

4. State Persistence

Tests verify that updates persist across requests:

The "Verify Update User Persisted" request lines 292-342 fetches user data after an update to confirm changes were saved.

5. Authentication Flows

Multiple tests validate authentication mechanisms:

• Token generation on login lines 140-144
• Token usage in Authorization header line 216
• Unauthorized access handling (401 responses)
• Protected endpoint enforcement

6. Query Parameter Filtering

Article list tests validate query parameters:

• Filter by author: GET /articles?author=johnjacob line 481
• Filter by tag: GET /articles?tag=dragons line 637
• Filter by favorited user: GET /articles?favorited={{USERNAME}} line 559
• Pagination with limit: GET /articles?limit=1 line 813
• Pagination with offset: GET /articles?limit=1&offset=1 line 856

Sources: api/Conduit.postman_collection.json19-34 api/Conduit.postman_collection.json356-389 api/Conduit.postman_collection.json782-874 api/Conduit.postman_collection.json292-342

---

Integration with Build System

Makefile Test Targets

The testing infrastructure is integrated into the repository's build system through several Makefile targets:

Test Execution Flow via Makefile

Primary Test Target

reference-implementation-test-with-postman Makefile66-77

This target orchestrates the complete test lifecycle:

1. Starts the API server with a hardcoded JWT secret for testing
2. Captures the server process ID in SERVER_PID
3. Sets up a trap to ensure cleanup on exit
4. Waits 0.3 seconds for server initialization
5. Verifies the server process is still running
6. Executes the test suite
7. Cleans up server processes
8. Reports results with colored output (green for success, red for failure)

Alternative Test Target

reference-implementation-test-with-postman-and-already-launched-server Makefile63-64

For faster iteration during development, this target assumes the server is already running and directly invokes the test script with:

• DELAY_REQUEST=50 - Reduced delay for faster execution
• APIURL=http://localhost:3000/api - Local server URL

Process Cleanup Target

running-processes-clean Makefile82-87

This target ensures all Nitro dev server processes are terminated:

The || true ensures the Makefile doesn't fail if no processes are found.

Sources: Makefile63-87

CI/CD Integration

The testing infrastructure is integrated into GitHub Actions workflows, enabling automated validation on every commit and pull request. The workflow configuration is located in .github/workflows/api.yaml and executes the same Makefile targets used for local development, ensuring consistency between local and CI environments.

For detailed information about the CI/CD pipeline configuration, triggers, and workflow steps, see CI/CD Pipelines.

Sources: Makefile66-77

---

Advanced Test Scenarios

Pagination Test Workflow

The collection includes a sophisticated test sequence to validate pagination behavior, ensuring that articlesCount always represents the total count of matching articles, not the number returned in the current page:

Pagination Validation Sequence

The test sequence Conduit.postman_collection.json660-911:

1. Create Article lines 660-726 - Creates first test article, stores slug
2. Create Second Article lines 728-779 - Creates second article for pagination testing, stores slug2
3. All Articles with limit lines 781-827 - Validates:
   • articles.length === 1 (only 1 article returned)
   • articlesCount === 2 (total count, not page size)
4. All Articles with offset lines 829-874 - Validates:
   • articles.length === 1 (still 1 article)
   • articlesCount === 2 (total unchanged by offset)
5. Delete Second Article lines 877-911 - Cleanup test data

The pagination tests explicitly verify that backends implement the specification correctly, where articlesCount must always represent the total number of matching articles, not just those returned in the current page lines 800-804 lines 845-846

Sources: api/Conduit.postman_collection.json660-911 api/openapi.yml696-741

Conditional Test Execution

Some tests include conditional logic to handle different testing contexts. The Register request lines 14-36 includes a guard for integration tests:

This allows the same collection to be used in both full integration test suites and quick validation scenarios where certain checks should be skipped.

Sources: api/Conduit.postman_collection.json20-32

---

Test Failure Debugging

When tests fail, Newman provides detailed output including:

1. Request details - HTTP method, URL, headers, body
2. Response details - Status code, headers, response body
3. Failed assertions - Which specific test assertions failed
4. Execution timing - Request duration and delay information

The test script output can be controlled via the DELAY_REQUEST environment variable to throttle request rate, which is useful when testing against rate-limited APIs or during debugging to make the test execution easier to follow.

The Makefile's test target Makefile66-77 also verifies that the server process started successfully before running tests (using kill -0 "$$SERVER_PID"), providing an early exit with error code 4 if the server failed to start, rather than letting all tests fail with connection errors.

Sources: Makefile66-77 api/run-api-tests.sh11