Integration Testing Go APIs With TDD: Using Testcontainers To Validate Database Interactions

TDD REST API in Go Using Testcontainers

What Are Testcontainers?

Testcontainers is a library that provides easy and lightweight APIs for bootstrapping local development and test dependencies with real services wrapped in Docker containers. Using Testcontainers, you can write tests that depend on the same services you use in production without relying on mocks or in-memory services.

Why Use Testcontainers Over Regular Docker and Docker Compose?

Testcontainers offers a more automated and consistent approach to managing Docker containers for testing compared to regular development containers. They ensure that each test runs in a clean, isolated environment by automatically starting and stopping containers as needed, eliminating the risk of environment drift or contamination between test runs.

Why Use Testcontainers With Go?

Testcontainers can be a valuable tool when building Go applications, especially for integration testing and scenarios that involve external dependencies. Here’s a concise explanation of why you might want to use Testcontainers:

1. Isolated testing environment: Testcontainers provide disposable, lightweight instances of databases, message queues, or other services.
2. Consistency: Ensures tests run against the same environment across different machines.
3. Ease of setup: Simplifies the process of setting up complex dependencies for tests.
4. Realistic testing: Allows testing against actual services rather than mocks.
5. Support for multiple databases: Easily switch between different database types for testing.
6. Version control: Test against specific versions of services.
7. Parallel testing: Run multiple tests simultaneously with isolated containers.
8. CI/CD integration: Works well in continuous integration pipelines.

Building and Testing REST API Using Testcontainers

In this project, we will walk through the process of creating a simple REST API in Go that interacts with a PostgreSQL database. We’ll also cover how to write tests for this API using Testcontainers, a powerful library that allows you to run your integration tests in a consistent and isolated environment. Let’s break down each part of the project.

Creating a Simple API Using Go

We’ll start by setting up the Go application that connects to a PostgreSQL database and serves user data through an HTTP API.

main.go:

package main

import (
 "context"
 "encoding/json"
 "log"
 "net/http"
 "os"

 "github.com/jackc/pgx/v4"
)

var db *pgx.Conn

func main() {
 var err error
 db, err = pgx.Connect(context.Background(), os.Getenv("DATABASE_URL"))
 if err != nil {
  log.Fatalf("Unable to connect to database: %v\n", err)
 }
 defer db.Close(context.Background())

 http.HandleFunc("/users", GetUsersHandler)
 log.Println("Starting server on :8080")
 log.Fatal(http.ListenAndServe(":8080", nil))
}

func GetUsersHandler(w http.ResponseWriter, r *http.Request) {
 rows, err := db.Query(context.Background(), "SELECT id, name FROM users")
 if err != nil {
  http.Error(w, `{"message": "Failed to query users", "status_code": 500}`, http.StatusInternalServerError)
  return
 }
 defer rows.Close()

 type User struct {
  ID   int    `json:"id"`
  Name string `json:"name"`
 }

 var users []User
 for rows.Next() {
  var user User
  err := rows.Scan(&user.ID, &user.Name)
  if err != nil {
   http.Error(w, `{"message": "Failed to scan user", "status_code": 500}`, http.StatusInternalServerError)
   return
  }
  users = append(users, user)
 }

 response := map[string]interface{}{
  "message":     "success",
  "status_code": 200,
  "data":        users,
 }

 w.Header().Set("Content-Type", "application/json")
 json.NewEncoder(w).Encode(response)
}

Let’s quickly go through what we did above. We established a connection to a PostgreSQL database using the pgx library, leveraging the DATABASE_URL environment variable to configure the connection. This connection is reused throughout the application. Then, we set up an HTTP server that listens on port 8080 and handles requests to the /users endpoint. The GetUsersHandler function is responsible for querying the database and returning user data in JSON format.

Testing with Testcontainers
Now that we have our API, the next step is to ensure that it works correctly by writing tests. We’ll use Testcontainers to create a PostgreSQL container for testing, ensuring a consistent and isolated environment.

main_test.go:

package main

import (
 "context"
 "encoding/json"
 "fmt"
 "io/ioutil"
 "net/http"
 "net/http/httptest"
 "testing"

 "github.com/jackc/pgx/v4"
 "github.com/stretchr/testify/assert"
 "github.com/testcontainers/testcontainers-go"
 "github.com/testcontainers/testcontainers-go/wait"
)

func TestGetUsersHandler(t *testing.T) {
 testCases := []struct {
  name          string
  setupDatabase func(conn *pgx.Conn, ctx context.Context) error
  expectedData  []map[string]interface{}
 }{
  {
   name: "Handle multiple users",
   setupDatabase: func(conn *pgx.Conn, ctx context.Context) error {
    _, err := conn.Exec(ctx, "CREATE TABLE users (id SERIAL PRIMARY KEY, name VARCHAR(50));")
    if err != nil {
     return err
    }
    _, err = conn.Exec(ctx, "INSERT INTO users (name) VALUES ('Alice'), ('Bob');")
    return err
   },
   expectedData: []map[string]interface{}{
    {"id": float64(1), "name": "Alice"},
    {"id": float64(2), "name": "Bob"},
   },
  },
 }

 for _, testCase := range testCases {
  t.Run(testCase.name, func(t *testing.T) {
   ctx := context.Background()

   // Start a PostgreSQL container
   req := testcontainers.ContainerRequest{
    Image:        "postgres:13",
    ExposedPorts: []string{"5432/tcp"},
    Env: map[string]string{
     "POSTGRES_PASSWORD": "password",
     "POSTGRES_DB":       "testdb",
     "POSTGRES_USER":     "user",
    },
    WaitingFor: wait.ForAll(
     wait.ForListeningPort("5432/tcp"),
     wait.ForLog("database system is ready to accept connections"),
    ),
   }
   pgContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
    ContainerRequest: req,
    Started:          true,
   })
   assert.NoError(t, err)
   defer func() {
    err := pgContainer.Terminate(ctx)
    assert.NoError(t, err)
   }()

   mappedPort, err := pgContainer.MappedPort(ctx, "5432")
   assert.NoError(t, err)

   host, err := pgContainer.Host(ctx)
   assert.NoError(t, err)

   dbUrl := fmt.Sprintf("postgres://user:password@%s:%s/testdb", host, mappedPort.Port())

   // Connect to the PostgreSQL container
   var conn *pgx.Conn
   for i := 0; i < 5; i++ {
    conn, err = pgx.Connect(ctx, dbUrl)
    if err == nil {
     break
    }
   }
   assert.NoError(t, err)
   defer func() {
    if conn != nil {
     err := conn.Close(ctx)
     assert.NoError(t, err)
    }
   }()

   // Setup database and insert test data
   err = testCase.setupDatabase(conn, ctx)
   assert.NoError(t, err)

   // Assign the global db variable for handler to use
   db = conn

   // Setup the HTTP request and response recorder
   reqHttp, err := http.NewRequest("GET", "/users", nil)
   assert.NoError(t, err)

   rr := httptest.NewRecorder()
   handler := http.HandlerFunc(GetUsersHandler)
   handler.ServeHTTP(rr, reqHttp)

   // Check the status code is what we expect
   assert.Equal(t, http.StatusOK, rr.Code)

   // Check the response body is what we expect
   var response map[string]interface{}
   body, err := ioutil.ReadAll(rr.Body)
   assert.NoError(t, err)

   err = json.Unmarshal(body, &response)
   assert.NoError(t, err)

   // Convert the "data" field from []interface{} to []map[string]interface{} for comparison
   actualData, ok := response["data"].([]interface{})
   assert.True(t, ok)

   var actualDataConverted []map[string]interface{}
   for _, item := range actualData {
    actualDataConverted = append(actualDataConverted, item.(map[string]interface{}))
   }

   // Assert the JSON response
   assert.Equal(t, "success", response["message"])
   assert.Equal(t, float64(200), response["status_code"])
   assert.Equal(t, testCase.expectedData, actualDataConverted)
  })
 }
}

Let’s break down above code and see what we have done, first we a test function TestGetUsersHandler that uses table-driven tests to organize multiple test scenarios. Each test case is represented by a struct containing:

• name: Name for the test case.
• setupDatabase: A function that sets up the test database by creating tables and inserting test data.
• expectedData: The expected result of the database query, which will be compared against the actual result returned by the API handler.

func TestGetUsersHandler(t *testing.T) {
 testCases := []struct {
  name          string
  setupDatabase func(conn *pgx.Conn, ctx context.Context) error
  expectedData  []map[string]interface{}
 }{
  {
   name: "Handle multiple users",
   setupDatabase: func(conn *pgx.Conn, ctx context.Context) error {
    _, err := conn.Exec(ctx, "CREATE TABLE users (id SERIAL PRIMARY KEY, name VARCHAR(50));")
    if err != nil {
     return err
    }
    _, err = conn.Exec(ctx, "INSERT INTO users (name) VALUES ('Alice'), ('Bob');")
    return err
   },
   expectedData: []map[string]interface{}{
    {"id": float64(1), "name": "Alice"},
    {"id": float64(2), "name": "Bob"},
   },
  },
 }

Then, we iterate over each test case defined in the testCases slice. For each test case, we use t.Run to execute the test in an isolated subtest, ensuring each scenario is tested independently. A new background context is created for each test run.

Then, we iterate over each test case defined in the testCases slice. For each test case, we use t.Run to execute the test in an isolated subtest, ensuring each scenario is tested independently. A new background context is created for each test run.

Now, we create a Testcontainer, which runs a PostgreSQL container. The ContainerRequest struct specifies the Docker image, ports, and environment variables needed to configure the PostgreSQL instance. The wait.ForAll function ensures that the container is fully ready before running the test by waiting for the PostgreSQL server to start listening on port 5432 and confirming that it’s ready to accept connections. Then, we check for errors during the container setup, and defer ensures that the container is properly terminated after the test completes.

   req := testcontainers.ContainerRequest{
    Image:        "postgres:13",
    ExposedPorts: []string{"5432/tcp"},
    Env: map[string]string{
     "POSTGRES_PASSWORD": "password",
     "POSTGRES_DB":       "testdb",
     "POSTGRES_USER":     "user",
    },
    WaitingFor: wait.ForAll(
     wait.ForListeningPort("5432/tcp"),
     wait.ForLog("database system is ready to accept connections"),
    ),
   }
   pgContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
    ContainerRequest: req,
    Started:          true,
   })
   assert.NoError(t, err)
   defer func() {
    err := pgContainer.Terminate(ctx)
    assert.NoError(t, err)
   }()

We then grab the port and host for the PostgreSQL container and construct the connection URL. The test then attempts to connect to the PostgreSQL database using pgx.Connect, retrying up to five times in case the connection fails initially. This ensures the test can handle slight delays in the container becoming ready..

   mappedPort, err := pgContainer.MappedPort(ctx, "5432")
   assert.NoError(t, err)

   host, err := pgContainer.Host(ctx)
   assert.NoError(t, err)

   dbUrl := fmt.Sprintf("postgres://user:password@%s:%s/testdb", host, mappedPort.Port())

   // Connect to the PostgreSQL container
   var conn *pgx.Conn
   for i := 0; i < 5; i++ {
    conn, err = pgx.Connect(ctx, dbUrl)
    if err == nil {
     break
    }
   }
   assert.NoError(t, err)
   defer func() {
    if conn != nil {
     err := conn.Close(ctx)
     assert.NoError(t, err)
    }
   }()

We call the setupDatabase function defined in the test case to create the necessary database schema and insert test data. The global db variable is then set to the active database connection, allowing the GetUsersHandler to query the database. An HTTP GET request is created and executed against the /users endpoint using httptest.NewRecorder to capture the response.

   // Setup database and insert test data
   err = testCase.setupDatabase(conn, ctx)
   assert.NoError(t, err)

   // Assign the global db variable for handler to use
   db = conn

   // Setup the HTTP request and response recorder
   reqHttp, err := http.NewRequest("GET", "/users", nil)
   assert.NoError(t, err)

   rr := httptest.NewRecorder()
   handler := http.HandlerFunc(GetUsersHandler)
   handler.ServeHTTP(rr, reqHttp)

Finally, the response from the handler is validated. The status code is checked to ensure it matches http.StatusOK. The response body is read and unmarshaled into a map[string]interface{}. The "data" field, initially a slice of interface{}, is converted to a slice of map[string]interface{} for proper comparison with the expected data. The assert.Equal statements then check that the response message, status code, and data match the expected values specified in the test case.

This structured approach ensures that the API handler is correctly processing requests and interacting with the database as expected, with clear validation of the outcomes.

   // Check the status code is what we expect
   assert.Equal(t, http.StatusOK, rr.Code)

   // Check the response body is what we expect
   var response map[string]interface{}
   body, err := ioutil.ReadAll(rr.Body)
   assert.NoError(t, err)

   err = json.Unmarshal(body, &response)
   assert.NoError(t, err)

   // Convert the "data" field from []interface{} to []map[string]interface{} for comparison
   actualData, ok := response["data"].([]interface{})
   assert.True(t, ok)

   var actualDataConverted []map[string]interface{}
   for _, item := range actualData {
    actualDataConverted = append(actualDataConverted, item.(map[string]interface{}))
   }

   // Assert the JSON response
   assert.Equal(t, "success", response["message"])
   assert.Equal(t, float64(200), response["status_code"])
   assert.Equal(t, testCase.expectedData, actualDataConverted)
  })
 }
}

Running the Project

1. Clone the project and navigate to the project directory.

git clone git@github.com:pgaijin66/go-testcontainers.git
cd go-testcontainers

2. Install dependencies

go mod tidy

3. Execute unit tests using Testcontainters

$ task test

task test
task: [build] go build -o app main.go
task: [test] go test -v ./...
=== RUN   TestGetUsersHandler
=== RUN   TestGetUsersHandler/Handle_multiple_users
2024/09/02 08:41:04 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 26.1.1
  API Version: 1.45
  Operating System: Docker Desktop
  Total Memory: 7840 MB
  Labels:
    com.docker.desktop.address=unix:///Users/pthapa/Library/Containers/com.docker.docker/Data/docker-cli.sock
  Testcontainers for Go Version: v0.33.0
  Resolved Docker Host: unix:///var/run/docker.sock
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: b68b3045cc70383fc283797770c787a3f7cb5ca078ce28e9b95ce78b290caad4
  Test ProcessID: 0605e174-1000-478d-b17e-cacc9be55c2a
2024/09/02 08:41:04 🐳 Creating container for image testcontainers/ryuk:0.8.1
2024/09/02 08:41:04 ✅ Container created: 5a4326a9432c
2024/09/02 08:41:04 🐳 Starting container: 5a4326a9432c
2024/09/02 08:41:04 ✅ Container started: 5a4326a9432c
2024/09/02 08:41:04 ⏳ Waiting for container id 5a4326a9432c image: testcontainers/ryuk:0.8.1. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms skipInternalCheck:false}
2024/09/02 08:41:04 🔔 Container is ready: 5a4326a9432c
2024/09/02 08:41:04 🐳 Creating container for image postgres:13
2024/09/02 08:41:04 ✅ Container created: 3a3547b2adb4
2024/09/02 08:41:04 🐳 Starting container: 3a3547b2adb4
2024/09/02 08:41:05 ✅ Container started: 3a3547b2adb4
2024/09/02 08:41:05 ⏳ Waiting for container id 3a3547b2adb4 image: postgres:13. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0x1400046fbc0 0x1400046fbf0]}
2024/09/02 08:41:05 🔔 Container is ready: 3a3547b2adb4
2024/09/02 08:41:05 🐳 Terminating container: 3a3547b2adb4
2024/09/02 08:41:05 🚫 Container terminated: 3a3547b2adb4
--- PASS: TestGetUsersHandler (1.34s)
    --- PASS: TestGetUsersHandler/Handle_multiple_users (1.34s)
PASS
ok   my-app 1.965s

Link to repository: https://github.com/pgaijin66/go-testcontainers

Conclusion

In this article, we’ve explored the power of combining Go and Testcontainers to create a robust, testable REST API. By leveraging Testcontainers, we’ve demonstrated how to set up a consistent and isolated testing environment that closely mimics our production setup. This approach allows us to test our API against a real database instance, ensuring that our tests accurately reflect real-world scenarios.

I know this article was long, so here are the key takeaways:

1. Testcontainers provides isolated, disposable instances of services like databases, ensuring that each test runs in a clean environment, which improves the reliability of integration tests.
2. By using real services wrapped in Docker containers, Testcontainers allows you to test your Go APIs against actual service instances, making your tests more reflective of real-world scenarios compared to using mocks or in-memory services.
3. Testcontainers ensures that tests run consistently across different environments, as the same Docker images are used regardless of the machine or CI/CD pipeline, reducing the likelihood of environment-related issues.
4. Testcontainers automates the setup and teardown of complex dependencies, such as databases, making it easier to manage and test against them without needing manual Docker or Docker Compose configurations.
5. Testcontainers can be effectively used in a Test-Driven Development (TDD) approach, ensuring that your Go APIs are thoroughly tested as they are being developed.
6. By testing against specific versions of services and supporting multiple databases, Testcontainers helps ensure that your code is resilient to changes in the environment, leading to more stable software in production.
7. Incorporating Testcontainers into your testing strategy can significantly enhance the robustness and reliability of your software, ultimately leading to more dependable applications in production.

P.S: I have started writing article focused on reliability engineering as a newsletter. If you want to get these article delivered to your mailbox, then please subscribe to my SRE/DevOps Newsletter: ( https://reliabilityengineering.substack.com ).