Testcontainers' customization for Postgres tests

Go's Testcontainers ephemeral Postgres tests

- 5 mins read

Series: Testcontainers

Testcontainers allows you to test your code with ephemeral containers right inside your tests. It provides different modules for simplifying the process, however, sometimes you many need to customize the container beyond the default parameters or it contents.

Source Code of the laboratory. All examples are functional, follow the instructions in the README file for setting up. The csv file is generated by the Docker Compose setup, you can regenerate the example by running docker compose up -d.

Testcontainers offers a generic container API, and for specific services, it provides modules that are helpers with the most common settings. Available modules can be found here.

In this post we will through some of the considerations in the latest API updates in Testcontainers v0.31.0.

If you don’t have a good idea what Testcontainers can do, I’ll try to ellaborate like this:

Running ephemeral containers of any kind for doing (mostly) integration or functional tests.

It is very convinient if you happen to have a large amount of Services As A Depenpency, if such a term exists. But certainly, it plays a role in test-driven development, simplifying the test for complex service architectures. The modules’ documentation used in the laboratory can be found in the following links:

The laboratory can be executed as follows

go run main.go # generic run, no test
go test -v generic_test.go --args -imageName=postgres:16-bookworm
go test -v ts_test.go

or by executing ./e2e.sh.

The laboratory will load and execute the data migration, see the contents of the initialization scripts inside the test folder.

Using Generic Container request

Building a generic container requires to initialize by creating a request (ContainerRequest), and passing the request to the GenericContainer constructor. We also set a flag at the test script so it can be possible to change the image on each test. If you happen to use a non-Postgres-module compatible image, this might be your choice.

Full example at generic_test.go.

// go test -v main_test.go -args -imageName=...
var imageName = flag.String("imageName", "postgres:16-bookworm", "URL of the image")

	ctx := context.Background()

	req := tc.ContainerRequest{
		Image:        *imageName,
		ExposedPorts: []string{"5432/tcp"},
		Env: map[string]string{
			"POSTGRES_PASSWORD":         "postgres",
			"POSTGRES_HOST_AUTH_METHOD": "trust"},
		WaitingFor: wait.ForLog("Ready to accept connections"),
		Files: []tc.ContainerFile{
			{
				HostFilePath:      "test/generic",
				ContainerFilePath: "/docker-entrypoint-initdb.d",
				FileMode:          0o666,
			},
			{
				HostFilePath:      "test/containerdata/devices.csv",
				ContainerFilePath: "/tmp/devices.csv",
				FileMode:          0o666,
			},
		},
	}

	postgresC, _ := tc.GenericContainer(ctx, tc.GenericContainerRequest{
		ContainerRequest: req,
		Started:          true,
	})
	defer func() {
		if err := postgresC.Terminate(ctx); err != nil {
			t.Fatalf("failed to terminate container: %s", err)
		}
	}()

Using Postgres Module with a non-vanilla image

What about images that are Postgres services but they aren’t a vanilla image? This module support different images that are compatible with the official Postgres image. An example of such could be the Timescale image, which is based on the official Postgres image, but adds the extension and its initialization.

The Postgres module allows starting the container in a single step, and provides a set of helper functions to extract the container information

Full example at ts_test.go.

	ctx := context.Background()
	dbName := "iot"
	dbUser := "postgres"
	dbPassword := "password"

	usageData := filepath.Join("test/containerdata", "devices.csv")
	r, err := os.Open(usageData)
	if err != nil {
		t.Fatal(err)
	}
	postgresContainer, err := postgres.RunContainer(ctx,
		tc.WithImage("timescale/timescaledb:latest-pg16"),
		// We execute the generator with docker-compose, so we have a deterministic test
		// postgres.WithInitScripts(filepath.Join("test/containerdata", "003_generator.sql")),
		postgres.WithInitScripts(filepath.Join("test/timescale", "004_init.sql")),
		postgres.WithInitScripts(filepath.Join("test/timescale", "005_load.sql")),
		postgres.WithDatabase(dbName),
		postgres.WithUsername(dbUser),
		postgres.WithPassword(dbPassword),
		tc.CustomizeRequest(tc.GenericContainerRequest{
			ContainerRequest: tc.ContainerRequest{
				Files: []tc.ContainerFile{
					{
						Reader:            r,
						HostFilePath:      usageData,
						ContainerFilePath: "/tmp/devices.csv",
						FileMode:          0o666,
					},
				},
			}}),
		tc.WithEnv(map[string]string{
			"TS_TUNE_MEMORY":   "1GB",
			"TS_TUNE_WAL":      "1GB",
			"TS_TUNE_NUM_CPUS": "2"}),
		tc.WithWaitStrategy(
			wait.ForLog("database system is ready to accept connections").
				WithOccurrence(2).
				WithStartupTimeout(10*time.Second)), // we add a large startup due that we are loading data
	)

	// Database pointer creation
	connStr, err := postgresContainer.ConnectionString(ctx, "sslmode=disable")
	if err != nil {
		log.Fatalf("failed to get connection string: %s", err)
	}
	db, err := sql.Open("postgres", connStr)
	if err != nil {
		log.Fatalf("failed to open database: %s", err)
	}
	defer db.Close()

	if _, out, err := postgresContainer.Exec(ctx, []string{"psql", "-U", dbUser, "-w", dbName, "-c", `SELECT count(*) from devices;`}); err != nil {
		log.Println(err)
		t.Fatal("couldn't count devices")
	} else {
		// read io.Reader out
		io.Copy(os.Stdout, out)
	}

Using docker compose

Another way would be reusing a Docker Compose definition.

Full example at compose_test.go.

networks:
  # A network for the data traffic 
  data:

services:
  timescale:
    image: timescale/timescaledb:latest-pg16
    ## Once in prod
    # restart: always
    container_name: "pgtc-ts"
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=iot
      - TS_TUNE_MEMORY=1GB
      - TS_TUNE_WAL=1GB
      - TS_TUNE_NUM_CPUS=2
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5
    networks:
      - data
    ports:
      - 15432:5432
    volumes:
      - ./_pgdata:/var/lib/postgresql/data
      - ./test/timescale:/docker-entrypoint-initdb.d
      - ./test/containerdata/devices.csv:/tmp/devices.csv

import (
	"context"
	"testing"

	"github.com/stretchr/testify/require"
	tc "github.com/testcontainers/testcontainers-go/modules/compose"
)

func TestSomething(t *testing.T) {
	compose, err := tc.NewDockerCompose("docker-compose.yaml")
	require.NoError(t, err, "NewDockerComposeAPI()")

	t.Cleanup(func() {
		require.NoError(t, compose.Down(context.Background(), tc.RemoveOrphans(true), tc.RemoveImagesLocal), "compose.Down()")
	})

	ctx, cancel := context.WithCancel(context.Background())
	t.Cleanup(cancel)

	require.NoError(t, compose.Up(ctx, tc.Wait(true)), "compose.Up()")

	// Do tests...
}

You may wondering about the number prefix in this example. Keep in mind that some Postgres images may have included other files in the docker-entrypoint-initdb.d folder. eg., the Timescale image comes with 2 files, so we don’t want to override them. The order of the files execution is relevant, so beware of this if your operations do require several steps.

This is a post in the Testcontainers series.
Other posts in this series:

  • May 26, 2024 - Testcontainers' customization for Postgres tests

comments powered by Disqus