During my last mission, I was working with a team that invested in testing to secure a Symfony legacy application and find ways to prevent some of the regressions. This was a really good move.

They mainly had two types of tests:

• Some tests were written with CucumberJs, automating real user workflows on the user interface via Puppeteer. Execution of these tests involved a lot of HTTP requests, database calls, code logic execution, and wait time for rendering.
• Some tests used PhpUnit, written with Symfony test clients, going all the way down to the database and back again.

These two test suites were slow. The first one took around 50 minutes on the build machine. The team decided to avoid running it in the CI on every push and ran them only at merge time. The second one took a little more than 20 minutes. Suffice it to say that this was too long to run the tests after every code change.

The company decided to invest a few days of my presence to speed up the tests. I worked on the PhpUnit test suite and managed to reduce the time by a factor of 5.

Having an impact on the entire test suite meant looking for a global solution and avoiding trying to edit every test one by one to make them faster. That would just take too long.

Parallelization with Paratest

So I went with parallelization. Instead of running each test one after the other, let’s run multiple tests at the same time, taking advantage of the multiple processors of our computers.

In PHP, we can use Paratest, a tool that will run multiple tests from PhpUnit at the same time.

The first step was to install Paratest with composer require --dev brianium/paratest. Then, run it with vendor/bin/paratest.

It worked; tests were running in parallel, but this was just a first step because a lot of tests started to fail.

The vast majority of tests were tied to the database, which is acting as a shared resource. This was the main cause of the new failures. Because the database is shared and multiple tests are running in parallel, they interact with each other via the database. Imagine that test A is trying to read some data from the database while test B deletes that data. If A passes before B, the test will pass, but if B passes before A, test A will fail.

I've explored multiple solutions to the problem of the shared database for parallel tests in a past article (in French). Some of them were not applicable in our case.

Removing the dependency on the database during testing would require too much work because the code had a tight coupling with the database via heavy usage of an ORM. Using a fake for the database wasn't possible.

Decorating each test with a transaction wasn't possible either because the framework was tweaked and used multiple connections to the database side-by-side.

The solution left was to use one database for each process. Each one of Paratest's runners will have its own database, which will prevent tests from interacting via the database.

We will return to the creation of a database for each test process later, but for now, let's focus on another issue: how can the application know which database to connect to?

Connecting to the right database

When Paratest starts a new process, it injects an environment variable named TEST_TOKEN which contains an identifier for the process. This is just an integer, starting from 1 up to the number of started processes.

Let's say that test databases are created using the original test database's name suffixed with the test token. For instance, if the original test database name is test_db, the first process database is named test_db_1.

The database name is passed to the application via an environment variable DATABASE_NAME. It's now a matter of combining the information coming from the two environment variables, DATABASE_NAME and TEST_TOKEN.

To do this, I created a Symfony Environment Variable Processor, which allows us to manipulate the environment variable used in parameters.

In the application config, a database_name parameter exists and is reused for every service definition in need of knowing which database to connect to. To read the DATABASE_NAME environment variable, it was originally defined as:

parameters:  
    database_name: '%env(DATABASE_NAME)%'

I changed it to:

parameters:  
    database_name: '%env(testdb:DATABASE_NAME)%'

The addition of testdb: indicates to the Symfony container to use the TestDBEnvVarProcessor to process the environment variable.

Here is the environment variable processor class:

final class TestDBEnvVarProcessor implements EnvVarProcessorInterface  
{  
    public function getEnv($prefix, $name, \Closure $getEnv)  
    {  
        $env = $getEnv($name);  
  
        $testToken = $getenv('TEST_TOKEN');  
        if($testToken === false) {  
            return $env;  
        }  
  
        return $env . '_' . $testToken;  
    }  
  
    public static function getProvidedTypes()  
    {  
        return [  
            'testdb' => 'string',  
        ];  
    }  
}

The getEnv method is there to transform the original environment variable to the test database name.

$getEnv($name) gives us the value of DATABASE_NAME, test_db in our case.

Then, we try to see if a TEST_TOKEN environment variable is defined. If it's not the case, we keep the database name as it is. If we have a TEST_TOKEN, we concatenate the original name with the token.

And we need to tell the framework where to look for the environment variable processor with a new service declaration.

services:
	XXX\YYY\TestDBEnvVarProcessor:  
	    tags: ['container.env_var_processor']

The process-scoped database will now be used when running with Paratest, and the original test database will keep being used if tests are run with PhpUnit.

A simpler and better way to achieve this

While I was writing this article, it occurred to me that a simpler solution to this problem exists: we can rewrite the DATABASE_NAME environment variable right in the PhpUnit bootstrap file.

$testToken = getenv('TEST_TOKEN');  
  
if($testToken !== false) {  
    $db = getenv('DB_NAME');  
    $testDB = $db . '_' . $testToken;  
    putenv("DB_NAME=${testDB}");  
}

I don't know why I didn't use that solution at that time. Maybe I tried it, missed something, failed, and found the other solution or just didn't think about it.

Anyway, I think this is a better solution for multiple reasons:

• It's less and simpler code.
• It doesn't require modifying the application; the environment variable modification is made outside of it. This is better because the application should just take an environment variable and trust that it was provided with the right information.
• Because the environment variable modification code is written in the test framework bootstrap, it is run only in test mode. Only when it's needed and no risk of issues in production.

If you have to choose between the two solutions, I think you should pick this one. I included the other solution in the article because this is how I actually did it for that project and to share the existence of Symfony's Env Var Processors. Using them for this use case looked like a good idea but maybe isn't after all.

Creating the test databases

So far, the only test database was created using a migration script and some data seeders during PhpUnit bootstrap.

We could reuse that data setup mechanism for every database. The only issue is that it was slow. When you launched the tests you had to wait for a little while - I don't recall exactly but I would say around 1 minute - before the tests started to be executed.

We wanted to avoid each process from waiting for its own migration to be made, and I found a solution to that problem.

But first, a little detour.

Paratest, PhpUnit, and the bootstrap file

When tests are run with Paratest, the PhpUnit bootstrap is called multiple times:

• Once before starting Paratest processes
• Once per process

You can see that by adding this line to your PhpUnit bootstrap:

file_put_contents(__DIR__ . '/log.txt',  "In phpunit bootstrap.php ${testToken} \n", FILE_APPEND);

After running Paratest with two processors, using ./vendor/bin/paratest -p 2, and 3 test classes, you'll see the following in log.txt:

In phpunit bootstrap.php    
In phpunit bootstrap.php 1   
In phpunit bootstrap.php 2
In phpunit bootstrap.php 1

It took me a while to discover that. I was facing weird behaviors that I didn't understand. I tried adding some echo statements, but because they were in other processes, I couldn't see them. I tried debugging, which is a super fun thing to do because you need to pass the debug settings to processes started by Paratest^[1].

Why was this detour interesting?

We've learned that the bootstrap file is run once per process, which means we can set up each database's data in its own process. This has a couple of advantages:

• We don't need to know how many processes Paratest will use prior to creating the databases. We will always create the right number of databases.
• The setup for each database is made in parallel, which means faster setup than a sequential one.

Also, we've learned that the bootstrap file is called once per test class. We can see that thanks to the repeated In phpunit bootstrap.php 1 line.

PhpUnit's bootstrap seemed to be a good place to create the databases for each processor. Even better, because we were going through the bootstrap for each test class, we could have a new database for each one of them. This is great because I was also on a side quest of removing test flakiness due to coupling via the shared resource that is the database. But, first, let's create multiple databases.

Creating multiple test databases

Below the pseudo code of my original try at creating all the databases:

function testDBName(): string {  
    return 'test_db_' . getenv('TEST_TOKEN');  
}

if (getenv('TEST_TOKEN') === false) {
	// Migrate and seed with original database name
}
else {
	// Migrate and seed with testDbName()
}

If we had no TEST_TOKEN set, either because we were not using Paratest or because we were in the first bootstrap, we run the migration and seeding as we did before. If the TEST_TOKEN was specified, migrate and seed the specific database.

This solution comes with a couple of issues:

• Even when we are using Paratest, we are creating the original database we won't need during the test execution. This is a loss of time.
• Worse, because the migration and seeding step is slow, the test suite execution time is now longer than it was before as that step is repeated for each and every test class. Running that in parallel doesn't compensate for that at all.

From slow to fast...

So here we are, with tests running in parallel and a test suite slower than it was before because of database seeding.

By chance, the other test suite, the one running on CucumberJs, offered me a solution. To improve the setup of tests for that test suite, the team had already tried an idea to improve database setup time.

Instead of running migration and data creation for each test, they created a dump of the database and reused it for every test. I could totally steal that idea and reuse it to speed up the PhpUnit test suite as well.

I changed the PhpUnit bootstrap file to migrate and seed the database and dump its content to a file when the TEST_TOKEN environment variable is missing. This ensures that we always have an up-to-date database including the latest schema and data before running the test. When the TEST_TOKEN environment variable is set, instead of going through the original database process, I used the dump to populate the test process dedicated database.

if (getenv('TEST_TOKEN') === false) {
	// Migrate and seed with original database name

	// Dump the content of the database
	$dumpCommand = "PGPASSWORD='${POSTGRES_PASSWORD}' pg_dump --no-owner -h ${POSTGRES_HOST} -U ${POSTGRES_USER} -d ${POSTGRES_DB} -Fc > ${DUMP_FILE}";
	exec($dumpCommand);
}
else {

	$testDbName = testDBName();
	
	// Drop the database if it exists, this prevents error when trying to create an already existing DB
	$createDabataseCommand = "PGPASSWORD='${POSTGRES_PASSWORD}' psql -h ${POSTGRES_HOST} -U ${POSTGRES_USER} -c 'DROP DATABASE IF EXISTS ${testDbName}'";  
	exec($createDabataseCommand);

	// Create the database
	$createDabataseCommand = "PGPASSWORD='${POSTGRES_PASSWORD}' psql -h ${POSTGRES_HOST} -U ${POSTGRES_USER} -c 'CREATE DATABASE ${testDbName}'";  
	exec($createDabataseCommand);

	// Restore the dump
	$restoreDumpCommand = "PGPASSWORD='${POSTGRES_PASSWORD}' pg_restore -h ${POSTGRES_HOST} -U ${POSTGRES_USER} -d ${testDbName} ${DUMP_FILE}";  
	exec($restoreDumpCommand);
}

The dump restoration trick combined with the parallel run reduced the test duration by a factor of 5. Still a bit slow to my taste to be able to TDD, but a vast improvement nevertheless. Running tests on local machines multiple times per hour started to be possible.

As you can see, while some ideas are applicable to multiple codebases, this is purely an ad-hoc solution. The solution was created with a lot of trial and error, sweat, hurray moments, and painful setbacks.

The point here is that a super slow test suite is something you can escape. You probably won't have a super fast test suite without a lot of test rework if your tests are all making database calls, but you can improve your daily life up to some point. The next steps are to decouple business logic code from infrastructure, use unit tests for the former and integration tests for the latter. This is more work and depending on the state of your codebase can be a difficult thing to do. Sometimes it can be helpful to get some assistance to go through these kinds of challenges. If you feel like your codebase or your test suite could benefit from some rework to make your life easier and speed up your delivery, let’s chat!