Charles Desneuf

Using PrismJs with AngularJs

2013-10-18T00:00:00Z

In a project I’m working on I need to display code snippets. I found PrismJs library, which is lightweight and cover a lot of languages. The application is based on AngularJs, and code snippets are displayed after having been read from an API and PrimsJs is run before code is injected in the template, when used as the documentation said, so highlighting doesn’t work. The problem can be tackled down with the creation of a directive. Here is the code :

angular.module('Prism', []).
    directive('prism', [function() {
        return {
            restrict: 'A',
            link: function ($scope, element, attrs) {
                element.ready(function() {
                    Prism.highlightElement(element[0]);
                });
            }
        } 
    }]
);

For each HTML element marked with the prism attribute we wait that the inner DOM is fully loaded, using the ready function, and then trigger Prism highlighting on it.

<pre>
    <code class="language-markup" prism>
    {{ snippet }}
    </code>
</pre>

Here is the Gist !

In modal nested states with AngularJs ui-router

2014-01-15T00:00:00Z

In an AngularJs application I'm working on we need to display details about an object on a modal and to be able to switch to an edit mode in the same modal.
An other requirement is that the modal is automatically opened when the user enters the application with a url pointing to an edit or a view mode.

We are using the modal system made by Ui Bootstrap and the Ui Router both created by the AngularUi team.

Ui Router allows to organize the interface using a state machine.

As described in the introduction we need at least three states. The first one shows a list of items, and the two others are our modal's states (view and edition of the item).

We also don't want the modal to close and open when switching between view and edition mode. Here helps a really cool feature of Ui Router : abstract states. An abstract state is a state that cannot be activated itself but can have child states. We use an abstract state to deal with the modal opening, view and edit mode are its child states.

Here is a plunker of the result :

In our application config we define a state for the list of items:

$stateProvider
  .state('list', {
    url: '/',
    template: '<ul><li ng-repeat="pony in ponies"><a ui-sref="view({id: pony.id})"></a></li></ul>',
    controller: function($scope) {
      $scope.ponies = ponies;
    }
  });

We add the abstract state :

$stateProvider.state('modal', {
  abstract: true,
  parent: 'list',
  url: '',
  onEnter: ['$modal', '$state', function($modal, $state) {
      console.log('Open modal');
      $modal.open({
        template: '<div ui-view="modal"></div>',
        backdrop: false,
        windowClass: 'right fade'
      }).result.finally(function() {
        $state.go('list');
    });
  }]
});

The abstract state is a child of the list state and therefore can be activated at the same time of the list. When this state is activated a modal is opened and it contains a basic template with an other ui-view name modal. We also use $modal promise system to go back to the list state when the modal is closed.

We finally add the edit and view states :

$stateProvider.state('view', {
  url: ':id',
  parent: 'modal',
  views: {
    'modal@': {
      template: '<h1></h1><br />\
      <a ui-sref="edit({id: pony.id})">Edit</a><br />\
      <a href="#" ng-click="$close()">Close</a>',
      controller: function($scope, pony) {
        $scope.pony = pony;
      },
      resolve: {
        pony: function($stateParams) {
          return ponies[$stateParams.id];
        }
      }
    }
  }
})
.state('edit', {
  url: ':id/edit',
  parent: 'modal',
  views: {
    'modal@': {
      template: '<h1>Edit </h1><br /> \
        <a ui-sref="view({id: pony.id})">View</a> <br />\
        <a href="#" ng-click="$close()">Close</a>',
      controller: function($scope, pony) {
        $scope.pony = pony;
      },
      resolve: {
        pony: function($stateParams) {
          return ponies[$stateParams.id];
        }
      }
    }
  }
});

They both inherit from the modal state and set the content of the modal view.
We use resolve to load the needed item accordingly to the url parameter, this allows the use to reach the application with an url such as /2/edit and to see both the items list and the modal open on edition mode. Cool huh ?

Here is the full sample of code which allows you to have two states in the same modal :

angular.module('app', ['ui.router', 'ui.bootstrap'])
  .config(['$stateProvider', '$urlRouterProvider', function($stateProvider, $urlRouterProvider) {
    
    $urlRouterProvider.otherwise('/');
    
    var ponies = [{
      id: 0,
      name: 'Griotte'
    }, {
      id: 1,
      name: 'Eole'
    }, {
      id: 2,
      name: 'Rox'
    }];
    
    $stateProvider
      .state('list', {
        url: '/',
        template: '<ul><li ng-repeat="pony in ponies"><a ui-sref="view({id: pony.id})"></a></li></ul>',
        controller: function($scope) {
          $scope.ponies = ponies;
        }
      })
      .state('modal', {
        abstract: true,
        parent: 'list',
        url: '',
        onEnter: ['$modal', '$state', function($modal, $state) {
            console.log('Open modal');
            $modal.open({
              template: '<div ui-view="modal"></div>',
              backdrop: false,
              windowClass: 'right fade'
            }).result.finally(function() {
              $state.go('list');
          });
        }]
      })
      .state('view', {
        url: ':id',
        parent: 'modal',
        views: {
          'modal@': {
            template: '<h1></h1><br />\
            <a ui-sref="edit({id: pony.id})">Edit</a><br />\
            <a href="#" ng-click="$close()">Close</a>',
            controller: function($scope, pony) {
              $scope.pony = pony;
            },
            resolve: {
              pony: function($stateParams) {
                return ponies[$stateParams.id];
              }
            }
          }
        }
      })
      .state('edit', {
        url: ':id/edit',
        parent: 'modal',
        views: {
          'modal@': {
            template: '<h1>Edit </h1><br /> \
              <a ui-sref="view({id: pony.id})">View</a> <br />\
              <a href="#" ng-click="$close()">Close</a>',
            controller: function($scope, pony) {
              $scope.pony = pony;
            },
            resolve: {
              pony: function($stateParams) {
                return ponies[$stateParams.id];
              }
            }
          }
        }
      });
    }]);

Add more element forms the nice way with AngularJs

2014-02-15T00:00:00Z

We sometimes have to deal with forms where user can add items to a list. Most of the time an "Add more" button is created and we add an input to the form when it's clicked. I'll show you in this post how to build this type of form using AngularJs form handling mechanism.

Here is the result of what will be discussed on this post, with some more things like CSS and placeholders :

As you can see the "Add an element" button is disabled if an item of the list is invalid (they are all required) and the "Save list" is disabled if one element of the form is invalid.

This example makes a deep use of the form directive. AngularJs allows to interlock form directives. Each directive will regiter itself upon the one above in the DOM tree. Using a form directive is helpful when you have to check the validity of several inputs without having to do a combination of all inputs states.

We will use the "ng-form" version of the directive because form tag can't contain an other form tag.

First create a basic form for the list title. The submit button is disabled when the form is not valid. The form tag is one of the way to create a form directive.

<form name="listForm" ng-submit="saveList()" novalidate>
    <label for="list-title">List title</label>
    <input id="list-title" type="text" ng-model="list.title" required />
    <input type="submit" value="Save list" ng-disabled="listForm.$invalid">
</form>

Use ng-repeat for the item inputs. Here two more forms are created using the ng-form attribute : one for the list of element (itemsForm) and one for each element (itemForm). We are now able to do validation for one item and for the list. User can't add an item to the list if one of the existing elements is invlaid because the "Add an element" button is disabled when itemsForm is not valid.

<form name="listForm" ng-submit="saveList()" novalidate>
    <label for="list-title">List title</label>
    <input id="list-title" type="text" ng-model="list.title" required />
    <b>List items</b>
    <ul ng-form="itemsForm">
        <li ng-repeat="item in list.items" ng-form="itemForm">
            <label for="itemText">List item</label>
            <input name="itemText" type="text" ng-model="item.text" required />
            <ul ng-show="itemForm.$invalid && itemForm.$dirty">
                <li ng-show="itemForm.itemText.$error.required">This field is required.</li>
            </ul>
        </li>
    </ul>
    <button ng-click="addElement()" ng-disabled="itemsForm.$invalid">Add an element</button>
    <input type="submit" value="Save list" ng-disabled="listForm.$invalid">
</form>

The code of the associated controller is easy to understand : when the addElement function is called a new empty object is pushed in the array of items and a new row will be displayed in the form.

angular.module('formDemo')
  .controller('FormCtrl', function ($scope) {

    $scope.list = {
        name: '',
        items: [{}]
    };

    $scope.addElement = function() {
        $scope.list.items.push({});
    };

    $scope.saveList = function() {
        alert('This is a dummy button but let\'s say the list has been saved !');
    };    
  });

If you have a question or want to discuss about this solution you can find me on Twitter or comment this post.

Smurfing a Twitter bot

2015-02-10T00:00:00Z

The idea

The idea came up a while ago when I was talking with colleagues : Would it be possible to make a bot able to retweet speaking as a Smurf ?

Smurfs, or Schtroumpfs in french, are an imaginary population of small blue people living in a mushroom village. They were born in the imagination of french illustrator Peyo. They speak a strange language, which is based on human language but where they replace some words with smurf.

I'm a follower of french newspaper Le Monde on Twitter (@lemondefr) and I decided that
it would be a good test to try to smurf their tweets.

Language analysis

The analysis of sentences is something complicated and might be harder in french than in some other language. I didn't even imagine for a second I could get along with this project without a good parser.

I searched the web for a few hours and finally discovered the work done by Alpage, a research group working at INRIA, the french computer science search institute.
They have created a set of tools that can analyse a sentence and returns lexems.

You can try it by yourself if you can speak a few words of french.

I tried to install the Alpage system but failed to do so and finally decided to make a call to a demo page, get the result and work with the received data.

Word transformation

Once the sentence is parsed into lexems the next step is to replace word by its smurf equivalent. Smurf language is not just about changing a word to smurf: you have to replace it with the smurf word in the same category.

Category	Smurf form	French word example
Adverb	schtroupfement	admirablement
Past verb, plural form	schtroumpfaient	venaient
Noun, plural form	schtroumpfs	poneys

Examples of french-smurf translations

I wrote a basic french-smurf translation dictionnary to help me with the transformation.

Doing the transformation of one word is not enough. I also had to transform previous words in some cases where gender and number influence the preceding pronoun.

Smurf form	French example
Je schtroumpfe	J'arrive
Du schtroumpf	De l'eau

Example of influence of gender and number on the translation

I didn't try to be clever in the selection of the french word to transform : this is a total random thing.

Integration with Twitter

The project is written in NodeJS and Twit does a good job when it comes to listen Twitter's stream API and to post a tweet so I didn't search for too long and decided to go with it.

The bot listen the stream API for each tweet from @lemondefr and delegate the transformation work to the transformation tool.

Once the text transformed the bot makes the call to tweet the smurfed version of the text.

Release

As always I had a hard time convincing myself I should release an uncompleted project, with some known bugs and possible improvements, but I finally did it.
The bot is hosted on a DigitalOcean droplet.

Some bugs are persisting and I have to manually restart the bot once in a while. I also make some changes in the transformation algorithm when a tweet is not doing so well.

Results

The bot is living is own life out there. Go and follow it !

As the transformed word is selected randomly luck plays a good part in the quality of the generated tweet.

The bot is able to do some cool sentences such as :

Droit de vote des schtroumpfs : Londres condamné à Strasbourg http://t.co/0mV5ahnxoK
— Le Schtroumpf (@leschtroumpffr) 10 Février 2015

Les experts divisés sur un schtroumpf attribué à Léonard de Vinci saisi en Suisse http://t.co/waHvCuE90i
— Le Schtroumpf (@leschtroumpffr) 10 Février 2015

Harvard interdit les relations sexuelles entre professeurs et schtroumpfs http://t.co/E1ELllkpzB sur @Campus_LeMonde pic.twitter.com/3nVVjIVJAE
— Le Schtroumpf (@leschtroumpffr) 9 Février 2015

Sometimes we are less lucky :

Spider-Man rejoint l'univers schtroumpf de Marvel http://t.co/FP4Xd40mlo
— Le Schtroumpf (@leschtroumpffr) 10 Février 2015

See the code

The code is freely available on github.
Let's say it upfront : it's really not a good piece of software but the job is done.

Hey ! I'm on Twitter too, if you want to chat about the bot or something else. Feel free to comment below as well.

TShirt a day

2015-07-06T00:00:00Z

I’ve been reading a lot of books and blog posts, watching a lot of conferences about development for the past year and I was searching a project to help me wrap my head around all the interesting concepts I’ve learned.

Some day I received another t-shirt from a punk rock band - I have so many t-shirts... - and had an idea : "What if I made a service where people could choose what t-shirt I should wear on a given day ?"

This is stupid but can be a decent project to play with Domain Driven Design concepts, to build a true Restfull API (ouch...), to work on some front-end again, to (maybe) try to use docker as a container, to do some Behaviour Driven Design, to practice my unit test writing skills...

If this project goes to production some day it could lead to a funny experimentation too because I have some t-shirts I probably should be embarassed to wear in some contexts.

(And if this goes to production this might lead to learn some more devops stuff. They say the more you do devops stuff the more you like doing it.)

This is project is open sourced, you can find it on Github and I’ll do my best to write some blog posts with the thing I’ve learned and struggle with.

Here are all the blog posts in the Tshirt a day serie :

Hey ! I’m on Twitter too, if you want to chat about this serie or something else. Feel free to comment below as well.

BDD and domain code

2015-07-06T00:00:00Z

Behat is a well known tool for end-to-end testing but as Konstantin Kudryashov pointed out in his article Introducing Modelling by Example it can be used as a modeling tool too.

As Konstantin points out writing features using an ubiquitous language shared by developers and stackholders reduce the cost of translation and thus the number of errors in the application.

In the t-shirt a day project one of the feature is that an admin can manage a t-shirt catalog. This feature is described using Gherkin language and scenario are tested using Behat.

Feature:
    As an admin
    I want to manage the TShirt Catalog

Scenario: Adding a TShirt to the catalog
    Given a new catalog is created
    When an admin adds a TShirt with description "La Ruda Japan Tour" to the catalog
    Then the catalog should contain 1 TShirt

Scenario: Removing a TShirt to the catalog
    Given a new catalog is created
    Given a TShirt with id 12345 is added to the catalog
    When an admin removes the TShirt with id 12345 from the catalog
    Then the catalog should contain 0 TShirt

As you can see scenarios are written using terms coming from the business and say nothing about the implementation, developers and stackholders are sharing the same language which remove one layer of translation.

The second layer of translation, from a language understood by developers to code, can be reduced using the DDD advice to use the ubiquitous language in code too. As you’ll see below the code expresses the feature requirements in a way that should allow non developers to read it with a little help.

<?php 
class ManageCatalogContext implements Context, SnippetAcceptingContext
{
    private $catalog;

    /**
     * @Given a new catalog is created
     */
    public function aNewCatalogIsCreated()
    {
        $this->catalog = new InMemoryCatalog;
    }

    /**
     * @When an admin adds a TShirt with description :description to the catalog
     */
    public function anAdminAddsATshirtWithDescriptionToTheCatalog($description)
    {
        $admin = new Admin;
        $admin->addTShirtWithDescriptionToCatalog($description, $this->catalog);
    }

    /**
     * @Then the catalog should contain :count TShirt
     */
    public function theCatalogShouldContainTshirt($count)
    {
        \PHPUnit_Framework_Assert::assertEquals($count, count($this->catalog->all()));
    }

    /**
     * @Given a TShirt with id :id is added to the catalog
     */
    public function aTshirtWithIdIsAddedToTheCatalog($id)
    {
        $this->catalog->add(new TShirt(new TShirtId($id), 'Dummy description'));   
    }

    /**
     * @When an admin removes the TShirt with id :id from the catalog
     */
    public function anAdminRemovesTheTshirtWithIdFromTheCatalog($id)
    {
        $admin = new Admin;
        $admin->removeTShirtWithIdFromCatalog(new TShirtId($id), $this->catalog);
    }
}
?>

As you can see the cost of translation has been reduced a lot using this approach.

As an aside I have to admit that I have some trouble writing user stories and the article What’s in a story helped me a lot and is a very good reference.

Hey ! I’m on Twitter too, if you want to chat about the bot or something else. Feel free to comment below as well.

Project macro structure

2015-07-12T00:00:00Z

File organization is something crucial if you have to maintain a project for the long run but is often overlooked by developers.

The way files are organized within a project is very important because it dictates how the team will work and at the end the overall quality of the software.

Using the traditional framework organization with a directory for each type of element (entities, services, controllers, ...) is one of the fastest way to produce tightly coupled pieces of software because all developers are working in the same area and have no boundaries to help them structure their code.

Furthermore this file organization doesn’t convey any information about what the software does. As Robert C. Martin noted in his conference "Architecture the lost years" we should be able to identify the purpose of a project by looking at its structure at the macro level.

Providing a file structure which expresses the goal of the software helps a lot in the definition of boundaries inside the project. This boundaries can be used to determine the teams working on the project too. Having boundaries is also the starting point to the analyze of dependencies within the system.

In the t-shirt a day project I have tried two files structure with the intent to give information about what the software does but I was unhappy with the first one that was not expressive enough in my opinion.

    .
    ├── README.md
    ├── behat.yml
    ├── bin
    ├── composer.json
    ├── composer.lock
    ├── phpunit.xml
    ├── src
    │   └── tshirtaday
    │       ├── catalog
    │       │   ├── features
    │       │   ├── src
    │       │   └── tests
    │       └── votes
    │           ├── features
    │           ├── src
    │           └── tests
    └── vendor

First file structure

In this first version I focused to much in my attempt to segregate domain code from a potential framework. I made a directory to support all domain code containing the two current modules (votes and catalog). This two modules were also containing their behat’s feature file.
The main issue with this approach is that you cannot tell what the system is about at first glance.

    .
    ├── README.md
    ├── behat.yml
    ├── bin
    ├── catalog
    │   ├── contexts
    │   ├── src
    │   └── tests
    ├── composer.json
    ├── composer.lock
    ├── features
    ├── phpunit.xml
    ├── vendor
    └── votes
        ├── contexts
        ├── src
        └── tests

Second file structure

After refactoring the file structure the two modules are now placed at the top level, which is a huge gain on the expressiveness of the system. Moreover the features' directory is also situated at the top level because they give a good description of what the entire system is about. I’m also trying to follow the example of Konstantin Kudryashov with is article Modeling by example and doing so might allow a feature to be accomplished by several modules altogether. For instance if I later introduce an API on top of the two modules I will need both the API module and the two others to interact to do the job.

This little refactoring helps showing the intent of the project. I’m currently happy with it and will see what will be the consequences of this decision during the lifetime of the project.

Hey ! I’m on Twitter too, if you want to chat about this approach or something else. Feel free to comment below as well.

Why are 404 pages still a thing ?

2015-07-20T00:00:00Z

In this article every time you’ll see "a 404 page" please read "The dumb 404 page with an apology and a link to the home page". This doesn’t concern 404 in other contexts, please keep - or start - using them in you APIs, for instance.

Context

I had a very interesting discussion with Rémi and Matthieu, two of my coworkers, today about caching. We encounter some performances issues on the Evaneos.com website and were arguing about the fact to place some cache between our API and our frontend.

For the sake of the argument we took the example of our itinerary list. View it as a product list on some other online store.

As we all agreed that setting a cache-control header stating that the current list of products will greatly improve performance issue because a copy could be kept in memory in the frontend website two points were made against having a cache.

First one is that the user adding the product to the list might want to see it right away on the website. Funny thing is that this type of user is also interested in having a really fast website and as we are working in the same office we probably can come to an arrangement balancing time before display and performances.

Second one is the one I want to tackle down here : "What if a user clicks on the link in the list and arrives on a 404 page because the product is no longer available?"

Note : I’m not taking Etag system into account here because they are slower - you still need to make a call to the API - and as you’ll see after don’t help in anything anyway.

You already have this problem

So, what happened?

A potential client sees a cached list of products. One item has been removed from the list, but the cache has not been updated yet because of the cache policy. The client finds her dream product, unfortunately that is the one we just removed, and clicks on it.
Let’s simplify here and say we are caching lists and not individual items, but this changes almost nothing. We now have the information that this product is not available - API returned 404/410 for instance - and our front website displays a 404 page to our client. This is bad UX.

Let’s try without cache.

A potential client sees a non cached list of products. She browses the list looking for her dream product. While she’s looking at the list someone using our backend removes a product. Finally, she founds the product she wants - guess which one? - clicks on it and arrives on a 404 page. This is bad UX.

We already have this issue.

We can pretend that the time between product list presentation and the client choosing a product is so short that the chances that we are removing a product during this time are low and decide not to care, but we will probably have an issue later anyway (ex: when the user adds the product to the basket).

Matthieu made an interesting observation when I bothered him later with all this, the website is not the only access to a product page : even when we don’t display a product in the list, search engines are still showing them anyway.

What do we do now?

We have a problem. Facing it is a good starting point.

In my point of view I don’t see why this second point should take us away from a massive performance boost because the problem is still here anyway without cache.

What about that 404 page?

We probably can do something cleverer than just showing a 404 page. We know what the customer was looking for, why not display to her similar products for instance?

We should go to business people and ask them what business opportunity we could take on this page instead of having a stupid 404 error.

Saying all this I don’t see why any online store would like to have dumb 404 pages and this is why I wonder why they are still a thing?

We can do better, improve UX, and as a business make more revenue.

This makes me think a lot about this quote by Udi Dahan :

"If you think you have a race condition, you don’t understand the domain well enough. These rules didn’t exist in the age of paper, there is no reason for them to exist in the age of computers. When you have race conditions, go back to the business and find out actual rules"

Race conditions are a hint that we probably don’t know enough and should ask more questions to business people, in order to come up with a better solution.

Hey ! I’m on Twitter too, if you want to chat about this or something else. Feel free to comment below as well.

Refactoring VoteValidator with tests

2015-07-21T00:00:00Z

A few days ago I decided that the t-shirt a day application doesn’t need the complexity of a vote validator with a validation handler and only a method stating if a vote is valid or not returning a boolean and then starting refactoring.

The process of recfactoring was not painfull thanks to the test suite already there but I must admit that I have spoted some flaws in my VoteValidator tests.

A cleaner VoteValidator

A vote is valid if :

The voter has note already voted for a Tshirt for the given day
The Tshirt has not been elected yet
A voting session is opened for that day

and I wanted the code to express this requirements clearly.

The isValid has been refactored to use 3 private methods each one dealing with one rule :

<?php
class VoteValidator
{
    public function isValid(Vote $vote)
    {
        return $this->voterHasNotAlreadyVotedForDay($vote->voterId(), $vote->day())
            && $this->tshirtExistsAndHasNotBeenElectedYet($vote->tshirtId())
            && $this->aVotingSessionIsOpenedForDay($vote->day());
    }
}
?>

This is the point I started to think that I probably should be able to switch lines and still have the test going green and tried to do so. Well, tests were failing.

Testing the right thing

I’ll take a simpler example that the one above with only two conditions to describe what I did wrong when I first wrote the tests.

Let’s define a simple class :

<?php
class Tuple
{
    public $a;

    public $b;

    public function __construct($a, $b)
    {
        $this->a = $a;
        $this->b = $b;
    }
}
?>

A tuple is valid if :

Element A exists
Element B exists

The tuple validator would have been something like this :

<?php
class TupleValidator
{
    private $elementARepository;

    public function __construct(ElementRepository $elementRepository)
    {
        $this->elementRepository = $elementRepository;
    }

    public function isValid(Tuple $tuple)
    {
        return $this->elementAExists($tuple->a)
            && $this->elementBExists($tuple->b);
    }

    private function elementAExists($a)
    {
        return $this->elementRepository->exists($a) !== null;
    }

    private function elementBExists($b)
    {
        return $this->elementRepository->exists($b) !== null;
    }
}
?>

In the TupleValidatorTest the method asserting that isValid returns false if A doesn’t exist would have been :

<?php
class TupleValidatorTest
{
    private $elementRepository;

    public function setUp()
    {
        $this->elementRepository = \Mockery::mock('ElementRepository');
    }

    /**
     * @test
     */
    public function should_return_false_if_A_doesnt_exist()
    {
        $a = 'A';
        $b = 'B';

        $tuple = new Tuple($a, $b);

        $validator = new TupleValidator($elementRepository);

        $this->elementRepository->shouldReceive('exists')->with($a)->andReturn(false);

        $this->assertFalse($validator->isValid($tuple));
    }
}
?>

The test will pass.

Now if I invert the two test in the isValid method :

<?php
// TupleValidator
public function isValid(Tuple $tuple)
{
    return $this->elementBExists($tuple->b)
        && $this->elementAExists($tuple->a);
}
?>

the test is failing because the ElementRepository mock wasn’t expecting a call for "exists" method with B as parameter.
The implementation and the test are tightly coupled, which is bad.

The mistake here is that I don’t set up the system properly. If I want to test that isValid returns false when A doesn’t exist I should ensure that B exists because I don’t want a false positive test passing because B is considered unexistant too.

Let me introduce two more methods in the test class in order to make tests easier to read :

<?php
// TupleValidatorTest
public function elementExists($element)
{
    $this->elementRepository->shouldReceive('exists')->with($element)->andReturn(true);
}

public function elementDoesntExist($element)
{
    $this->elementRepository->shouldReceive('exists')->with($element)->andReturn(false);
}
?>

We can now write the tests with the correct set up :

<?php
// TupleValidatorTest
/**
 * @test
 */
public function should_return_false_if_A_doesnt_exist()
{
    $a = 'A';
    $b = 'B';

    $tuple = new Tuple($a, $b);

    $validator = new TupleValidator($elementRepository);

    $this->elementDoesntExist($a);
    $this->elementExists($b);

    $this->assertFalse($validator->isValid($tuple));
}
?>

This test will pass whatever the order of the assertions in isValid is.

I think starting with the test for the positive result helps because you have to set up the whole system in order to make it pass and you have a template for all the negative results.

Here are all the correct tests for the TupleValidator :

<?php
// TupleValidatorTest
/**
 * @test
 */
public function should_return_true_if_A_and_B_exist()
{
    $a = 'A';
    $b = 'B';

    $tuple = new Tuple($a, $b);

    $validator = new TupleValidator($elementRepository);

    $this->elementExists($a);
    $this->elementExists($b);

    $this->assertTrue($validator->isValid($tuple));
}

/**
 * @test
 */
public function should_return_false_if_A_doesnt_exist()
{
    $a = 'A';
    $b = 'B';

    $tuple = new Tuple($a, $b);

    $validator = new TupleValidator($elementRepository);

    $this->elementDoesntExist($a);
    $this->elementExists($b);

    $this->assertFalse($validator->isValid($tuple));
}

/**
 * @test
 */
public function should_return_false_if_B_doesnt_exist()
{
    $a = 'A';
    $b = 'B';

    $tuple = new Tuple($a, $b);

    $validator = new TupleValidator($elementRepository);

    $this->elementExists($a);
    $this->elementDoesntExist($b);

    $this->assertFalse($validator->isValid($tuple));
}
?>

When writing a test be sur to set up the system in a state that allows you to test what you think you are testing.

This is a rather long post and even if I noticed something else during the refactoring I’ll stop now and keep this for a futur article.

Hey ! I’m on Twitter too, if you want to chat about testing or something else. Feel free to comment below as well.

Test for behaviour not for implementation

2015-08-30T00:00:00Z

While working on SearchCache I made a test which was ensuring that two shared search results coming from a same search - using the same search parameters - should be stored using the same key.

Here it is :

<?php
public function testStoringSharedResultWithSameParametersUsesTheSameKey()
    {
        $searchCache = new SearchCache($this->searchResultStore, $this->keyGenerator);

        $params = [
            'name' => 'test',
            'age'  => 12,
        ];

        $result1 = [1, 'AA', 3, "HUG76767"];
        $result2 = [1, 'AA', 3, "AAA"];

        $this->shouldStoreSharedResult($result1, $key1);

        $searchCache->storeSharedResult($params, $result1);

        $this->shouldStoreSharedResult($result2, $key2);

        $searchCache->storeSharedResult($params, $result2);

        $this->assertEquals($key1, $key2);
    }
?>

The method shouldStoreSharedResult allows me to get the key used when the result is stored so I can later compare them for equality.

This was not feeling right to me. It bothered me so much that I removed the test before adding it back. I had the uncomfortable impression that the test was knowing way to much thing about the system.

As you can see the test is tightly coupled to the implementation and gives us some information about it. For instance we know that the system uses keys.

Further thinking lead me to ask myself what I wanted to achieve with the system : I want the searchCache to replace an old version of a shared search cache with a newer one when possible.

I made some refactoring on my tests in order to have tests reflecting what the system should do instead of how it does its job. Furthemore I decided that I could get rid of my mocks created with Mockery and use a stub instead. The test now looks like this and doesn’t leak any details about the implementation :

<?php
public function testIfAPreviousVersionOfSharedResultIsStoredItsReplacedWhenANewOneIsStored()
    {
        $params = [
            'name' => 'test',
            'age'  => 12,
        ];

        $result1 = [1, 'AA', 3, "HUG76767"];
        $result2 = [1, 'AA', 3, "Oho"];

        $this->storeContainsSharedResult($params, $result1);

        $this->searchCache->storeSharedResult($params, $result2);

        $key = $this->searchCache->getCopyOfSharedResult($params);

        $this->assertEquals($result2, $this->searchCache->getResult($key));
    }
?>

I have the impression that using a mocking library pushed me on the coupling path and I’ll try to keep an eye on this in order to avoid making that same mistake again since I have already been proven guilty of this several times. Using mocks adds up a lot to the risk of creating false positive tests - see this article - and having a test suite that give you a false impression of confidence is probably worst than not having tests...

Previous article was linked to a blog post where Fabio Pereira discussed the issue of writing a test which is a mirror of the implementation in his Tautological Test Driven Development article. This is exactly what I was doing, Tautological TDD.

I still have to rewrite some of the tests to decouple them from the implementation but I’m pretty confident that writing them with behaviour in mind and without a mocking library will help make them cleaner and simpler.

I guess this episode taught me a valuable lesson. Cool.

Hey ! I’m on Twitter too, if you want to chat about testing or something else. Feel free to comment below as well.

OOP Layers

2015-09-16T00:00:00Z

Yesterday at work I ran into a pretty nasty bug and while looking through the code I spotted something which even if it wasn't the primary cause of all the mess made it really worst.

The code was going like this :

<?php
class AClassThatDoesSomethingWithThingsAndStuff {

    public function aMethodThatAssociatesSeveralThingsWithAStuff(array $things)
    {
        $aStuff = null;

        foreach($things as $thing)
        {
            $aStuff = $this->aMethodThatUseASomething($thing, $aStuf);
        }
    }

    private function aMethodThatAssociateOneThingWithAStuff(Thing $thing, Stuff $stuff = null)
    {
        if($stuff == null) {
            $stuff = $this->somehowGetAStuff();
        }

        $thing->associateWithASomething($stuff);

        return $stuff;
    }
}
?>

As you can see it seems that we always need a Stuff in order to make an association with a Thing. Furthemore we hope to use that same Stuff for all the things.

The main issue with this piece of code is the Stuff we use is going through two layers: the Stiff is created in a sub-layer (in aMethodThatAssociateOneThingWithAStuff), going up one layer (in `aMethodThatAssociatesSeveralThingsWithAStuff) before returning to the sub-layer again.

An other issue is that we are not really in control of the Stuff used.

I think the code would have been better if it had been written this way :

<?php

class AClassThatDoesSomethingWithThingsAndStuff {

    public function aMethodThatAssociatesSeveralThingsWithAStuff(array $things)
    {
        $stuff = $this->somehowGetAStuff();

        foreach($things as $thing)
        {
            $this->aMethodThatUseASomething($thing, $aStuf);
        }
    }

    private function aMethodThatAssociateOneThingWithAStuff(Thing $thing, Stuff $stuff)
    {
        $thing->associateWithASomething($stuff);

    }
}
?>

This code doesn't break the layers between the methods. The Stuff is get on the top level and passed one level down when needed. We are also able to choose what Stuff we want to pass to the sub-level method.

And it's simpler...

We can even make this way more fun if aMethodThatAssociateOneThingWithAStuff can throw exceptions.
The code will probably ended up with a try-catch structure inside of the foreach loop. (Our code end up like this at least...)

See below :

<?php
class AClassThatDoesSomethingWithThingsAndStuff {

    public function aMethodThatAssociatesSeveralThingsWithAStuff(array $things)
    {
        $aStuff = null;

        foreach($things as $thing)
        {
            try {
               $aStuff = $this->aMethodThatUseASomething($thing, $aStuf);
            }
            catch(Exception $exception) {
                // Do what you want here to treat the exception
            }
        }
    }

    private function aMethodThatAssociateOneThingWithAStuff(Thing $thing, Stuff $stuff = null)
    {
        if($stuff == null) {
            $stuff = $this->somehowGetAStuff();
        }

        $thing->associateWithASomething($stuff);

        // Do something else, possibly throwing an exception

        return $stuff;
    }
}

?>

Imagine the method somehowGetAStuff has some side effects: writing a new Stuff in the database for instance.

When an exception is thrown the execution enters the catch part of the try-catch and doesn't overwrite the value of $aStuff.
As long as the method will throw an exception without letting a chance to $aStuff to be initialized somehowGetAStuff` will be called, with all its side effects...

Breaking layers in OOP can have some really bad consequences and , as you can see, it makes the code harder to read and will probably create some bugs.

This example is based on methods but classes are layers too and thus should not become entangled neither.

As a rule of thumb you should probably be suspicious at a piece of code when you see:

A test for existence for something passed to your method.
A memory allocation without desallocation.
A layer in charge of retrieving something it needs. This is breaking the inversion of dependency principle.
A method called openSomething without a closeSomething, or the opposite, in the same layer.

because this code is probably running through several layers.
I guess we can make this list way longer.

The header picture is a photography of the Quebrada de Humahuaca in Argentina. The strange look of the mountain, called Cerro de los Siete Colores - The Hill of Seven Colors -, is the result of thousand years of marine sediment. See how beautiful layers can be ?

I took the picture from this website, and I really don't know who I should thanks for it and attribute it to. I'm not sure they do neither...

Hey! I'm on Twitter, if you want to discuss about OOP feel free to come by and say hi! You can comment below as well.

Travis and different PHP versions in the same repo

2015-11-22T00:00:00Z

At work we have one git repository for several projects. While this has some advantages their is also some drawbacks. Setting up a CI is one of them, especially when all the applications can't run on the same PHP version.

We use Travis in order to ensure that our tests are still green after a modification but as we are currently working on some refactoring of our legacy app written for PHP 5.3 - We are late, we know... - we wanted to be able to work on a more recent PHP version. The issue is that Travis' build matrix will fail either on the PHP 5.3 build or on the PHP 5.6 one.

We finally came up with a solution, which even if not ideal allows us to see if we are breaking something : we run some of the tests only for one version or the other.

First thing we create a variable for each version of PHP in the before_script section of travis.yml:

before_install:
    - if [[ ${TRAVIS_PHP_VERSION:0:3} == "5.3" ]]; then export PHP53=false; else export PHP53=true; fi
    - if [[ ${TRAVIS_PHP_VERSION:0:3} == "5.6" ]]; then export PHP56=false; else export PHP56=true; fi

Notice that we are assigning false to the associated variable when the PHP version matches.

Now, for each command that Travis should execute we are able to specify if it should be run for the current build version :

install:
    - $PHP53 || ( cd $TRAVIS_BUILD_DIR/applications/app1 && composer install )

This must be read as "Cd to app1 directory and run composer install for PHP 5.3".

In the end our travis.tml looks like this:

language: php

php:
  - 5.3
  - 5.6

before_install:
    - if [[ ${TRAVIS_PHP_VERSION:0:3} == "5.3" ]]; then export PHP53=false; else export PHP53=true; fi
    - if [[ ${TRAVIS_PHP_VERSION:0:3} == "5.6" ]]; then export PHP56=false; else export PHP56=true; fi

install:
    - $PHP53 || ( cd $TRAVIS_BUILD_DIR/applications/app1 && composer install )
    - $PHP56 || ( cd $TRAVIS_BUILD_DIR/applications/app2 && composer install )

script:
  - $PHP53 || ( cd $TRAVIS_BUILD_DIR/applications/app1 && ./bin/phpunit )
  - $PHP56 || ( cd $TRAVIS_BUILD_DIR/applications/app2 && ./vendor/bin/phpspec run && ./vendor/bin/behat )

If you want a command to be run for several PHP versions you can specify them as follow:

install:
    - ( $PHP53 && $PHP56 ) || ( cd $TRAVIS_BUILD_DIR/applications/app1 && composer install )

which should be read "Cd to app1 directory and run composer install for PHP 5.3 and PHP 5.6".

Handy, right ?

Hey ! I'm on Twitter too. Come say Hi ! You can feel free to comment below as well.

Removing ifs

2016-05-08T00:00:00Z

I'm not a big fan of if statements.

They make source code harder to reason about, force us to write more test cases and increase cyclomatic complexity...

We even debated with some colleagues on the way to lunch if it would be possible to make an application without any one of them. We finally agreed that it would be impossible.

However in some cases it's totally doable to remove if statements. Here are a few options.

Keeping them outside of the application

The application I work on has to be integrated with some external services. Some of them ping our system on an a callback URL with an HTTP call. This is a fairly common scenario.

I had to work on the controller action dealing with those callback calls. The controller action. Yep. We have only one action for several externals services. Therefore the major part of the code is dealing with trying to recognize who is calling, base on the shape of the message. Coming with a lot of ifs.

A simpler solution would have been to create one action for each external service, pushing the if statements outside of our code directly into the external services' configuration.

Depency inversion principle

Sometimes we have to make use of slightly different version of an algorithm, whether its for test purpose or for some other reason.

I've stumble upon a piece of code that was using a boolean flag specifying if code was run in test mode or not. A trivial example could be something like this :

<?php
class Writter {

    public function write($text, $test = false)
    {

		if($test) {
			echo "Test : " . $text;
		}
		else {
			echo $text;
		}
	}
}

class TypeMachine {

	private $writter;

	private $isInTestMode;

	public function __construct(Writter $writter, $isInTestMode)
	{
		$this->writter = $writter;
		$this->isInTestMode = $isInTestMode;
	}

	public function type($text)
	{
		$this->writter($text, $this->isInTestMode);
	}
}

This code has some flaws.

The TypeMachine has to implement some sort of logic in order to choose the value of the second parameter and this is not it's responsabilty to deal with that. This creates coupling between TypeMachine and Writter.

A better solution would be to inject a different Writter when running in test mode :

<?php
interface Writter {

	public function write($text);
}

class SimpleWritter {

    public function write($text)
    {
		echo $text;
    }
}

class TestWritter {

	public function write($text)
	{
		echo "Test : " . $text;
	}
}

class TypeMachine {

	private $writter;

	public function __construct(Writter $writter)
	{
		$this->writter = $writter;
	}

	public function type($text)
	{
		$this->writter($text);
	}
}

We have added a Writterinterface and two implementations, SimpleWritterand TestWritter. We can now choose which one we want to use regarding of the context.
As you can see the code of the TypeMachine class is simpler too and all classes are easily testable.

As an aside we should be carefull when we start using flag parameters in method prototype because it usually indicates some sort of missing concept. A cool article was written on 8th Light blog on how to remove this sort of parameters.

Null object pattern

An other common use of if conditionals is to determine if an optional colaborator was provided before performing an operation on it.

One of the best example is the use of a logger :

<?php
interface Logger {
	public function log($logMessage);
}

class TypeMachine {

	private $logger;

	public function setLogger(Logger $logger)
	{
		$this->logger = $logger;
	}

	public function type($text)
	{
		if($this->logger) {
			$this->logger($text . 'was typed.');
		}

		// do something
	}

}

Every time we want to log something we have to check whether the logger was set or not.

Using the Null object pattern, which consists in creating an implementation of an interface with no behaviour, we can remove this checks.

<?php

interface Logger {
	public function log($logMessage);
}

class NullLogger {
	public function log($logMessage){
	// do nothing
	}
}

class TypeMachine {

	private $logger;

	public function __construct()
	{
		$this->logger = new NullLogger;
	}

	public function setLogger(Logger $logger)
	{
		$this->logger = $logger;
	}

	public function type($text)
	{
		$this->logger($text . 'was typed.');

		// do something
	}

}

When the TypeMachine is instanciated its logger property is set to an instance of NullLogger allowing us to remove each if($this->logger) checks.

If you want to learn more about the Null object pattern you definitely should look at Sandy Metz's Nothing is something talk.

I hope this article will help removing some if statements in this world making it a better place. If you have some other ways to remove if statements please share them !

Hey! I'm on Twitter, if you want to discuss about OOP feel free to come by and say hi! You can comment below as well.

Write code for the dumbest person you know

2016-05-27T00:00:00Z

I wrote this text a few months ago while I was angry and decided not to publish it right away, letting it sink in. I kept it, made a few changes and finally decided to publish it.

I'm dumb

You're lucky. You're lucky because you're clever and I'm not.

I noted that I was dumb when I was unable to understand how some data was injected into an object during its instantiation.
I know what a depency injection manager is but that time was not like the others. It took me one hour to figure where that configuration parameter was coming from. I had to go through 8 code files and 3 configuration files and I had to make several full text searches in order to jump from file to file because following the code was misleading. This exercise was really hard to me because I'm barely able to remember my own phone number so I had to write down to paper your way to do DI because I couldn't keep it in memory.

I have to say that I started to be pissed off - allright, this is not uncommon - because your code was telling me that I was stupid, and no one likes beeing told he is stupid.

Someone then came by and told me it was the tool, a cool feature provided by the tool, that you were using. Again, I know what dependency injection manager is, and how it works, but that one magical feature which allows you to make the same thing as usual but in a very clever way, I don't know it. Probably because I had never needed it. I guess appart from beeing dumb I also lack of culture about that tool features.

I was sure that I was dumb when I couldn't understand what you're abstraction was for. For me all the interesting thing your code was doing was concentrated in one class. That one class I found after going through several classes doing I don't really know what. That one class which was wrapping a class from a library.

I probably missed something but I have to say that I started to be pissed off again - I think there is some kind of pattern here. I tried to understand, because, after all, you're clever. Why would you do something that complicated when just using the class from the library, or your wrapper, would have done the job ? Maybe you were protecting us from the coupling to that library and making its use simpler. I have to confess that I was surprised when I saw a massive chain of getters going all the way down to get the library object you were wrapping and pass it to the some other object.

Those are some of the times I noticed I was dumb. Hard truth...

Thank you for beeing an eye opener.

I used to be clever though

I used to be clever though. I used to write some pretty pieces of code doing all kind of cool stuff. I was theorically able to change the way the whole system was working by changing one parameter in a configuraion file. My code was able to cope with all future issues I had foreseen. I knew how to do some tricky things with all sort of tools.

But time goes on and aging probably made me dumb.

Aging or discoveries and failures.

Abstractions are good when you know what you're abstracting. Apparently you only know what your abstracting based on the information you currently have. I couldn't predict future and the next needs. I tried though, I failed, and ended up locked down by my clever abstraction. And as we all do I started to cheat, broke my abstraction, used getters in order to get the internals and be able to manipulate them.

Not once but several times.

I made complicated stuff too. They looked hard to understand at first glance but once you managed to wrap your head around the problem being solved, everyone could see how legant the solution was. At least, I was. I wrote it after all. Then I moved on, worked on something else and had to came by to it later probably because of something to fix. It seems like complicated code tend to be buggy. Worst, apparently time had increased code complexity. I was now unable to understand easily what was going on with this code and all the choices I made a few days before.

I have seen people reluctant to use some of my code. In my opinion it was not that hard to understand but they probably needed someone to guide them the first time they used the library because it was magic in some kind of way. We had a solution, we probably should have written documentation. Or simpler code : something the junior of your team can grasp by himself.

I was pissed with myself on those occasions.

I guess some time you will be pissed with yourself too. I hope so. It means your code is used, and that's cool.

Please stop

Please stop writing complicated stuff. It's harder to read, to understand, to test, and even to produce. Moreover it's unnecessary, it doesn't protect you from future requirements, it's hard and it costs time and money.

Please stop doing something everyone knows about in some other clever way. It's cool that you know everything about your tools, but what will be cooler is that you know when to use or not a feature. I don't want to be suprised while reading code.

I want to read code as I read books. Simple ones. Like Harry Potter. Translated in french. As I said, I'm dumb and I'm lazy...

Take a break, think two minutes about the piece of code you're about to craft : can you make it simple ? Can the dumbest person you know understand it ? Will he be surprised ?

As you've been an eye opener to me I hope these words will be an eye opener to you too.

Go write some code. Simple one.

Thanks a lot to Remi and Emilien for reviewing this post.

Whether you're clever or not we can discuss on Twitter. Feel free to comment below as well.

Stop using the database for everything

2016-06-07T00:00:00Z

First thing first, database are useful in a lot of cases and we need them. We need them when we have to store information provided by users or information changing on a regular basis.

This is not the case of all data an application needs in order to run - currencies, countries, languages, application settings, to name a few. This types of information don't change that often and adding a database for them doesn't provide much value. Maybe a developer won't be needed in order to change or add a value but I'm inclined to think that more than often no tool is provided to business people to change it by themselves, and a developer will have to run a query on the database nevertheless.

Adding a database in those cases is a cost with no value added. More code is needed, additional complexity is created, a database has to be maintained and the resulting system is probably slower - not in an important order of magnitude of course - than the same system without a database.

These additional costs are not the main reasons I dislike the use of database for such information.

In the long run special cases will probably be introduced in the application - do this for this country and that for that other one for instance. In order to face these special cases basically two solutions exist. Either you add the differentiating piece of data in the database, which might mean changing table schema and so on or you start cheating and use the reference - id, ISO standard reference - of the concept with a special case thus coupling code and database content altogether.

From what I've seen we, developers, will choose solution 2, maybe because we are lazy, in the middle of a rush or don't mesure the consequences of it.

Once the coupling is created it will eventually grow to the point where almost all data from the database is stored in the code or in a configuration file dedicated to mappings between code artifacts and database stored values. In the end a developper intervention will be necessary every time to update theses pieces of code. Back to square one.

If the data doesn't change much why not store it in the code after all ? Having only one golden source that is easily changeable - adding a property to a value object is way easier than changing a table schema - instead of two or more sources that will have to be maintained in cohesion.

When you're dealing with some data that doesn't look likely to change often try not to use a database. Start with a simple representation based on value objects if you are into POO. If in the long run that information appears to change a lot then use a database. Start with the simpler solution that works.

Once again, thanks a lot to Remi for his feedbacks.

If you are a DBA, hate me, and want to tell me that I'm an awful person I'm on Twitter. Feel free to comment below as well.

Unique instance anti-pattern

2016-06-10T00:00:00Z

I'm back from the past. I have been trying to introduce some tests in one of the oldest part of our codebase. I'm back from a time where using Service Locator was the norm for a lot of people, Inversion of Control concept wasn't widespread and Singleton wasn't presented as an anti-pattern yet.

I have to say that I feel funny pointing to some articles from Uncle Bob from 2004 while I'm speaking about a code that is less that 7 years old.

As one of my co-worker said last week :

It's funny to see that some people were 10 years ago where we are now. I'm wondering what they are doing now and where we will be in 10 years.

(F*** this quote is gigantic. Ok, I'm mixing message and form here...)

I do too.

So, here goes the masterpiece.

<?php
class AClassWithADecentName {

    static private $alreadyInstantiated = false;
	
    public function __construct()
    {
    	if (static::$alreadyInstantiated) {
           throw new Exception("You can't instantiate that class twice");
        }
        
        static::$alreadyInstantiated = true;
    }
}
?>

If you try to instantiate it twice your whole application will blow up because of the exception.

This is worst than a singleton : you can't instanciate an object twice and you can't use the first created instance in an other scope than the one where it was created, unlike the singleton. This is the unique instance anti-pattern.

It turns out I got the chance to speak with the man who created that beast. Remember my co-worker from last week ?

We spoke about the context at the time he wrote that piece of code. The team was composed of unexperienced/stubborn developers and he was trying to enforce a design and he wanted to prevent the misuse of that class. To be fair a more appropriate name for the class could be AClassWithADecentNameDoingThingWeAreNotVeryProudOfInAWayWeDislike.

The thing is that class is untestable the way it is now and the unique reason is to prevent developers to do something. I strongly believe that explanations and code reviews are way more powerful to prevent bad code from being created in the long term and so I don't want to harm code design as a protection against other developers. They have access to the codebase and they can mess with it anyway.

Protection code passes a message - do not use me - that should have been transmitted in some other form - teaching -.

You know : Give a man a fish... Teach a man to fish...

In order to prevent some other beasts from being created let's make a deal : We will not damage our code to prevent other developers to do something and we will explain why something should be done or not instead.

We will not conflate the message and the form anymore.

If you love bad images, bad jokes and anti-pattern I'm onTwitter. Feel free to comment below as well.

Extra readable tests

2016-08-10T00:00:00Z

I’ve been doing some experiments with tests for the last few weeks. In this article I’ll share some tricks I’ve learned from others, mainly Sandro Mancuso, Matthias Verraes, and, of course, from Growing Oriented Object Software Guided By Tests - the GOOS - by Nat Pryce and Steve Freeman.

In this article I will refactor the following test one step at a time in order to make it even more readable than it is right now.
The test framework I’m using here is PhpUnit but the following technics work with other frameworks - I’ve applied some of them with PhpSpec. By the way some technics are heavily inspired by this great tool.

<?php
class SendWelcomeEmailToMemberTest extends PHPUnit_Framework_TestCase {
    protected function tearDown() {
        Mockery::close();
    }

    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        $mailSender = Mockery::mock(MailSender::class);

        $welcomeMailSender = new WelcomeMailSender($mailSender);

        $mailSender->shouldReceive('send')
           ->with(Mockery::any(new Email(
           'charles@test.fr',
           'us@chorip.am',
           'Welcome',
           'Hey! Welcome!')
           ))
           ->once();

        $welcomeMailSender->sendTo(
            new Member('Charles', 'charles@test.fr')
        );
    }
}

Snake-case test name

As you can see I don’t use the classical PhpUnit test naming form, testWhatItSupposedToDo, but prefer to use snake-case notation and to start test name with it_ - which forces me to use the @test annotation. Starting the test name with it_ helps me to think harder in order to come up with a good name for the test.

Having good test name open an interesting possibility: being able to generate a documentation readable by all members of the project using PhpUnit’s testdox option.

Arrange - Act - Assert

First change is to reorganize the test to be in the form of Arrange - Act - Assert. Having all tests following this model helps knowing what the expectations are, as they all are at the end of the scenario. In the current version an assertion - an email should be sent - is made before the acting step.

The change here is to use a Spy instead of a Mock. You can read on the different types of doubles here.

<?php
    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        // Arrange    
        $mailSender = Mockery::spy(MailSender::class); // Use a spy instead of a mock

        $welcomeMailSender = new WelcomeMailSender($mailSender);

        // Act
        $welcomeMailSender->sendTo(
            new Member('Charles', 'charles@test.fr')
        );

        // Assert
        $mailSender->shouldHaveReceived('send')
            ->with(Mockery::any(new Email(
                'charles@test.fr',
                'us@chorip.am',
                'Welcome',
                'Hey! Welcome!')
                ))
            ->once();
    }

Builder

Let’s now imagine that the class Member is central to our application and is used in a lot of tests, all of them having to call the constructor in order to get Members.

A new requirement is made : A member should have a birth date. We must add an extra parameter to all calls to Member constructor to fix all the failing tests. Trust me, the day you’ll face this situation you probably be very disappointed.

A solution is to introduce a builder which will encapsulate the call to Member constructor. We are now able to do the change in only one place.

Using a builder for member and email gives us the following test :

<?php

    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        $mailSender = Mockery::spy(MailSender::class);

        $welcomeMailSender = new WelcomeMailSender($mailSender);

        $charles = (new MemberBuilder)
            ->withFirstName('Charles')
            ->withEmailAddress('charles@test.fr')
            ->build();

        $welcomeMailSender->sendTo($charles);

        $welcomeEmail = (new EmailBuilder)
            ->from('us@chorip.am')
            ->to('charles@test.fr')
            ->withSubject('Welcome')
            ->withContent('Hey! Welcome!')
            ->build();

        $mailSender->shouldHaveReceived('send')->with(Mockery::any($welcomeEmail))->once();
    }

The code for MemberBuilder is the following :

<?php
final class MemberBuilder {
    private $firstname;
    private $emailAddress;
    private $birthDate;

    /**
     * MemberBuilder constructor.
     */
    public function __construct() {
        $faker = \Faker\Factory::create();
        $this->emailAddress = $faker->email;
        $this->firstname = $faker->firstname;
        $this->birthDate = $faker->date;
    }

    public function build() {
        return new Member($this->firstname, $this->emailAddress, $this->birthDate);
    }

    /**
     * @param $firstname
     * @return MemberBuilder
     */
    public function withFirstName($firstname) {
        $this->firstname = $firstname;

        return $this;
    }

    /**
     * @param $emailAddress
     * @return MemberBuilder
     */
    public function withEmailAddress($emailAddress) {
        $this->emailAddress = $emailAddress;

        return $this;
    }

    /**
     * @param $birthDate
     * @return MemberBuilder
     */
    public function bornOn($birthDate) {
        $this->birthDate = $birthDate;

        return $this;
    }
}

As you can see the builder constructor provides a default value for each property of Member. This is really convenient because it allows to provide only the properties that matters for the test we are writing. If I need a Member and only his firstname matters I can call (new MemberBuilder)->withFirstName('John')->build(). This way we can reduce the noise in the test, displaying only the relevant bits of information, making the test easier to understand.

Moreover I’m using Faker, a library helping to create fake data for a lot of types. Using random data ensures tests are not making any assumptions based on a fixed value somewhere.

Builder functions

If we want to make the test even easier to read we can hide the call to the builders behind a function. If we introduce the two following functions :

<?php

function aMember() {
    return new MemberBuilder;
}

function anEmail() {
    return new EmailBuilder;
}

We can rewrite our test like this :

<?php
    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        $mailSender = Mockery::spy(MailSender::class);

        $welcomeMailSender = new WelcomeMailSender($mailSender);

        $charles = aMember() // Creates a member
            ->withFirstName('Charles')
            ->withEmailAddress('charles@test.fr')
            ->build();

        $welcomeMailSender->sendTo($charles);

        $welcomeEmail = anEmail() // Creates an email
            ->from('us@chorip.am')
            ->to('charles@test.fr')
            ->withSubject('Welcome')
            ->withContent('Hey! Welcome!')
            ->build();

        $mailSender->shouldHaveReceived('send')
            ->with(Mockery::any($welcomeEmail))->once();
    }

Helper methods

If we want to write our tests using the business vocabulary we can introduce helper methods as following :

<?php
class SendWelcomeEmailToMemberTest extends PHPUnit_Framework_TestCase {
    private $mailSender;
    private $welcomeMailSender;

    public function setUp() {
        $this->mailSender = Mockery::spy(MailSender::class);
        $this->welcomeMailSender = new WelcomeMailSender($this->mailSender);
    }

    protected function tearDown() {
        Mockery::close();
    }

    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        $charles = $this->it_exists(aMember()
            ->withFirstName('Charles')
            ->withEmailAddress('charles@test.fr')
        );

        $this->welcomeMailSender->sendTo($charles);

        $this->it_should_send(anEmail()
            ->from('us@chorip.am')
            ->to('charles@test.fr')
            ->withSubject('Welcome')
            ->withContent('Hey! Welcome!')
        );
    }

    private function it_exists(MemberBuilder $member) {
        return $member->build();
    }

    private function it_should_send(EmailBuilder $email) {
        $this->mailSender->shouldHaveReceived('send')
            ->with(Mockery::any($email->build()))->once();
    }
}

As you can see the test reads easily and business people would be able to understand what is going on reading it. The test can now serve as documentation both for developers and business people.

Going further

Create plug-in for the test framework

The test could be even clearer if we could replace $this before in the call to helper methods by given() and then(). I made the following code work using a class with a static property but I think we could do something better with a plug-in for the test framework.

<?php

    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        $charles = given()->it_exists(aMember()
            ->withFirstName('Charles')
            ->withEmailAddress('charles@test.fr')
        );

        $this->welcomeMailSender->sendTo($charles);

        then()->it_should_send(anEmail()
            ->from('us@chorip.am')
            ->to('charles@test.fr')
            ->withSubject('Welcome')
            ->withContent('Hey! Welcome!')
        );
    }

Generate documentation

I think the tests are clear enough but if the business people are really reluctant to look at code or to deal with versioning systems we probably can parse the test class and generate a documentation in a better format for them.

All these technics can be used one at a time as they solve different problems but when combined they give a really good result.

As always using them as a cost which should be balanced regarding the interest of the project you are working on.

I hope these tricks will help you write tests you’re happy with in the long run.

And if you need some help with your test I think I can help. Have a look at my video course in French or let's have a chat and see what we can do together.

It should not throw an exception

2016-08-27T00:00:00Z

I’m very interested in living documentation because it provides some great values. First, you finally have documentation, which is up to date and under version control. The other great thing about it is that it forces to look at the code through another perspective : through the eyes of a non developer person.

A few weeks ago I experimented and extracted documentation from code and tests. Tests were used to describe an object properties, its business rules. Some test method names were feeling a bit to technical to me and were not making sense to business people.

In the case of a DatePeriod object two tests names were odd :

it is initializable
it throws an exception if end date precedes start date

The first one is created automatically by PhpSpec in order to ensure the DatePeriod class is created. This test case should be removed as soon as another test case is created as it doesn’t provide value anymore.

The second one is more interesting. It doesn’t describe what the DatePeriod does but how it does it. This is a form of coupling between the test case and the implementation. As seen before coupling between the two of them is not a good idea because it makes harder for coming up with another implementation. In that case throwing an exception is probably the best implementation though.

What could be a better name then? We’re looking for something that speaks to business people. "It doesn’t allow end date to precede start date" does the job here.

I think looking at the code through the prism of documentation really helps making it better. I’m always amazed to see how business understanding, modeling, testing and documentation really fit together - 4 sides of the same coin (Sorry).

Hey! I’m on Twitter I only throw some bad jokes but no exception. Feel free to comment below as well

Ask about the freelancer

2016-11-26T00:00:00Z

A few days ago I was involved in a discussion on twitter about a modeling problem. As a support for the discussion we were talking about a time tracking system and the concept of time sheet. I proposed two ways I would model the requirements. I indicated that I would choose one solution over the other if the concept of time sheet was really relevant for tracking, and that I couldn't find an example where it would be the case.

A few hours later I thought that a freelancer would need to have more than one time sheet opened at the same time if she worked for several clients.

At work we also have a concept of freelancers that wasn't really taken into account during the creation of the model which forces us to do some workarounds in order to deal with them.

This two things in mind it stroked me that freelancers make for great personas when you're speaking with business people and trying to find out the properties of a system.

Freelancers have two really interesting properties:

They can work for several companies/groups at the same time.
It's easy to think they'll leave.

This two properties open for really insightful discussions:

What should happen when a user leaves the system?
Is something owned by the user or the group she's belonging to?

I have the impression that the leaving property all by itself helps discovering corner cases in the domain and prevents making modeling mistakes.

It's never just someone who stops using the system.

For instance we probably have to transfer something linked to a leaving person to someone else in the same group. Does it means that this thing is owned by the group and is taken care of by a member of the group at a given moment? Therefore should we model explicitly the relation between that thing and the group (the thing is owned by the group and taken care of by that person) or is the relation between the thing and the person enough (the thing is owned by the group because the thing is taken care of by the person who belongs to the group)?

Asking about what happen in the end can help understand what are the true relationships between domain concepts.

I guess that this observation can be generalized to all model objects, not only to person.

Asking for what happen to something reaching an end-of-life state leads to better insights as you don't know the full history until you know the end of it.

Hey! I'm on Twitter and if you don't want to consider this as a finished thing I'll be happy to talk with you about it there! You can comment below as well.

What if someday...

2017-04-15T00:00:00Z

"What if someday..."

These three words are probably one of the main causes of accidental complexity and late delivery in the software industry. As software developers we always try to guess what the future will be, to anticipate future needs, in order to be prepared. The issue is that it's something we suck at.

We try to predict what will be the next features needed and doing so we add a lot of complexity to the software we build and maintain.

Nothing comes for free. Adding that new frontend framework, that queuing system or that cool design pattern you just learned about to the project has a cost and the team will have to support it in the long run. Building it this way will probably be longer than a simpler solution. You'll need to have someone around able to understand it and to maintain it. You might have to adapt your tooling for deployment, testing and logging. You may see your build time increase. You can expect to deal with some new failures you weren't aware of. This are a few examples but I'm sure we can find some others.

So, here a few other questions, as valid as the previous one:

What if the day we have to scale for a large number of users never comes?

What if by the day we actually have to offer offline service we can use some other tool better suited for that need?

What if we've imagined all the possible future requirements and made a system that can deal with them all except one That one we have to implement after all, that one we can't introduce because we've been too clever and have created a system locked by its abstractions.

Sounds familiar?

Deciding to implement something based on assumption is basically taking a bet, which means there is probabilty to lose something, probably time and/or money. Our job as software developper is to reduce the risk of building something wrong or useless and to minimize the loss in case we built it wrong after all.

Using a simple solution, or even a boring one, will allow to deploy the feature earlier than trying to build a full system able to cope with everything.
This means we'll deliver value earlier, possibly making money earlier. This also means we'll have user feedback earlier and we'll be able to take advantage of the feedback loop to test our assumptions and see if we're building the right thing. This is where the heart of agile is.
We've also saved some time we can use to build something else on another part of the system or building a prototype of the more complexe solution and learn about it.

Striving for simplicity is a good way to maximise the value of our system.

Next time you have to choose whether you should introduce a new tool to your stack please take the time to think and reflect on all the tradeoffs coming along.

Hey! I'm ranting on Twitter too and you can come and tell me I'm wrong over there if you want to! You can comment below as well.

Naming collision

2017-08-28T00:00:00Z

The context

I don't know if I already shared that I work at Evaneos, a travel market place. Our job is to interconnect future travelers with travel agencies all over the world so they can benefit a trip tailor made by someone on site with a good knowledge of the destination. The agency's first job is to respond to a potential traveler request with a travel proposition after discussing the traveler's wishes.

Lately, I've been working on a rewrite of our routing system which for a given request selects the appropriate agency.

A request can be in one of these 3 forms - here I keep the common vocabulary shared in the company:
From itinerary: The traveler selects an example itinerary she is interested in. We route the request to the agency which created that itinerary.
From agency: The traveler decides to talk directly with an agency. I bet you guessed which agency will receive the request.
From scratch: The traveler indicates which destination she would like to travel to and we match her with the agency which can provide what we believe to be the best travel proposition according to the few information we have on what she wants to do.

The collision

During the rewrite we decided to provide some key insights to the people dealing with the settings of the from scratch algorithm.

One of this insights is the sales rate for the requests made from scratch.

During a meeting with business people when we were presenting that new tool someone noted that the sales rates shown were not the same as the one in the tool they were already using. As she pointed out, in the precedent tool what was considered from scratch were what the new tool was naming from scratch and from agency.

Boom.

We talked about it a few minutes but as this was not the point of the meeting we decided to postpone that discussion and to come back to it later.

The feeling

This left me with the feeling of something wrong going on and as I kept working on the project that discussion was coming to my mind from time to time.
It made no sense to me that what we were considering as from scratch could also include from agency request type.

It took me a while, some sleepless nights, a lot of showers, and a week of holidays to finally get it.

We were simply not talking about the same thing. At all.

The old tool shows the sales rate for a travel proposition - which is different thing than a request for a travel proposition - that isn't based on an preexisting itinerary, hence "from scratch".
It makes sense to take into account what we call from scratch and from agent in the routing system to compute the sales rate of propositions made from scratch.

On one side we are talking about requests and on the other about the resulting proposition.

The history behind

It's been only a few months since we started to separate the concept of request from the one of proposition. Before that, and still now because of old habits, we were referring to this concepts as "a quote". This is a good example of Developper/Database Driven Design with all the associates problems.

After all what is the difference between a request and a proposition if not a status changing from one integer to the other in a table row?

You can imagine the fun we had when we decided to track all propositions made for a given request. Yes, the first one isn't always the good one and there is some back and forth between the traveler and the agency.

I truly believe that sharing a word for this two different concepts is one of the main reasons behind the misunderstanding.

The future forward

I think the solution is pretty straightforward. First, we really need to focus on using the appropriate names, request or proposition, when talking to each other.

I think we should also, at least, rename "from scratch" to "from destination" in the routing context. This would prevent the confusion and is coherent with naming pattern used by the two others which represent the page on the website where the request was made.

I said "at least" because I'm not convinced we should tie the domain logic to the graphic interface and I believe we can come up with something better than a word such as meaningless as from is.

As a start a request is not made from a destination but by someone who want to go to that destination. "A traveler makes a request for a trip to Italy", "For"*1 isn't a great word either but the direction seems to be correct at least. This needs some more digging.

Note that using "from" when talking about the proposition seems legitimate, the action direction is ok. "A proposition is built/prepared/conceived from an itinerary", "a proposition is built/prepared/conceived from scratch". Because an action takes a verb I think we would be better with including it in our discussions and in the code. This would add meaning and reduce the risk of another collision*2.

The next step is to work on the naming with the business people, trying to find something that makes sense and provide meaning. This will probably require to use verbs alongside preposition but this is for the best and that will reduce the risk of future collisions.

*1 I hesitated with going with "to" instead of "for". None of them are really convincing. Looks like a good warning that they probably aren't the best suited for the job.

*2 "That proposition built from scratch and travel starts from Paris". Not the best example ever but you get my point.

Hey! I'm on Twitter and if you feel like playing the "Jeu des mots" with me I promise we'll have some fun! You can comment below as well.

Using data provider to express business rules in test

2017-09-25T00:00:00Z

When we write tests we want to achieve several things: ensure that the code works, create documentation, gain feedback about the ease of use of the system we are building and the quality of its design.

Creating documentation with tests goes through good test case naming. We want the name of our test cases to express the behavior of the system under test, or even better, the business rules of the domain.

As I was working around data providers with PhpUnit I noticed they offer a good opportunity to work toward that goal.

<?php
class DishTest extends PHPUnit_Framework_TestCase {

	public function test_is_vegan_if_empty(){
		$dish = new Dish([]);
		$this->assertTrue($dish->isVegan());
	}

	public function test_is_not_vegan_if_it_contains_meat()
	{
		$dish = new Dish(['meat']);
		$this->assertFalse($dish->isVegan());
	}

	public function test_is_not_vegan_if_it_contains_cheese()
	{
		$dish = new Dish(['cheese']);
		$this->assertFalse($dish->isVegan());
	}

	public function test_is_not_vegan_if_it_contains_egg()
	{
		$dish = new Dish(['egg']);
		$this->assertFalse($dish->isVegan());
	}
}

This test class fails at expressing the business rule. Every test case name is an example.

Using a data provider we can manage to reduce the number of test cases and to express the business rule in a clearer way.

<?php
class DishTest extends PHPUnit_Framework_TestCase {

	public function test_is_vegan_if_empty(){
		$dish = new Dish([]);
		$this->assertTrue($dish->isVegan());
	}

	/**
	 * @dataProvider nonVeganDishes
	 */
	public function test_is_not_vegan_if_it_contains_non_vegan_aliments($dish)
	{
		$this->assertFalse($dish->isVegan());
	}

	public function nonVeganDishes()
	{
		return [
			'A dish with meat' => [new Dish(['meat'])],
			'A dish with cheese' => [new Dish(['cheese'])],
			'A dish with egg' => [new Dish(['egg'])],
		];
	}
}

Through the introduction of the data provider we made a separation between the business rule, in the test case name, and the supporting examples, provided by the data provider.

We've partly achieved our goal, the two remaining test cases convey way more information about the business rule. Still having the same rule expressed in a positive and a negative way feels really strange.

We can keep the business rule expressed in the positive way if we modify the data provider in order to include the expected result.

First, because we want to keep our tests as readable as we can, we should introduce two constants

const IS_VEGAN = true;
const IS_NOT_VEGAN = false;

Then we can modify the tests cases and the data provider.

<?php
class DishTest extends PHPUnit_Framework_TestCase {

	const IS_VEGAN = true;
	const IS_NOT_VEGAN = false;

	/**
	 * @dataProvider dishes
	 */
	public function test_is_vegan_if_it_contains_only_vegan_aliment($dish, $isVegan)
	{
		$this->assertEquals($isVegan, $dish->isVegan());
	}

	public function dishes()
	{
		return [
			'An empty dish is vegan' => [new Dish([]), self::IS_VEGAN],
			'A dish with fruits is vegan' => [new Dish(['fruits']), self::IS_VEGAN],
			'A dish with meat is not vegan' => [new Dish(['meat']), self::IS_NOT_VEGAN],
			'A dish with cheese is not vegan' => [new Dish(['cheese']), self::IS_NOT_VEGAN],
			'A dish with egg is not vegan' => [new Dish(['egg']), self::IS_NOT_VEGAN],
		];
	}
}

Our test class now fully communicates the business rule as a domain expert would express it.

Listening to the test

Tests provide a good opportunity to reflect on the quality of the system design.

When we take a look at the test name and the associated examples we can see the mismatch between them. We are talking about non vegan food in the name but are using aliment names such as meat or cheese. This is a hint we might want the aliment to convey the information about whether it's vegan or not. We then would be able to simplify our test with factory methods for vegan or non vegan aliment.

Hey! I'm on Twitter wher I sometimes talks about testing too ! You can comment below as well if you feel like doing so !

Faster environment with xDebug and Docker

2020-05-05T00:00:00Z

I have to admit that I've always been lazy when it came to set up xDebug as it felt tedious the few times I've to do it. It made me an almost perpetual member of the var_dump debug team.

Nevertheless, the last time I had to set it up was more straightforward, thanks to PhpStorm, and I don't think the difficulty to set up is a valid excuse anymore.

I then had a debugger…

And I had a speed issue. My tests were running slower than before, and every request took way too much time. Using Docker on OSX didn't help at all.

I found a few tips that helped me go back to the same state as before.

So …

Tips 1: Do not start xDebug

Ahah… Yes, you can thank me for this one. The best way to avoid xDebug slowing everything down is by not having xDebug running. So, disable it by commenting the path to the extension in the .ini file loading it. Another idea is that if you have an xdebug.ini file in the PHP configuration directory, rename it to something else, xdebug.ini.back, for instance.

This being said, how can we manage to start xDebug when needed?

Tips 2: Use the on-demand PhpStorm mode when debugging tests

As explained in the PhpStorm documentation, an "on-demand" mode is available. It will enable xDebug only when debugging tests, which will allow running your test at full speed most of the time.

I'll let you dig in the documentation as it will be better explained than if I do it myself.

Tips 3: Use the on-demand mode without PhpStorm

PhpStorm is not doing magic for the "on-demand" mode but taking advantage of the -d PHP CLI option. The -d allows defining INI entry in the PHP configuration. When starting the "on-demand" mode, PhpStorm adds -d zend_extension=xdebug.so to the PHP CLI options, which enables the extension. You can do the same thing if you're running your tests without PhpStorm.

As an example here is how you can run PhpUnit with xDebug activated:

php -d zend_extension=xdebug.so ./vendor/bin/phpunit

You'll probably need to pass more options if you want to connect xDebug to a debugger client, so this not as convenient as running it through PhpStorm as it sets everything needed to work out the box. As an example, here are all the option passed by PhpStorm when I start tests in debug mode :

-d zend_extension=xdebug.so -d xdebug.remote_enable=1 \
-d xdebug.remote_mode=req -d xdebug.remote_port=9100 \
-d xdebug.remote_host=host.docker.internal

This is still a good option if you need xDebug without a client listening. I guess the primary use case would be for coverage reports.

Tips 4: Enable xDebug with an environment variable in your docker container

Going in the container and modifying files every time you need to enable or disable xDebug is cumbersome. I've tested another option for the last few weeks, and it works very well.

I've modified my PHP Dockerfile to add an entry point script, which is run when the container starts and can access environment variables.

The entry point script looks for an environment variable and decides to rename, or not, the xdebug.ini file in the PHP configuration directory.

As an example, here is the entrypoint.sh file I'm using in a php5.6 container. You'll probably need to adapt it to suit your container.

#!/bin/bash
set -e

if [[ ! -z "$DISABLE_XDEBUG" && "$DISABLE_XDEBUG" = true && -f "/etc/php/5.6/mods-available/xdebug.ini" ]]; then

mv /etc/php/5.6/mods-available/xdebug.ini /etc/php/5.6/mods-available/xdebug.ini.back

cat >&2 <<EOF
⚠️  xDebug is disabled
EOF

elif [[ -z "$DISABLE_XDEBUG" || "$DISABLE_XDEBUG" = false && -f "/etc/php/5.6/mods-available/xdebug.ini.back" ]]; then
mv /etc/php/5.6/mods-available/xdebug.ini.back /etc/php/5.6/mods-available/xdebug.ini
fi

exec "$@"
And the Dockefile is edited to include the entry point:

COPY ./entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
CMD ["php-fpm5.6"]

The container can be started either with docker-compose or docker run and xDebug activated by setting the environment variable DISABLE_XDEBUG to false and disabled by setting it to true. When the variable is missing, xDebug will stay activated.

We now have a convenient way to enable xDebug only when needed and save some precious time not waiting in front of loading pages. The only thing required is to change the value of the variable and restart the container.

Extra tip

xDebug settings can be overridden using the XDEBUG_CONFIG environment variable, which means there is no need to update the xDebug config file every time you have to change a setting or if you and your teammates need different settings.

The best example is probably the remote_host. According to how you run docker, this setting could be a changing IP address or host.docker.internal if you're using Docker for Mac.

In order to set the remote host to host.docker.internal the XDEBUG_CONFIG should be set remote_host=host.docker.internal.

I hope all these tips will help you improve the speed of your development environment.

If you want to share some tips as well come say “Hi!” on Twitter or feel free to comment below.

Being lazy with Postman scripts and dynamic variables

2020-09-28T00:00:00Z

I'm probably really late to the show as Postman is known to be a great tool when it comes to work with API. I've never took the time to dig in what this tool can do but I recently had a workflow involving generating and copy-pasting UUIDs from responses to requests and decided to see if this could be automated and it turns out it can, relatively easily even.

Problem

Pretend we are working with an API allowing us to create an inventory for goods stored in fridge. The API is simple and offers two actions :

Registering a fridge to be tracked is done via a PUT on /fridges/{fridgeId} where {fridgeId} is an UUID. The request body is not relevant for this example. A valid call will return a json response containing a field id with the provided UUID.
Tracking that a product is stored in the fridge is done via a PUT on /fridges/{fridgeId}/products where {fridgeId} is an UUID identifying an already registered fridge. The content of the request body and the response are not relevant for this example.

We would like to be able to generate an UUID when the registration route is called and avoid relying on another tool to get an UUID and then copy-past it in the URL input.

We'd also like to keep track of that UUID and be able to call the product stored tracking route with that newly created UUID without having to mess with input URL as well.

Generating a new UUID

Postman is able to deal with variables and even provides some dynamic variables which are replaced with some random values when the call is executed.
One of them is {{$guid}} and is replaced by an UUID.

We can change the input field to make us of {{$guid}}, the URL can be modified to include /fridges/{{$guid}}

Now everytime we will send the request a new UUID will be generated and used.

First problem solved!

Reusing the UUID for storing product

We are able to access the newly created UUID looking at the response of the registering call. This is the UUID we want to reuse for our call to store a product.

In order to automate the process and avoid copy-pasting the UUID from the response to the URL input field each and everytime we will take advantage of Postman scripting capability.

Postman allows to run scripts before and after every call. Just below the URL input, alongside requests parameters you'll notice two tabs, Pre-request Script and Tests.

The Tests tab can be used to execute tests after a request is made but can also be used to execute a script.

By adding pm.environment.set('fridgeId', pm.response.json().id) in the Tests tab input we instruct Postman to read the response, look for the id field in the JSON and store it in a fridgeId variable.

We are now able to use the fridgeId for any subsequent call. The URL used for storing a product can be changed to /fridges/{fridgeId}/products.

A new workflow

We've greatly simplified our workflow :

Everytime we want to register a new fridge we only need to visit the request tab for fridge registration and click send
We can store a product in the newly created fridge by clicking send on the other request tab as the request will use the fridge UUID directly.

No more UUID generation using external tool and no more copy pasting !

Passing data to session while testing in Elixir

2020-12-01T00:00:00Z

I struggled a while when I tried to pass data to the session while setting up a Liveview test. I googled a lot and asked on the elixir slack for a solution, but everything I saw seemed quite complicated, and I didn't manage to make it work.

I probably messed up something.

I finally found relief and a blog post with a one liner that does it :

Plug.Test.init_test_session(conn, user_id: 1)

The Plug documentation doesn't say much more.

Testing that a GenServer exited

2020-12-14T00:00:00Z

In the project I'm working on, I need to keep a GenServer up for a while, but I don't want to keep it up forever as I want to free resources.
The solution is to use the timeout mechanism with an handle_info(:timeout, _) callback stoping the loop.

This is straightforward, but then another question came up: How should I test that ?

After digging for a while I discovered that the solution is to monitor the GenServer process in the test. Once the GenServer process exits a :DOWN message should be sent to the test process. The test can assert that the correct message was received.

    test "MyGenServer exits after an_action", fixtures do

      # Start the GenServer
      {:ok, pid} = start_supervised(MyGenServer)

      # Monitor the GenServer process
      ref = Process.monitor(pid)

      # Start an action on the GenServer
      MyGenServer.an_action(pid)

      # Assert that the :DOWN message for the GenServer process was received
      # I decided to allow the message to arrive in a timeframe 10 times bigger than the allowed inactive period is 
      # If the message isn't received by then the test is marked as failed
      assert_receive {:DOWN, ^ref, :process, pid, :normal}, MyGenServer.allowed_inactive_period() * 10
    end

This solution applies to any type of Elixir process, not only to GenServers.

Building an event-sourced game with Phoenix Liveview: Introduction

2020-12-22T00:00:00Z

I’ve been playing with Elixir for a while now. I’ve created a Twitter bot, but I wanted to explore further what Elixir and Phoenix could help build for real-time collaboration between people. I was also curious about building a domain model with a functional programming language.

During the first French lockdown, I had the idea of a project that could be a good experiment: a game. As we had some fun playing Doctor Pilule, a French board game, I decided to build a digital version helping us to play with friends and family remotely.

Doctor Pilule

The game is quite simple. Players are split into several teams, and each round, one player tries to make her other team members guess as many words as possible. The trick is that all players are given two uncommon handicaps, one spoken (ex: start all sentences by "I might sound crazy but...") and one driving actions (ex: you think you are the Eiffel tower).
The first team to reach 20 guessed words wins.

Why this game?

First, as I said, it’s fun to play, and we needed some fun this year. It also comes with some interesting challenges to deal with, which are making for a good experiment:
it’s real-time
it needs to deal with a timer
all players don’t see the same thing at the same time (ex: the word to be guessed)
it’s better if you play it with video
game’s logic is complex enough to tinker with building a model with functional programming way without being overwhelming

And then?

I felt I could build something relatively quickly, helped by Phoenix Liveview and be able to play with friends after a few weeks.

Of course, it didn’t work out as expected.

As with many side projects, the fun part is not in delivering something but in learning. I spent a lot of time playing around, trying stuff, and being stuck and stubborn about solving problems not providing real value in the end—things we should probably avoid while at work; however, not so bad in a side project.

I still didn’t reach the video part and still have things I’d like to play with. Nevertheless, I managed to build the game logic, display relevant information to players in real-time, event-sourced the system, and learn a few things along the way. This blog post series is about some of these learnings.

Building an event-sourced game with Phoenix Liveview: Architecture

2020-12-23T00:00:00Z

In this article, we’ll cover the macro architecture of the DoctorP project.

The application is split into three parts: game logic, execution, and views.

Game logic

Game logic is pure business logic and is not concerned with the runtime properties of the system. Here seats the code expressing the game’s rules, written as much as possible using game terminology.

It’s written following the functional core imperative shell pattern, which means that all the code here is pure.

Game logic receives a command, the current game state, and everything else it might need to do its work.
The game rules are applied, returning the result without producing any side effects. In a non-event-sourced system, the result would be the new game state. Here, as we are doing event sourcing, the code returns a list of events.

Separating the game logic from the execution keeps our business logic free from runtime considerations. We can defer decisions on how we want our system to run to a later point. It also provides the ability to write fast-running unit tests without messing with processes.

Execution

In the execution part, we specify the runtime properties of the system.
I’ve decided that a GenServer, GameServer, will back each game room.

GameServer serves different purposes:
It manages all commands and queries related to one game room. Having one process for each game room improves reliability. A game room can’t be blocked by something taking time in another game room, and in case of a crash, only one game room is affected.
It stores all events produced by the actions taken in the game room. As this is a game and keeping all the data is not adding a lot of value, I’ve decided to store events in the process. One major downside to this solution is that we cannot get back to where the game was in case of a crash or server restart. Everything is lost.

GameServer are supervised using a dynamic supervisor. Because multiple GameServer can be up simultaneously, a registry keeps the relation between game id and process id using the "via tuple mechanism."

Views

Views use Phoenix Liveview, which provides the real-time capabilities needed for the game with server-rendered HTML. No need to write a single line javascript.

I’ve created a view for each phase of the game: waiting for players to arrive, playing the game, and displaying the result. An additional view seats on top of them and decides which one to show.
I’ve had difficulties trying to use LiveView components in place of the game phase views, but I guess an alternative implementation could use them.

All views use events produced by the game logic to decide what to show.
When a view is mounted, it fetches all past events from the GameServer, builds its own data structure of what to present, and renders. During the game events are published by the `GameServer’via the PubSub mechanism offered by Phoenix. Each view subscribes to the game channel and waits for new events to arrive. Once an event is received, the data structure is updated, and the view is modified accordingly.

Here is a sketch of the macro architecture of the project, summing up what was said so far.

We’ll cover some parts of this project more in-depth in other articles of this series. Let me know if you’d like me to cover some specific aspects.

Bibliography

To spawn, or not to spawn ? - Saša Jurić: Saša’s blog post is one of the main sources of inspiration for the architecture of this project.

Building an event-sourced game with Phoenix Liveview: An event sourced model

2021-01-13T00:00:00Z

This article on the series of building an event-sourced game with Elixir Phoenix and LiveView focuses on the game logic.

It would be too long and probably unclear for the reader to cover everything in one article, so I’ve decided to start from the beginning and rebuild everything from the start, omitting details, sometimes being a bit vague, and lying a bit about the final implementation.

In this first part about the game logic, we’ll see how I built some basis for an event-sourced model.

An event-sourced model

Before discussing the architecture of the game logic, we need to understand what event sourcing is.

As summarized by Martin Fowler, it means that we "Capture all changes to an application state as a sequence of events."

In an event-sourced system, when something occurs, an event, or a list of events, is produced. It differs from a more traditionally designed system where the new state is returned.

To get the current system state, we go through all past events and apply them one by one to rebuild the state incrementally.

We now have the basic understanding we’ll need to continue. Still, I invite you to read Martin’s article if you want to get all advantages and difficulties related to this pattern.

The game is event-sourced, which means that when something happens, events are produced and stored somewhere. Next time we want to act on the game, we’ll get all events and apply them one by one to rebuild the current state before applying the action.

Events

As we’ve seen, events are a big part of the design: they describe what happened during the game.

Events are designed as structures containing a map called data that stores all information about an event.

For instance, when a red card is dealt to a player, we need to know which card was given to whom.

To avoid duplicating the same code in each event module, I came up with a module defining a macro we can reuse:

defmodule DoctorP.Game.Events.Event do
  defmacro __using__(_opts) do
    quote do
      defstruct data: %{}

      alias __MODULE__

      def with(data) when is_list(data) do
        data = data
               |> Enum.into(%{})

        __MODULE__.with(data)
      end

      def with(data) when is_map(data) do
        %__MODULE__{data: data}
      end
    end
  end
end

This macro declares the structure with the data map and a with function.
The with function has two clauses and can be called with a map or a keyword list to build the event with the appropriate data.

Declaring an event becomes as easy as using the module:

defmodule DoctorP.Game.Events.DealtRedCardToPlayer, do:
  use DoctorP.Game.Events.Event

and were now able to create an event

DealtRedCardToPlayer.with(player: player, card: red_card)

Rebuilding the state

As explained in the first part of the article, the current state is rebuilt from all the past events.

I’ve created a GameState module that provides a build_state function. build_state takes a list of events, history, and rebuilds the state by calling apply_event function with the state and the event being applied. The apply_event function returns the state once the event is applied.

Here is a simplified version with the state stored in a map and coping with GameStarted and PlayerJoinedTeam events.

We’ll see in another article the code as it is, which is slightly more complex.

defmodule DoctorP.Game.States.GameState do

  def build_state(history) do
    
    defaultState = %{
      isStarted: true,
      players: []
    }

    List.foldl(history, defaultState, fn event, state ->
      apply_event(state, event)
    end)
  end

  def apply_event(state, %GameStarted{} = event) do
    %{state | isStarted: true}
  end

   def apply_event(state, %PlayerJoinedTeam{} = event) do
    %{state | players: [event.data.player | state.players]}
  end

end

We can now call build_state with a history :


[]
|> GameState.build_state() # %{isStarted: false, players: []}

[
  %GameStarted{},
]
|> GameState.build_state() # %{isStarted: true, players: []}

[
  %GameStarted{},
  PlayerJoinedTeam.with(player: "Charles"),
]
|> GameState.build_state() # %{isStarted: true, players: ["Charles"]}

[
  %GameStarted{},
  PlayerJoinedTeam.with(player: "Charles"),
  PlayerJoinedTeam.with(player: "Paul")
]
|> GameState.build_state() # %{isStarted: true, players: ["Paul", "Charles"]}

Producing events

We’ve just seen how we can rebuild the state from events, but where do events come?

Events are the result of actions taken on the game.

I’ve decided to use command objects to represent actions. Each command is an elixir structure containing the field necessary to describe the intention.

For instance, here is the module for the command allowing to add a player with a given name:

defmodule DoctorP.Game.Commands.AddPlayer do

  defstruct player_name: nil

end

The command is dealt with by a handle function with a clause for each event. Pattern matching does its job to find the appropriate one:

def handle(state, %AddPlayer{}) do
end

def handle(state, %MarkWordAsGuessed{}) do
end

Each clause body contains the game logic we want to apply based on the state and command passed as parameters. The result of the action is expressed as a list of events:

def handle(state, %AddPlayer{player_name: name}) do
  [
    PlayerJoinedTeam.with(player: name)
  ]
end

As we can see, we need to build the game state prior to handling a command. For convenience, we can add a handle_message function to our GameState module, which takes the list of past events and the command. This function’s job is to rebuild the state and apply the command.

def handle_message(history, command) do
  build_state(history)
  |> handle(command)
end

With this in place, we have a basis for expressing our game logic with events and commands. In the next articles, we’ll see how I evolved this to deal with errors, describe the game’s state more clearly, and manage the timer.

Building an event-sourced game with Phoenix Liveview: Handling errors

2021-01-20T00:00:00Z

In the previous article, we’ve set-up everything required for a basic event-sourced model.

We can:

record the decisions taken by the game logic in the form of events
rebuild the current game state from a list of past events
handle a command that triggers game logic

So far, every command we’ve talked about was accepted by the game logic, processed, and events were produced. But sometimes, things are not going as expected, and the command is rejected.

One example, we don’t accept several players with the same name. When a player tries to register, but another player is already there with the name selected, we need to refuse the command and obviously inform the player.

Rebuilding the state and our memory

First, as a hrefresher, let’s see how the state is built.

Let’s pretend that a player named Jack wants to register.

An AddPlayer command is issued by the UI:

history = //... list of past events
command = %AddPlayer{player_name: "Jack"}

action_result = GameState.handle_message(history, command)

The command is dispatched to the correct handler, the game logic produces a PlayerJoinedTeam event we would find in the action_result variable in the snippet above:

def handle(state, %AddPlayer{player_name: name}) do
  [
    PlayerJoinedTeam.with(player: name)
  ]
end

This event joins the others in history, the list of past events.

When we want to add a second player, we need to rebuild the state.

As seen in the previous article, GameState module contains an apply_event function with a clause matching against %PlayerJoinedTeam{} :

 def apply_event(state, %PlayerJoinedTeam{} = event) do
    %{state | players: [event.data.player | state.players]}
  end

When a PlayerJoinedTeam event is applied, the player is inserted into the list of players.

Hi! I’m Jack!

Jack #1 is already in, some other player wants to join as well, and is named Jack too, Jack #2.

A business rule disallows two players to share the same name, so we’ll need to reject Jack #2 attempt to register as Jack, and we’ll ask him to pick another name.

The traditional way of dealing with errors in Elixir code is the OK/Error tuple.

When something is going OK, the function returns a tuple in the form

{:ok, result}

and when something failed

{:error, the_error}

This is the path I decided to go with, but I made a slight twist and introduced an ActionResult module.

defmodule DoctorP.Game.ActionResult do

  alias __MODULE__

  defstruct [
    events: [],
    error: nil
  ]

  def new(), do:
    %ActionResult{}

  def new(events), do:
    %ActionResult{events: events}

  def error(error), do:
    %ActionResult{error: error}

  def add(%ActionResult{error: error}, %ActionResult{}) when not is_nil(error), do:
    %ActionResult{error: error}

  def add(%ActionResult{}, %ActionResult{error: other_error}) when not is_nil(other_error), do:
    %ActionResult{error: other_error}

  def add(%ActionResult{events: events}, %ActionResult{events: other_events}), do:
    %ActionResult{events: events ++ other_events}

end

This module is a glorified OK\Error tuple but has functions that will come in handy later. They allow us to combine multiple ActionResult together, concatenating events list when everything is right, returning an error when something went wrong.

Armed with this module, let’s see how we can implement the business rule.

First, we add a private has_player_named function that returns true when we already know a player with the name given in arguments and false otherwise.

defp has_player_named(state, player_name) do
  state.players
  |> Enum.any?(fn x -> x == player_name)
end

We can now change the handle function to

def handle(%{} = state, %AddPlayer{player_name: player_name}) do
  cond do
    has_player_named(state, player_name) -> ActionResult.error(:player_name_not_available)
    true -> ActionResult.new([PlayerJoinedTeam.with(player: player_name)])
  end
end

When a player tries to register with a name already taken by someone else, the function produces an ActionResult with an error :player_name_not_available. That same ActionResult is then returned by the GameState.handle_message function called with the command at the beginning of the process.

That’s it!

And that’s a lot... We’ll keep for a following article how to deal with the ActionResult once it’s returned from GameState.handle_message.

Building an event-sourced game with Phoenix Liveview: Expressing domain concepts in the code

2021-01-28T00:00:00Z

In the past few articles, we’ve put a lot of code in the GameState module.

The module is responsible for state rebuilding and all the game logic. It knows all details of handling card decks, timer, scoreboard, players, ...

The module size keeps growing, and it’s time to start splitting, keeping things maintainable.

This article describes one way of splitting the code. In the next article, we’ll see the second one.

Splitting up

So far, the GameState module contains all the information in a map and all the logic to deal with adding a player, dealing card to a player, starting the timer, ...

I decided to introduce several modules, each responsible for one concept in the game, BlueDeck, RedDeck, Scoreboard, Teams, Timer, ...

Adding modules helps in several aspects. First, each domain concept becomes more explicit and is encapsulated in one place. Secondly, it improves testability: we can test each concept independently without going through GameState.

Let’s look at what changes when it comes to the scoreboard module but first, let’s get back to how we handle the score in our current system.

When a game starts, we want every team score set to 0. Because we are in an event-sourced system, this logic occurs in the apply_event function, more precisely in the clause matching with the GameStarted event.

def apply_event(%{} = state, %GameStarted{}) do
  %{state | scores: List.duplicate(0, Enum.count(state.teams))}
end

Introducing the Scoreboard module allows to encapsulate the logic elsewhere and to express the intent more clearly:

def apply_event(%{} = state, %GameStarted{}) do
  %{state | scoreboard: Scoreboard.for_teams(state.teams)}
end

At this point, the Scoreboard module looks like the following:

defmodule Scoreboard do

  defstruct [
    scores: []
  ]

  def for_teams(teams) do
    %Scoreboard{scores: List.duplicate(0, Enum.count(teams))}
  end

end

Incrementing team score

When a team wins a point, we want to keep track of it by producing an event. The Scoreboard module is an excellent place to put this logic. It makes for a more cohesive base code: everything related to tracking points stays inside the same module.

def increment_team_score(scoreboard, team) do
  current_score = Enum.at(scoreboard.scores, team)

  [TeamGotAPoint.with(team: team, new_score: current_score + 1)]
end

In the GameState module, we can use this function whenever we need to increment a team score.

Let’s take a detour and add a new clause for the add function in the ActionResult module we’ve defined in the previous article.

defmodule ActionResult do
  # ...

  def add(%ActionResult{events: events}, other_events) when is_list(other_events), do:
    %ActionResult{events: events ++ other_events}

end

We now can add a list of events to an ActionResult directly. It avoids needing to encapsulate every return value from the modules we’ll create from now on in an ActionResult.

We can now increment team score in the GameState module when we mark a word as guessed.

def handle(state, %MarkWordAsGuessed{}) do
  ActionResult.new()
  |> ActionResult.add(Dictionary.give_word_to_player(state.dictionary, state.current_player)
  |> ActionResult.add(Scoreboard.increment_team_score(state.scoreboard, state.current_team_id))
end

I’ve not simplified the code too much to show that sometimes multiple modules are called while handling a message. Here we want to increment a team score and give a new word to the current player.

The ActionResult module makes it convenient to group events coming from several modules.

Rebuilding Scoreboard state

We’ve seen that state contains the scoreboard in previous snippets, but we haven’t seen yet how the scoreboard state is maintained.

We keep track of teams' scores with events here as well. Scoreboard can be event-sourced too.

defmodule Scoreboard do
  # ...
  def apply_event(%Scoreboard{} = scoreboard, %TeamGotAPoint{data: %{team: team, new_score: team_score}}), do:
    %Scoreboard{scores: List.update_at(scoreboard.scores, team, fn _ -> team_score end)}
  end
end

And in GameState

def apply_event(%{} = state, %TeamGotAPoint{} = e), do:
  %{state | scoreboard: Scoreboard.apply_event(state.scoreboard, e)}

Going further

During this article’s redaction, an idea popped up in my head that I haven’t tried but is worth sharing.

GameState's apply_event function now does very little. It takes the state and an event as parameters and dispatches to another module.

Looking at the previous example close to other pieces of code we see an emerging pattern:

def apply_event(%{} = state, %TeamGotAPoint{} = e), do:
 %{state | scoreboard: Scoreboard.apply_event(state.scoreboard, e)}

def apply_event(%{} = state, %DictionaryShuffled{} = e), do:
  %{state | dictionary: Dictionary.apply_event(state.dictionary, e)}

def apply_event(%{} = state, %GaveWordToPlayer{} = e), do:
  %{state | dictionary: Dictionary.apply_event(state.dictionary, e)}

We could avoid writing these pieces of code for each event by generalizing. Something like:

def apply_event(%{} = state, e) do

  Enum.reduce(state, fn {key, value}, state ->
      module = value.__struct__

      Map.put(state, key, apply(module, :apply_event, [state, e]))
  end)
end

I haven’t tried this code, I don’t know if it works, nor it’s even a good idea.

We’ve seen one way of breaking apart the GameState module. In the next articles, we’ll see another decision I’ve made to reduce its size further down and improve communication around domain concepts.

Building an event-sourced game with Phoenix Liveview: Making game states explicit

2021-02-04T00:00:00Z

In the last article, we’ve seen the first design choices I’ve taken to reduce the size of the GameState module. Now, it’s time to see the second decision I’ve made.

During this project, I wanted to experiment with expressing business logic with the type system in a functional language.

Business logic as types

While creating types to express business logic is something I’ve seen applied in OOP codebases, I’ve never played with that idea in a functional language.

This idea is presented in Scott Wlaschin's book Domain Modeling Made Functional. The book examples are written in F#, a language I haven’t got the chance to play with yet, but delivers interesting points applicable to Elixir nevertheless. It can also serve as a lightweight introduction to Domain Driven Design, and it’s worth a read.

The main point Scott makes is that by modeling using the type system we are able to create a robust and documented system expressed in the language of the business.

One example of modeling with types is creating different types for different states. For instance, an email address could be validated or not validated yet, leading to the creation of UnvalidatedEmailAddress and ValidatedEmailAddress types, probably a workflow, able to transform an UnvalidatedEmailAddress to a ValidatedEmailAddress.

The game can be seen as a state machine, either waiting for players to register, waiting for a round to start, or in a round. Instead of having one type, materialized by an Elixir module with a struct containing many attributes to represent all states, as we have so far, Scott’s advice leads us to create three types, BetweenRound, InRound, Waiting.

I admit having been lazy here and could probably have given more extended and more explicit names.

The three modules:

defmodule States.Waiting do

  # aliases and includes omitted

  defstruct [
    teams: Teams.new()
  ]
end

defmodule States.InRound do

  defstruct [
    :blue_deck,
    :red_deck,
    :teams,
    :scoreboard,
    :dictionary,
    timer: Timer.round_time()
  ]
end

defmodule States.BetweenRound do

  defstruct [
    :blue_deck,
    :red_deck,
    :teams,
    :scoreboard,
    :dictionary
  ]
end

Each module declares a structure with the precise information needed to do its job correctly. We can get rid of the permissive map we were using until now.

We see some structure attributes are coming with default values. Every time the game arrives in that state, we want to reset that value.
timer in the InRound state is probably the best example: when a round starts, the timer is always set to the expected round duration.

We can also notice that InRound and BetweenRound states share many attributes as we want to preserve the game data while doing back and forth between these two states. When moving from InRound to BetweenRound we want to keep each team score.

Grouping behaviors into states

Now that we have three states, we can move functions related to each of them in the appropriate module.

The handle function matching with the MarkWordAsGuessed message can move to the InRound module, the one dealing with AddPlayer in Waiting, and so on.

defmodule States.Waiting do
	# ...

  def handle(%Waiting{} = waiting_room, %AddPlayer{player_name: player_name}) do
  # ...
  end

end


defmodule States.InRound do
  #...

  def handle(%InRound{}, %MarkWordAsGuessed{}) do
  #...
  end

end

An interesting thing to note here is that thanks to the introduction of structures, we can pattern match the current state. It makes it more straightforward which actions can be taken on each state.

Furthermore, it will prevent any action made against the wrong state. If MarkWordAsGuessed message were to be dispatched when we’re in Waiting state, the application would crash.

If we want to avoid crashing, we can add a clause, matching on all unmatched messages, that returns an error:

def handle(%Waiting{}, _), do:
  ActionResult.error(:action_not_allowed)

We also have to group the state mutation functions.

defmodule States.Waiting do

  def apply_event(%Waiting{} = state, %PlayerJoinedTeam{} = e), do:
    %Waiting{state | teams: Teams.apply(state.teams, e)}

  def apply_event(%Waiting{} = state, %PlayerLeftTeam{} = e), do:
    %Waiting{state | teams: Teams.apply(state.teams, e)}

end

defmodule States.InRound do

  def apply_event(%InRound{} = state, %TeamGotAPoint{} = e), do:
    %InRound{state | scoreboard: Scoreboard.apply_event(state.scoreboard, e)}

end

Here, we pattern match on the current state structure, and functions return a structure.

Transition between states

We now have multiple states and cleaned up the GameState module.

All examples I’ve shared so far show events that, when applied, stay in the same state. This is great, but we still need to transition from state to state, going from Waiting to BetweenRound and doing back and forth between BetweenRound and InRound.

The solution is simple. When needed, the apply_event can return the structure of the next state.


defmodule States.Waiting do

  def apply_event(%Waiting{} = state, %GameStarted{}), do:
    %BetweenRound{teams: state.teams, scoreboard: Scoreboard.for_teams(teams)}

end

defmodule States.BetweenRound do

  def apply_event(%BetweenRound{} = state, %RoundStarted{}) do
    %InRound{
      blue_deck: state.blue_deck,
      red_deck: state.red_deck,
      dictionary: state.dictionary,
      teams: state.teams,
      scoreboard: state.scoreboard
    }
  end

end

def module States.InRound do
  def apply_event(%InRound{} = state, %RoundEnded{}) do
    %BetweenRound{
      blue_deck: state.blue_deck,
      red_deck: state.red_deck,
      dictionary: state.dictionary,
      teams: state.teams,
      scoreboard: state.scoreboard
    }
  end

end

I’m still puzzled about directly using another module structure in a different module. For instance, I don’t like that Waiting module knows how to create the Scoreboard for the BetweenRound structure. It would probably be better to introduce a function in each module, some sort of constructor, dealing with all the details. For lack of a name I liked, I’ve decided to keep the code as is. If you have an idea, feel free to tell me!

Here is a schema of part of the state machine, with events leading to the same states and others occasioning transitions.

What’s left to GameState

We’ve moved all the game logic away from the GameState module to the states modules. That improvement comes with an issue. We don’t have one single entry point to dispatch messages. We need to know the current state before select which module’s handle function to call.

We also still need someplace to put the logic of rebuilding the current state based on history.

GameState is the perfect place for this!

defmodule DoctorP.Game.States.GameState do

  alias States.Waiting

  def dispatch_message(message, history]) do
    build_state(history)
    |> handle(message)
  end

  def build_state(history) do
    List.foldl(history, %Waiting{}, fn event, state ->
      apply_event(state, event)
    end)
  end

  def handle(state, command) do
    state
    |> module()
    |> apply(:handle, [state, command])
  end

  def apply_event(state, event) do
    state
    |> module()
    |> apply(:apply_event, [state, event])
  end

  defp module(state), do: state.__struct__
end

This all that’s left in the module.

The dispatch_message function stays the same, ensuring that the state is rebuilt before handling the message.

build_state is slightly changed to deal with states as structure instead of a map. The first value of the state is a Waiting structure. Indeed, each game starts by waiting for players to registers.

In apply_event, we need to know which state module is the good one based on the state before calling apply_event on it. This is done by the module function, which reads the __struct__ key on the current state.

That’s it!

We’ve separated all the game logic from the event-sourcing and message handling one.

Doing so, we’ve also improved the ability to understand our system: by looking at the list of modules, we’re able to know that the game can be in three states.

Sure we’re still far away from everything that using the type system to express business logic. It’s only the first step. The same ideas could be applied to the modules we’ve seen in the last article. Typespecs would also improve the documentation and express what can’t be done by the system. It’s not something I’ve worked on yet. Maybe later then.

In the next article, we’ll probably start looking at the runtime characteristics of the game.

Building an event-sourced game with Phoenix Liveview: Game Server

2021-02-15T00:00:00Z

In the previous articles, we’ve looked at the design of the game’s rules. Now is an excellent time to focus on the runtime perspective of the application.

We’ve seen that we can interact with a game via the GameState module by providing a list of events and a command. We still need some way to store the events, and for this, we’ll use a GenServer.

Skeleton of `GameServer`

GameServer module is a GenServer which allows keeping track of the events in a process.

defmodule DoctorP.GameServer do
  use GenServer

def start_link(args) do
  game_id = Keyword.get(args, :game_id)
  GenServer.start_link(__MODULE__, args, name: process_name(game_id))
end

def init(args) do
  game_id = Keyword.get(args, :game_id)
  {:ok, %{game_id: game_id, events: []}}
end

def process_name(game_id), do:
  {:via, Registry, {DoctorP.GamesRegistry, game_id}}

end

We can start a GameServer using the start_link function, passing a keyword list containing a game id. The server’s initial state is a map containing the game id and an empty list of events.

We also want to keep track of several games at the same time and to be able to interact with them using their game id. I’ve decided to use the via tuple method to link a game id and the server PID in a Registry. process_name returns a tuple that indicates which registry and key to use to find the server.

Playing

The next thing to do is to be able to act on the GameServer.

All actions follow the same patterns. For the sake of brevity, let’s focus solely on adding a player.

First, I’ve introduced a client side function. It allows for an easy to use interface for other modules:

def add_player(game_id, player_name), do:
  GenServer.call(process_name(game_id), {:add_player, player_name})

When called, add_player issues a call to the server, using the process_name based on the game_id and asking to add a player with the provided player name.

Its server-side counterpart is more interesting as it is responsible of dispatching a command to the game, storing the events, and replying to the caller.

def handle_call({:add_player, player_name}, _from, state), do:
  %AddPlayer{player_name: player_name}
  |> handle_command(state)
  |> reply()

First, the AddPlayer command is built and piped into to a private handle_command function, alongside the state . The result of this call is, in turn, piped into a private reply function.

Let’s dive into these!

Handling the command

defp handle_command(command, state), do:
  command
    |> dispatch_command(state)
    |> build_new_state(state)

defp dispatch_command(command, state), do:
  GameState.dispatch_message(command, state.events)

defp build_new_state(%ActionResult{events: events} = result, state), do:
  {result, %{state | events: state.events ++ events}}

handle_command is made of two parts, each one in its function.

First, we use the GameState module to dispatch the command. We can see that all previous events, currently stored in the state, are passed as a second parameter. It ensures the command handler knows the current state.

Secondly, the command dispatch result, an ActionResult is piped into build_new_state. Here the new state is created by appending the news events to the events currently stored. Note that build_new_state returns a tuple containing the result of the dispatch and the new state.

Replying to the caller

Once the command is dispatched, the game logic applied and, the new state created we need to reply to the caller.

This is the job of the reply function we’ve seen in the handle_call function.

build_new_state returns a tuple containing an ActionResult and the desired state after the command handling, which serves as a parameter for our new reply function.

defp reply({ %ActionResult{}, new_state}), do:
  {:reply, :ok, new_state)

Here we simply reply with the :ok tuple and use the second part of the tuple for the new server’s state.

The first part of the tuple seems useless by now, but it’s actually important.

We’ve seen in the article about error handling that part of ActionResult's job is to keep track of errors. So far, reply only handles the positive cases.

I’ve decided to communicate errors to the caller by responding with a tuple starting with the :error atom and containing the error provided by the game.

To do this we need to add a clause for the reply function matching with ActionResult containing errors:

defp reply({ %ActionResult{error: error}, new_state}, state) when not is_nil(error), do:
  {:reply, {:error, error}, new_state)

The GameServer can now deal with both positive outputs, made of new events and errors, and communicates them to callers.

There are still exciting things to say about the GameServer, from how it communicates with the Liveviews, to stopping them and more. These are the things we’ll cover in the next articles!

Building an event-sourced game with Phoenix Liveview: Acting on the game from the views

2021-02-22T00:00:00Z

In the last article, we’ve seen the GameServer implementation. It’s now time to look at the views, specifically how players interact with the game.

Starting the game from the LiveView

I decided to use LiveViews as they are convenient for real-time interactions without requiring any Javascript code.

In this article, we’ll focus on the LiveView used for players to register as it exhibits handling form, sending messages to the GameServer, and handling potential errors.

Let’s first create a new module for our LiveView. When the view is mounted, the game_id available in the route parameters is stored in the state.

defmodule DoctorPWeb.WaitingRoomLive do

def mount(%{game_id: game_id}, _session, socket) do

 state = %{game_id: game_id}

  socket = socket
  |> assign(state)

  {:ok, socket}
end

end

The simplest case to handle in this state is handling the click on the "Start game" button. The LiveView receives an event without any parameters and sends a message to the GameServer.

@impl true
def handle_event("start_playing", _params, socket) do
  GameServer.start_playing(socket.assigns.game_id)

  {:noreply, socket}
end

Thanks to the client function we’ve added to GameServer and being able to find it using the game ID, the implementation is easy. When a start_playing event is received by the LiveView process a call is issued to GameServer.start_playing(socket.assigns.game_id).

Registering a player from the LiveView

Registering a player is more complex because it involves a form for the player to specify her name.

Following ideas from this article, I managed to use Ecto as a validation library to create a form that is not tied to a database.

We declare a module describing the form:

defmodule RegisterPlayer do
  use Ecto.Schema
  import Ecto.Changeset

  embedded_schema do
    field(:player_name, :string)
  end

  alias __MODULE__

  def changeset(register_player, params \\ %{}) do
    register_player
    |> cast(params, [:player_name])
    |> validate_required([:player_name])
    |> validate_length(:player_name, min: 1)
  end

  def apply_changes(changeset) do
    Ecto.Changeset.apply_changes(changeset)
  end

end

Once this module is created, we can use the form in the LiveView.
In the mount function we need to add the form changeset.

state = %{game_id: game_id, changeset: RegisterPlayer.changeset(%RegisterPlayer{})}

And modify the template to display the form:

<%= form_for @changeset, "#", [phx_submit: :register_player, id: "player_registration_form"], fn f -> %>

<%= label f, :player_name %>
<%= text_input f, :player_name, autofocus: true, placeholder: "What’s your name?" %>
<%= error_tag f, :player_name %>

<%= submit "Register", phx_disable_with: "Registering..." %>
<% end %>
</form>

With this in place, the next step is to handle the form by creating a clause for handle_event matching the register_player event.

def handle_event("register_player", %{"register_player" => register_player} = params, socket) do

  changeset =
    %RegisterPlayer{}
    |> RegisterPlayer.changeset(register_player)
    |> Map.put(:action, :insert)


 with true <- changeset.valid?,
                   registration <- RegisterPlayer.apply_changes(changeset),
                   :ok <- GameServer.add_player(socket.assigns.game_id, registration.player_name) do
    {:noreply, socket}
  else
    _ -> {:noreply, assign(socket, :changeset, changeset)}
  end

end

In the first part of the function, a new changeset for the form is created based on the form’s data.

If the changeset is valid, the changes are applied to an empty RegisterUser structure. We access data from that structure to call GameServer.add_player.

If anything is not going as expected, we enter the else part of the with structure. The changeset is given back to the view, allowing to display form validation errors to the player.

Displaying errors from the `GameServer`

Another type of error is the ones returned by the GameServer when it cannot handle a command.

Let’s take an example with the registration process. As seen in the article about errors, we disallow two players to share the same name. If a player tries to register with a name taken by another player, an :player_name_not_available is returned by GameServer.add_player.

Sure, we could add some logic in the WaitingRoomLive LiveView module to keep track of all registered players and prevent form submission if needed, but what if two players try to register with the same still available name at the exact same moment?

The GameServer, as the source of truth, will handle one command after the other and refuse the second one.

Let’s see how to handle this.

First, we add the add_player_name_already_taken_error to the RegisterPlayer module, the one responsible for the registration form.

defmodule RegisterPlayer do
	# ....
  @name_already_taken_error_message "This name is already taken by another player"

 def add_player_name_already_taken_error(changeset), do:
    changeset
    |> add_error(:player_name, @name_already_taken_error_message)
end

This function adds an error for the :player_name field to the changeset.

Then, we can modify the with structure we’ve seen previously to use that function when needed.

with true <- changeset.valid?,
                 registration <- RegisterPlayer.apply_changes(changeset),
                 :ok <- GameServer.add_player(socket.assigns.game_id, socket.assigns.player_id, registration.player_name) do
  {:noreply, socket}
else
  {:error, :player_name_not_available} -> {:noreply, assign(socket, :changeset, RegisterPlayer.add_player_name_already_taken_error(changeset))}
  _ -> {:noreply, assign(socket, :changeset, changeset)}
end

If GameServer.add_player returns the {:error, :player_name_not_available} error tuple, we assign a modified changeset with the error to the socket.

That’s it for this article. We’ve covered all needed for players to act against the game. In the next article, we’ll see how the view constructs its state from the game’s state.

Building an event-sourced game with Phoenix Liveview: Building the view’s states from the events and reacting to changes

2021-03-01T00:00:00Z

In the last article, we’ve seen how the view communicates with the GameServer to send commands and act on the game. This article focuses on how the views construct their states.

The game being event-sourced its state is derived from the list of past events, and view states are not going to be different. In CQRS-ES language, they are called projections.

The first mode of building the view is rebuilding. When the view is loaded, we want it to represent the current state of the game. For example, when the game is waiting to start, some players might join while others are still not on the view. When a new player arrives, we want to display the list of all previously registered players.

The second mode is the running mode of building the view: when something occurs in the game, we want displayed information to change if necessary.

Rebuilding the view

First, let’s introduce the EventSourcedLiveView module that will encapsulate logic commons to all event-based views.

defmodule DoctorPWeb.EventSourcedLiveView do

  @callback view_init( Phoenix.LiveView.unsigned_params() | :not_mounted_at_router,
              session :: map(),
              socket :: Phoenix.LiveView.Socket.t()) :: {integer(), map()}

  defmacro __using__(opts \\ []) do

    quote do

      use DoctorPWeb, :live_view

      @impl true
      def mount(params, session, socket) do

        {game_id, init_state} = view_init(params, session, socket)

        socket = socket
                 |> assign(init_state)

        {:ok, socket}
      end
    end
  end
end

The module declares a macro that allows creating a specific type of LiveView, the event-sourced ones.

When the view is mounted, the init_view function is called and is expected to return a tuple containing the game ID and a map used as the initial state.

We can remove the mount function from the WaitingRoomLive we’ve seen in the previous article and instead write:

defmodule DoctorPWeb.WaitingRoomLive do
  use DoctorPWeb.EventSourcedLiveView

  def view_init(%{game_id: game_id}, session, socket) do
    {
      game_id,
      %{
        changeset: RegisterPlayer.changeset(%RegisterPlayer{}),
        game_id: nil,
      }
    }
  end
end

To rebuild the state from past events, we’ll modify the EventSourcedLiveView module.

The mount function fetches all known events from the GameServer and passes them to set_states, with the init_state we have from the previous call to init_view.

events = GameServer.get_events(game_id)
socket = socket
         |> assign(set_state(init_state, events))

set_state function goes through all events and looks for a clause of the apply_event matching each one of them. This is the same mechanism as for rebuilding the state we’ve seen when introducing event-sourced systems.

defp set_state(state, history), do:
  Enum.reduce(history, state, fn e, state -> apply_event(e, state) end)

With this in place, let’s see how we can display the teams as an example. We’ve seen before that the game dispatches a PlayerJoinedTeam event when a player registers, and we can use these events to achieve what we want.

First, we add an empty list as the list of teams in the initial state:

def view_init(%{game_id: game_id}, session, socket) do
    {
      game_id,
      %{
        changeset: RegisterPlayer.changeset(%RegisterPlayer{}),
        game_id: nil,
	teams: []
      }
    }
end

Then, we add the apply_event function clause matching with the PlayerJoinedTeam event.

defp apply_event(%PlayerJoinedTeam{data: %{player: player, team: team_id}} = event, state) do
  state
  |> Map.put(:teams, add_player_to_teams(state.teams, player, team_id))
end

defp add_player_to_teams(teams, player, team_id) do
  team = teams
         |> Enum.at(team_id, [])
         |> (fn t -> List.insert_at(t, length(t), player) end).()

  if Enum.at(teams, team_id, nil) == nil do
    List.insert_at(teams, team_id, team)
  else
    List.replace_at(teams, team_id, team)
  end
end

This function, alongside the private add_player_to_teams function, adds the player to the indicated team in the list in the LiveView state.

Modifying the state while the game is running

We’ve seen just above how the view state is built from the already known events when the view is mounted, but we haven’t touched the funnier part yet, modifying the view when something occurs in the game. We’ll see this in that second section.

To communicate between the GameServer and the LiveViews, we’ll use the Phoenix PubSub system.

We introduce a GamePubSub module. Its role is to make it convenient to publish and subscribe to message for a specific game.

defmodule GamePubSub do

  def subscribe(game_id), do:
    Phoenix.PubSub.subscribe(DoctorP.PubSub, game_topic(game_id))

  def publish(game_id, message), do:
    Phoenix.PubSub.broadcast(DoctorP.PubSub, game_topic(game_id), message)

  def game_topic(game_id), do: "game_#{game_id}"

end

Next, the GameServer we’ve worked on in a previous article can be modified to dispatch events:

defmodule GameServer
#...
defp handle_command(command, state), do:
  command
    |> dispatch_command(state)
    |> dispatch_events(state)
    |> build_new_state(state)
end

defp dispatch_events(%ActionResult{events: events} = result, state) do
  for event <- events, do: GamePubSub.publish(state.game_id, event)

  result
end

The handle_command now pipes the result of dispatch_command into dispatch_events function.

dispatch_events calls the publish function of the newly created GamePubSub module for each event. The function returns its first parameters, the ActionResult coming from dispatch_command, which allows continuing to pipe into build_new_state.

Every time a command is dispatched and the game produces events, the GameServer broadcasts them via PubSub. That means we can subscribe and wait for events.

The EventSourcedLiveView module will precisely do this.

The mount function is changed to

def mount(params, session, socket) do

  {game_id, initial_state} = view_init(params, session, socket)

  if connected?(socket) do
    GamePubSub.subscribe(game_id)
  end

  socket = socket
           |> assign(set_state(initial_state, GameServer.get_events(game_id)))

  {:ok, socket}
end

When the view is connected, we subscribe for new events about the game.

For every message broadcasted by the PubSub system, the handle_info function is called and calls the apply_event function, hoping that a matching clause for the event exists. In that case, the state is modified and assigned to the socket.

@impl true
def handle_info(event, socket) do
  state = apply_event(event, socket.assigns)
          |> clean_assigns()

  {:noreply, socket |> assign(state)}
end

defp clean_assigns(assigns) do
  {_, clean_assigns} = Map.pop!(assigns, :flash)
  clean_assigns
end

A note here: I had to add the clean_assigns function to prevent LiveView assigns' error for the :flash reserved keys.

The apply_event function called in handle_info is the same one used when the view is mounted. It means the logic we created for building the list of teams is already working!

That’s it! We’ve seen everything needed for our views to display the correct information when they are mounted or when something occurs in the game.

In the next article, we’ll see the part I enjoyed the most coding in this project, as it forced me to rethink some pieces and improve the design, which is dealing with the timer.

Building an event-sourced game with Phoenix Liveview: Decrementing the timer

2021-03-08T00:00:00Z

One thing that always comes with challenges in applications is time. In Docteur Pilule, we have an instance of time in the form of a timer: a round is 40 seconds long.

When I started coding the game, I wanted to display the decrementing counter, changing every second, to players.

In this article, we’ll see that thanks to Elixir, Liveview, and the design choices made so far, it’s pretty easy to do.

Sending a message to self

Here I propose a solution based on what Greg Young describes as "sending a message to self".

The idea is that a component passes a message to a delivery mechanism with a date it would like to receive the message and goes back to its own business. When it’s time, the delivery mechanism sends the message to the component to be handled as any other message.

Future message container

We’ve seen in a previous article that every action returns an ActionResult structure. The ActionResult module is the mean of communication between the game rules and the game server. It can be extended to handle future messages.

First, we add a new key in the structure, scheduled_messages. That key points to a list of tuples containing a delay in milliseconds and a message.

defmodule ActionResult do

  defstruct [
    events: [],
    scheduled_messages: [],
    error: nil
  ]

We can then complete the module with functions helping us deal with combining several ActionResults that may contain scheduled messages.

def new(events, scheduled_messages), do:
  %ActionResult{events: events, scheduled_messages: scheduled_messages}

def add(%ActionResult{events: events, scheduled_messages: scheduled_messages}, %ActionResult{events: other_events, scheduled_messages: other_scheduled_messages}), do:
  %ActionResult{events: events ++ other_events, scheduled_messages: scheduled_messages ++ other_scheduled_messages}

def add(%ActionResult{events: events, scheduled_messages: scheduled_messages}, other_events) when is_list(other_events), do:
  %ActionResult{events: events ++ other_events, scheduled_messages: scheduled_messages}

Decrementing the timer

Choosing to start, decrement, or stop the timer is a game rule and thus resides in the application’s game logic part. As seen previously, each concept in the game gets its module with functions acting as commands and the apply_event callback returning a new state when an event is applied.

Even if the Timer module needs to deal with scheduled messages, it is no different than other modules.

defmodule Timer do
 # ...
  @round_time 40
  @one_second :timer.seconds(1)

  def start_time(), do:
    ActionResult.new(
      [
        RoundTimeStarted.with(
          remaining_time: @round_time
        ),
      ],
      [{@one_second, %Tick{}}]
    )

  def decrement_time(1), do:
    ActionResult.new([%RoundEnded{}])

  def decrement_time(current_time) do
    time = current_time - 1
    ActionResult.new(
      [RoundTimeTicked.with(remaining_time: time)],
      [{@one_second, %Tick{}}]
    )
  end

  def apply_event(_time, %RoundTimeTicked{data: %{remaining_time: remaining_time}}), do: remaining_time

  def apply_event(_time, %RoundTimeStarted{data: %{remaining_time: remaining_time}}), do: remaining_time

end

The module defines two module attributes, @round_time and @one_second, respectively describing the round duration in seconds and one second expressed in milliseconds.

The start_time function returns an ActionResult containing a RoundStarted event with a remaining time set to @round_time. The ActionResult also contains a scheduled message, %Tick{}, programmed to be received in one second.

Similarly, decrement_time decrements the time, produces a RoundTimeTicked event, and schedules a %Tick{} for one second later.

If the timer’s value is 1, the first clause of decrements_time is selected. It produces an %RoundEnded{} event and doesn’t schedule a %Tick{} as there is no need to decrement the timer anymore.

apply_events for %RoundTimeTicked{} and %RoundTimeStarted return the remaining time contained in the event as the new timer state.

The InRound module, which encapsulates all game behaviors during a round, stays simple.

When a player starts a round, the timer starts, and a word is distributed.

def start_round(game), do:
  ActionResult.new()
  |> ActionResult.add(Timer.start_time())
  |> ActionResult.add(Dictionary.give_word_to_player(game.dictionary, Teams.current_player(game.teams)))

When a %Tick{} message is received, the timer is decremented.

def handle(%InRound{} = game, %Tick{}) do
  Timer.decrement_time(game.timer)
end

The application of an event related to the timer is delegated to the Timer module. The apply_event` function returns the new timer state.

def apply_event(%InRound{} = state, %RoundTimeStarted{} = e), do:
  %InRound{state | timer: Timer.apply_event(state.timer, e)}

def apply_event(%InRound{} = state, %RoundTimeTicked{} = e), do:
  %InRound{state | timer: Timer.apply_event(state.timer, e)}

The game logic now deals with the timer: it can decide when the timer must be started, schedule the timer to be decremented a second later, and decrement and stop it.

This is everything to know about changes in the game logic. We now need to talk about the delivery mechanism for the scheduled message.

The `GameServer` as scheduled messages delivery mechanism

The part title totally spoiled it: we’re going to use the GameServer as our delivery mechanism for scheduled messages.

Using Process.send_after, Elixir makes it very easy to send to a process a message in the future.

Let’s add a private schedule_messages function in the module:

defp schedule_messages(%ActionResult{scheduled_messages: scheduled_messages} = result) do
  for {delay, message} <- scheduled_messages, do:
    Process.send_after(self(), {:scheduled_message, message}, delay)

  result
end

This function looks for the scheduled messages inside an ActionResult, and for each one of them sends to self(), the current GameServer process, a tuple message {:scheduled_message, message} delayed by the amount of time requested by the game logic. It conveniently returns the ActionResult, which means we can add this function to a chain of piped functions.

Once the delay is over, the GameServer process handle_info callback is called with the message.

We can create a clause for handle_info and let it handle the message as a command and then store the state.

def handle_info({:scheduled_message, message}, state), do:
  new_state = handle_command(message, state)
  {:noreply, new_state}

We’ve already seen the handle_command function. This is the function we call when the GameServer receives a command. As a reminder, the function dispatches the command to the game logic, broadcast events for the view, and collect events to store them in the state.

It’s the perfect place to add schedule_message. Adding it here means that messages are scheduled when a command is handled and when a scheduled message is handled, precisely what we want. The first case takes care of starting the timer to tick when the round begins, and the second case deals with scheduling the next tick when a tick happens.

defp handle_command(command, state), do:
  command
    |> dispatch_command(state)
    |> dispatch_events(state)
    |> schedule_messages()
    |> build_new_state(state)

With this in place, the game logic can express its need to be notified later to take decisions without being coupled to any infrastructure concerns. As promised, it required only a few code additions, no massive rework, and keeps the application well organized.

Building an event-sourced game with Phoenix Liveview: Unit testing patterns

2021-04-07T00:00:00Z

We’ve seen a lot about the game but haven’t talked yet about testing. When dealing with event-sourced system testing comes with some interesting patterns. We’ll see some of them in this article.

Testing in event-sourced system

As with everything else with an event-sourced system, testing is based on events. The three tests parts, arrange, act and assert, can be mapped as follow:

Arrange: Create a history of events

Act: Recreate state from events and dispatch a command

Assert: Ensure the system produced some events

Here is an example of a test ensuring that the second player is enlisted in the second team.

defmodule WaitingRoomTest do
  use ExUnit.Case, async: true

  describe "Add player" do

    test "adds second players to the second team" do
    # Arrange: create a history where a player already joined the first team
	  history = [
      PlayerJoinedTeam.with(player: %{name: :player_1}, team: 1)
    ] 
      
    # Act: Dispatch a command to add a second player based on the history
    # Rebuilding state from history is dealt with by GameState module
    %ActionResult{events: events} = GameState.dispatch_message( %AddPlayer{player_name: :player_2}, history)

    # Assert: Ensure that an event indicating that the second player joined the second team was produced
    assert Enum.member?(events, PlayerJoinedTeam.with(player: %{name: :player_2}, team: 2))

    end
  end
end

I like my tests to read easily and decided to introduce a new module to take advantage of Elixir’s piping to improve expressiveness. We can build on top of the fact that GameState.dispatch_message returns an ActionResult.

The first function in the new module checks if an event was published.

defmodule ActionResultHelper do

  def published_event?(%ActionResult{events: events}, event), do:
    Enum.member?(events, event)

end

We can rewrite the previous test with this in place once we’ve imported the ActionResultHelper module.

test "adds second players to the second team" do
  history = [
    PlayerJoinedTeam.with(player: %{name: :player_1}, team: 1)
    ] 
      
  assert dispatch_message( %AddPlayer{player_name: :player_2}, history)
	  |> published_events?(PlayerJoinedTeam.with(player: %{name: :player_2}, team: 2))

end

The separation between the act and the test’s assert parts is more blurry, but I think the test better conveys what the system does, so I’m ok with that.

Verify that an error is returned

We can add more functions to the ActionResultHelper module to verify that the game’s behavior is as expected. One case is ensuring that the game correctly returns an error when needed.

First, let’s add a function.

defmodule ActionResultHelper do

def errored_with?(%ActionResult{error: error}, expected_error), do:
  error == expected_error

end

Then, in the tests, we can write something like

test "refuses a player with a name already taken" do
  player_name = :same_player_name

  history = [
    PlayerJoinedTeam.with(player: %{name: player_name}, team: 3),
  ]

  assert %AddPlayer{player_name: player_name}
         |> dispatch_message(history)
         |> errored_with?(:player_name_not_available)
end

Again, this reads very well!

Ensuring that a message is scheduled

We’ve seen that the application sometimes needs to schedule a message that it would like to receive in the future. We can follow the same pattern to verify that the game logic well produces these messages.

Introduce a new function to the helper module

defmodule ActionResultHelper do

def scheduled_message_in?(%ActionResult{scheduled_messages: scheduled_messages}, message, time), do:
  Enum.member?(scheduled_messages, {time, message})

end

And write a test. Here we ensure that when a Tick message is dispatched the next one is scheduled for one second later.

test "schedules a tick message for 1 second later" do
  
  history = ...

  assert %Tick{}
         |> dispatch_message(history)
         |> scheduled_message_in?(%Tick{},  :timer.seconds(1))
end

Being lazy

Sometimes a command creates a lot of events, and it is painful to manually build them all to create a history of a game after several rounds.
We can streamline the arrange part of tests if we use commands to create the history. In that case, we need to collect the events produced when the command is dispatched and add them to the current list of events to complete the history.

defmodule ActionResultHelper do

  def start_history(), do:
    ActionResult.new()

  def dispatch_and_collect_events(%ActionResult{} = history, message) do
      history
      |> ActionResult.add(GameState.dispatch_message(message, history |> event() , opts))
  end

  def events(%ActionResult{events: events}), do: events

end

Here I’ve added a few more functions to the helper module. StartHistory returns an empty %ActionResult{} structure. The dispatch_and_collect_events function dispatches a message to the game using the events stored in an ActionResult and adds events resulting from the dispatch to it. Events function returns the events contained in the ActionResult.

We can use these functions to build history. For instance, in the next snippet, we build the history up to after the game starts.

history = start_history()
  |> add([  #ActionResult.add
   PlayerJoinedTeam.with(player: player(:player_A1), team: 1),
   PlayerJoinedTeam.with(player: player(:player_A2), team: 1),
   PlayerJoinedTeam.with(player: player(:player_B1), team: 2),
   PlayerJoinedTeam.with(player: player(:player_B2), team: 2)
  ])
  |> dispatch_and_collect_events(%StartGame{blue_deck: blue_cards, red_deck: red_cards, dictionary: words})
  |> events()

Another instance where it’s super helpful is for the last tick of round, where many things are going on.

With the three following lines, we’re now able to start a round, fast-forward to the penultimate tick and trigger one last tick.

|> dispatch_and_collect_events(%StartRound{})
|> add([RoundTimeTicked.with(remaining_time: 1)])
|> dispatch_and_collect_events(%Tick{})

Being super lazy

When a player marks a word as guessed, the game produces a lot of events. We’ve seen how the introduction of the dispatch_and_collect_events can help here. One last issue persists when testing that the game correctly ends after 20 words guesses; we need to copy-paste dispatch_and_collect_events(%MarkWordAsGuessed{}) a lot.

We can fix this by adding an extra, optional parameter to dispatch_and_collect_events, which states how many times we want to repeat the message.

def dispatch_and_collect_events(%ActionResult{} = history, message, opts \\ []) do
  repeat = Keyword.get(opts, :repeat, 1)

  Enum.reduce(1..repeat, history, fn _x, %ActionResult{events: events} = result ->

    result
    |> ActionResult.add(GameState.dispatch_message(message, events, opts))

  end)
end

This change allows reducing the amount of typing necessary to build the history.

Here we build a history where a player marked 20 words as guessed.

history
|> dispatch_and_collect_events(%MarkWordAsGuessed{}, repeat: 20)

That the last one of the helpful patterns for testing applications built using event-sourcing I’ve introduced in this project. I hope they can give you some ideas for your own projects!

An alias to learn aliases

2021-06-11T00:00:00Z

I’m trying to save some time by reducing the number of keystrokes I need to do something.

I’m pretty efficient with my IDE’s shortcuts, but I have vast room for improvement in my console.

The obvious solution there is relying on aliases.

I’ve been using the git ssh plugin for several years, but I only use a handful of the provided aliases, mostly because I never took the time to memorize past the few main ones.

Now that I’m trying to save some typing, I’m learning the other ones. I have an issue, though: I spend a lot of time looking at the documentation to know which alias shortens the command I want to enter. In that case, it means I switch to my browser, get the right tab, or go to the website, use the search to find the suitable alias by typing part of the command. I don’t save a lot of keystrokes, and I’m surely not gaining time.

Please don’t take me wrong; time spend learning or practicing to improve in the long run is time well spent. Nevertheless, I decided to find a way to enhance my learning process.

The best way to reduce the amount of time needed to find the proper alias for a command is to stay in the console and avoid looking at the documentation.

After a quick search, I discovered that the alias command lists all aliases set up. That’s a start.

Combining alias and grep makes it efficient to find the correct alias for the job.

That still requires some typing, if your shell doesn’t have some sort if autocomplete at least.

As I’m in a quest for laziness and poking around with aliases, I decided to add a new one, fa, for “find alias” defined as alias fa='alias | grep'.

Just two letters, no more windows switching, and a lot of time saved.

Generating dates and UUIDs easily in PhpStorm

2021-08-13T00:00:00Z

Legacy codebases tend to lack tests. One way to introduce some safety net before messing around with the code is to add characterization tests.
In this kind of test, we feed some input to the system under test and look at the outputs, ensuring that they stay the same test run after test run. It’s far easier to fix inputs once and for all.

When creating this type of test, we want to avoid moving or random values. Two of the most common are dates and UUIDs. While it’s possible to generate UUIDs using external tools, there is room for improvement if you want to do everything directly in PhpStorm. Even if typing a date is not as painful, we can gain some time here too.

PhpStorm, and probably all IntelliJ IDEs, allow creating Live templates. Live templates are templates that we can expand by typing an abbreviation and triggering the autocomplete. What’s even more remarkable is that they can evaluate expressions when the abbreviation is expanded. This is what we’ll use.

Generating the current date

For generating the current date, we can use the date expression provided by PhpStorm and bind it to a variable.

In PhpStorm settings, go to Editor > Live Templates
Expand the PHP section
Click on the Add button, + on the right side, and select Live template
In the abbreviation field, enter $date$ . This is what we’ll type to expand the template.
In the template text field, enter '$date$'$END$.
Click Edit variables
In the line for the variable named date, add the expression date("yyyy-MM-dd’T'HH:mm:ss.SSSz") and select Skip if defined
Click Define below the ⚠️ No applicable contexts
Select PHP > Expression
Save and close the settings with the Ok button.

Generating a random UUID

Generating a UUID is slightly different because PhpStorm doesn’t provide a function to create UUIDs directly, but it, fortunately, allows to run Groovy scripts.

In PhpStorm settings, go to Editor > Live Templates
Expand the PHP section
Click on the Add button, + on the right side, and select Live template
In the abbreviation field, enter $uuid$ . This is what we’ll type to expand the template.
In the template text field, enter '$uuid$'$END$.
Click Edit variables
In the line for the variable named uuid, add the expression groovyScript("UUID.randomUUID().toString()") and select Skip if defined
Click Define below the ⚠️ No applicable contexts
Select PHP > Expression
Save and close the settings with the Ok button.

And voila, we now can quickly insert the current date or a UUID in our code.

Thanks to PhpStorm allowing us to execute Groovy code, we can probably think of other ideas and save more time. For instance, it would probably make sense to generate a random date instead of the current one, as we could be in a particular case if the code has conditions based on the current date.

An alias to easily debug PHP scripts

2021-12-28T00:00:00Z

As I said before, I’m trying to save some time when using the CLI, and I rely more and more on aliases.

Aliases are fantastic to save some keystrokes, and they also significantly help with something else: freeing mental space.

Some commands take a lot of arguments and options, and it’s not always simple to remember all of them. I’ve decided to stop remembering everything and created an abstraction on top of the one I often use, hiding the long list of arguments under an alias or a command. If I still have trouble remembering the alias, I can still rely on my alias to find aliases.

One of the examples of such a command with a long list of arguments is the PHP command when it comes to debugging a PHP script.

On my machine, with PhpStorm, it goes like php -dxdebug.mode=debug -dxdebug.client_host=127.0.0.1 -dxdebug.client_port=9000 -dxdebug.start_with_request=yes script.php.

You probably could have different options on your machine.

I’ve introduced a simple alias, xdebug, to avoid thinking about specifing mode, port and client:

alias xdebug='php -dxdebug.mode=debug \
 -dxdebug.client_host=127.0.0.1
 \ -dxdebug.client_port=9000
 \ -dxdebug.start_with_request=yes'

Now, debugging a script is as easy as xdebug script.php and I don’t have to look for the documentation, or look through my zsh history, to see how I need to construct my command.

Keystrokes and time saved, memory freed!

Composing data using Postgres Foreign Data Wrapper

2022-02-18T00:00:00Z

Say you decided to go the micro-service way, split your database into pieces, and are now requested a feature that needs to display data stored from several of these databases. Fetching the data from several sources and composing it somehow is a totally valid solution. Unfortunately, it starts to be tricky if you need to do some paging, sorting, and filtering based on the data from several of the databases. A solution to this problem could be to denormalize data, create some data store just for that purpose and copy the relevant data there.

A more accessible alternative solution is available if you’re using PostgreSQL: thanks to the Foreign Data Wrapper extension, one Postgres database can query tables from another database as if they were on the same serve.

Displaying race results for my pony club

Let’s imagine that I’m currently contracting for a Pony Club. They’re really into IT and decided to build a system with two microservices. The first one knows about the club’s ponies, the second one stores their results during some competitions. It would be so nice to display our ponies' results alongside some information about them, and why not do some filtering.

First let’s create our two service databases.

version: '3.5'

services:
  postgres_ponies:
    image: postgres
    container_name: ponies_db
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - "5432:5432"

  postgres_race_results:
    image: postgres
    container_name: race_results_db
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - "5433:5432"

This docker compose file will start two Postgres databases, one named ponies_db and the other one race_results_db.

We will then create some tables and add data to the two instances.

Once connected to the ponies_db, with docker exec -ti ponies_db psql -U postgres run the two following queries:

CREATE TABLE IF NOT EXISTS ponies (
   id serial PRIMARY KEY,
   name VARCHAR ( 50 ) UNIQUE NOT NULL,
   status INT NOT NULL DEFAULT 0);

INSERT INTO ponies(name, status) VALUES
 ('Noisette', 1),
 ('Griotte', 2),
 ('Eole', 0);

Here we created a ponies table and inserted three ponies, with an id, a name, and some status stored as an integer.

In the race_results_db, that you can connect to with docker exec -ti race_results psql -U postgres, run:

CREATE TABLE IF NOT EXISTS race_results (
   id serial PRIMARY KEY,
   race VARCHAR ( 50 ) NOT NULL,
   pony INT NOT NULL,
   timing INT NOT NULL);


INSERT INTO race_results(race, pony, timing)
VALUES
    ('Fun fun', 1, 137),
    ('Fun fun', 2, 125),
    ('Fun fun', 3, 133),
    ('Epic party race', 2, 79),
    ('Epic party race', 3, 77);

This just created a race_results table, each row being an id, a race name as a string, a pony - an integer, mapping to the id of a pony in the ponies_db database, and some timing in the second.

Now, let’s create the link between the two databases.

Still in the race_results database install the postgres_fdw extension:

CREATE EXTENSION IF NOT EXISTS postgres_fdw;

Declare a remote server, pointing to the ponies_db server:

CREATE SERVER ponies
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host 'ponies_db', port '5432');

and tell which user to use for the connection:

CREATE USER MAPPING FOR USER
SERVER ponies
OPTIONS (user 'postgres', password 'postgres');

At this point, we can create a remote table, mapping from the ponies’table from the ponies_db` database:

CREATE FOREIGN TABLE ponies (
  id serial,
  name VARCHAR ( 50 ),
  status INT)
SERVER ponies
OPTIONS (schema_name 'public', table_name 'ponies');

And now we can run a query with a join between race results and ponies, using the newly created foreign table:

SELECT p.id, p.name, r.race, r.timing, p.status
FROM race_results r
JOIN ponies p ON r.pony = p.id;

1,Noisette,Fun fun,137,1
2,Griotte,Epic party race,79,2
2,Griotte,Fun fun,125,2
3,Eole,Epic party race,77,0
3,Eole,Fun fun,133,0

But, but, but... the coupling

You might be mumbling something about how nice that’s solution is, but it creates some coupling with the schema of the ponies_db, and you’d be right. If the team working around the ponies_db feels the need to rename a column or change how they handle their statuses, it will break everything.

Fortunately, we can create foreign tables based on views. This is really nice as it means the ponies_db team can have an abstraction layer to hide their schema behind. If at some point they decide to change a part of the schema, as long as they modify the view to ensure not to break the contract, everything will keep running.

Furthermore, having a view helps expose a better API. So far, we don’t know what the integer pony statuses mean. In the new view, we can map each value to a string, increasing the understandability of the API.

Let’s make the changes.

In the ponies_db, we create a public_ponies view based on the ponies table:

CREATE VIEW public_ponies AS
    SELECT
           id,
           name,
           CASE status
               WHEN 0 THEN 'resting'
               WHEN 1 THEN 'available'
               WHEN 2 THEN 'injured'
           END AS status
    FROM ponies;

In the race_results_db we will remove the existing remote table and create a new one based on the view:

DROP FOREIGN TABLE ponies;

CREATE FOREIGN TABLE ponies (
    id serial,
    name VARCHAR ( 50 ),
    status VARCHAR)
SERVER ponies
OPTIONS (schema_name 'public', table_name 'public_ponies');

Now, we can rerun our query with the join:

SELECT p.id, p.name, r.race, r.timing, p.status
FROM race_results r
JOIN ponies p ON r.pony = p.id;

and we will get

1,Noisette,Fun fun,137,available
2,Griotte,Epic party race,79,injured
2,Griotte,Fun fun,125,injured
3,Eole,Epic party race,77,resting
3,Eole,Fun fun,133,resting

Limitations

While this solution allows doing powerful things without much work, it’s not magical. The two databases need to talk to each other and exchange data which means that if they are far away from one the other, the query will take more time than if the data was all on the same server. Depending on your use case, it might be an issue or not. In the case of an asynchronous job, it’s probably not; for a synchronous query, you’ll have to see for yourself. Nevertheless, it would still be a decent solution to have while you’re creating a read database aggregating all the data.

Using Property Based Testing to test priorities

2022-02-28T00:00:00Z

I was recently working on a project with an Angular frontend where we displayed an action button. That button had to be disabled when some conditions were met.

Examples of conditions are:

not the proper access level
not enough "points in the wallet" to do the action
action can’t be performed on the selected item

There is no complex logic to knowing if a condition is triggered. Nevertheless, the service tasked with deciding if a condition was met was growing and was taking on more dependencies.

When the application was in a state where a condition was on, we displayed a tooltip to explain why the button was disabled.

Now, this is where the fun starts. Multiple conditions could be detected at a point in time: the action can’t be performed on the selected item, and you don’t even have enough points anyway, for instance.

We could argue that it would be better to display all messages at once to avoid our users playing the annoying error message game where they fix an error just to discover a new one again and again. Unfortunately for the user, it was decided to display error messages one by one for some reason.

It was unfortunate for us as well, as we now had to decide which message to display when several conditions were met. Obviously, not all states are born equals, and some are more important than others. The logic in the service was now also tasked with ordering the messages.

The code looked something like this:

export class ButtonGuardLogic{
    

    vote() {
        
        switch (true) {
            case !this.authenticationService.isAuthenticated:
                return "You need to be authenticated";
            case this.anotherService.someState === 'not_a_good_state':
                return "You’re not in a good state";
            case this.selectedItem.totalPointsNeeded > this.walletService.availablePoints:
                return "You don’t have enough points in your wallet";
            case this.selectedItem.isSomething:
                return "You can’t do this on an item that is in isSomething state";
            default:
                return '';
        }
    }
}

I changed it slightly to remove everything related to Observables as it’s not the interesting part and to hide some of the business logic. You can still get the idea.

I was pairing with a colleague, planning to add a new message based on a condition that would require taking on a new dependency, and with almost no tests to help us.

As you can imagine, if you want to cover all cases and ensure that the order of messages keeps being respected when multiple conditions are on, you need a lot of tests. If you’re only interested in if the condition is met or not, and not on the data required by each condition to decide if it’s on, we’re talking 2^(number of conditions) test cases. For the code above, 16 test cases are needed. With the new condition we were about to add, it grows to 32.

We wanted to improve the code and agreed that it would be better to have each condition on its own, where we could test it separately and have a sort of glue class, selecting the right message to display, also tested aside.

As we weren’t great Angular devs and were lost with the Observables and the way to test them, we didn’t refactor the code and conformed to the current design. It bothered me, and I tried to see how I could improve it and better split testing conditions and message display.

I’ll pretend we had only three conditions to simplify the examples for the following.

First, let’s extract the conditions:


export type ErrorMessage = string;
export type PositiveMessage = '';

export type GuardMessage = ErrorMessage | PositiveMessage;

export type ButtonGuard = {
    vote(): GuardMessage;
}

export class ButtonGuardA implements ButtonGuard {
    vote(): GuardMessage {
        return 'Error A';
    }
}

export class ButtonGuardB implements ButtonGuard {
    vote(): GuardMessage {
        return 'Error B';
    }
}


export class ButtonGuardC implements ButtonGuard {
    vote(): GuardMessage {
        return 'Error C';
    }
}

Here we defined a ButtonGuard type, containing a vote method that can reply with an ErrorMessage or a PositiveMessage. Each guard, or condition, implements this type, and can get the needed dependencies to decide if it wants to return a message or not. As all guards are separated, they can easily be tested.

Next, we add another type of ButtonGuard, which takes all the other guards, requests their votes, and returns the first ErrorMessage found. As with the switch case in the first design, the display order is still represented: here, it’s the order of the guards we placed in the array.

export class ButtonGuardLogic implements ButtonGuard{

    constructor(
        private readonly guardA: ButtonGuardA,
        private readonly guardB: ButtonGuardB,
        private readonly guardC: ButtonGuardC
    ) {}

    vote() {

        return [this.guardA, this.guardB, this.guardC]
            .map((guard) => guard.vote())
            .find(guardMessage => guardMessage !== '') || '';
    }
}

If no ErrorMessage was returned the ButtonGuardLogic returns with an empty string, the PositiveMessage type.

You may have noticed that this is the Composite design pattern in action.

Arguably, this code is already an improvement. We still need to test that the messages are displayed in the expected order, and it’s time for Property Based Testing to come into play.

PBT frameworks work by trying many combinations of more or less random data on each test run for properties we’ve defined. The canonical example of a property is reversing twice a string should return that same string. A PBT framework will give us a lot of strings to try that property, including some we probably wouldn’t think about trying (empty string, "null", string with non-Latin characters, a string containing numbers, ...).

Here the property we want to test is:

"The displayed error message is the message with the highest priority returned from all guards."

Because we have extracted all conditions in simple guards, we can now easily say which of them are in an error state or not by providing a stubbed implementation. The PBT framework’s job is to tell us which guards are in an error state. It’s equivalent to saying that the PBT framework has to generate a list of boolean values, one for each guard, and we will map these booleans to set up the stubbed implementations.

As we know which guards are in an error state, we also know which error messages are produced and can get the one with the highest priority out of them. Then we can compare that value with the one returned by the implementation.

We need to have a way to control our individual guards. For this, I introduced a FakeButtonGuard:

class FakeButtonGuard implements ButtonGuard {
    voteValue = '';

    vote(): GuardMessage {
        return this.voteValue;
    }
}

We will also need some constants that will come in handy later:


const errorMessageByGuard = {
    A: 'Error message from guard A',
    B: 'Error message from guard B',
    C: 'Error message from guard C',
};

const positiveMessage = '';

let guards: {[guardName: string]: FakeButtonGuard} = {
    A: new FakeButtonGuard(),
    B: new FakeButtonGuard(),
    C: new FakeButtonGuard(),
};

const priorities : {[priority: number]: string} = {
    1: "Error message from guard A",
    2: "Error message from guard B",
    3: "Error message from guard C",
    4: ""
}

const guardCount = Object.keys(guards).length;

For each guard, we define its error message. Here it’s a string, and it could be a constant defined in the guard classes as an improvement.
We also instantiate fake button guards for each guard and list error messages in the expected priority order. guardCount gives us a simple way to get the number of guards.

Before each test, we instantiate a ButtonGuardLogic:


let buttonGuardLogic: ButtonGuardLogic;

beforeEach(() => {

    buttonGuardLogic = new ButtonGuardLogic(
        guards['A'],
        guards['B'],
        guards['C'],
    );

});

We are ready to add our test.

Using Fast-Check, a PBT framework for typescript, we generate an array of three, the number of guards, boolean values.

it('displays error message by order of priority.', () => {

    fc.assert(fc.property(
        fc.array(fc.boolean(), {minLength: guardCount, maxLength: guardCount}),
        (activations: boolean[]) => {

           
        }
    ));

});

Next, we need to define a few functions.

The buildGuardValues function associates the activations of guards with their messages. It produces a dictionary where each guard name (A, B, or C) maps to either an empty string when the boolean value is true or the error message associated with the guard otherwise.

const buildGuardValues = (activations: boolean[]) : GuardMessagesByGuardName => 
  Object.entries(errorMessageByGuard)
    .reduce((guardValues, [guardName, errorMessage], index) => {
        guardValues[guardName] = activations[index] ? positiveMessage : errorMessage;

        return guardValues;
    }, {});

setupGuards function sets each of the FakeGuards with their expected responses:


const setupGuards = (guardValues: GuardMessagesByGuardName) => 
  Object.entries(guardValues)
    .forEach(([guardName, value]) => guards[guardName].voteValue = value);

With these two functions, we can map the generated array of boolean to the list of the guards' messages and initialize each guard with the value it’s expected to return.

Next, we need to go through all the expected messages to find the one with the highest priority. To do this, I introduced the three following functions:


const getPriority = (guardMessage: GuardMessage): number =>
  Object.entries(priorities)
      .find(([_priority, errorMessage]) => errorMessage === guardMessage)
      .map(([priority, _errorMessage]) => parseInt(priority))
      .shift();


const compareGuardMessagePriority = (guardMessage1: GuardMessage, guardMessage2: GuardMessage) : CompareResult => 
  getPriority(guardMessage1) < getPriority(guardMessage2) ? -1 : 1;


const getMessageWithHighestPriority = (guardMessages: GuardMessage[]) : GuardMessage => 
    guardMessages
    .sort(compareGuardMessagePriority)
    .shift();

getPriority gets the priority associated with an error message.

compareGuardMessagePriority compares two messages.

Finally, getMessageWithHighestPriority gives the message with the highest priority using the compareGuardMessagePriority function.

The message returned by getMessageWithHighestPriority is the one we expect given the provided array of booleans.

We can now tie all the pieces together to write the body of the test.

it('displays error message by order of priority.', () => {

    fc.assert(fc.property(
        fc.array(fc.boolean(), {minLength: guardCount, maxLength: guardCount}), (activations: boolean[]) => {

            const guardValues = buildGuardValues(activations);
            setupGuards(guardValues);

            const buttonActivationWithHighestPriority = getMessageWithHighestPriority(Object.values(guardValues));

            const context = buildContext(activations);

            expect(buttonGuardLogic.vote())
                .to.equal(buttonActivationWithHighestPriority);
        }
    ));

});

We now have a test that will cover a lot of permutations of error activations.

When it is time to add a new condition, it will easily be created in its own ButtonGuard; that guard will easily be testable independently of the others. Then we will have to slightly modify the ButtonGuardLogic to add the new guard and its error message and let Fast-Check deal with testing the new permutations.

That's it !

And if you need some help with your test I think I can help. Have a look at my video course in French or let's have a chat and see what we can do together.

Test suite guidelines

2022-03-09T00:00:00Z

Yesterday I watched again the "Breaking up (with) your test suite" talk by Justin Searls, and it resonated with an idea I recently had while working on a project for one of my clients. I even suspect that my idea is actually Justin’s one and that my brain conveniently remembered it when I needed it.

I planned to write about this for a while, and I guess it’s a sign that I should stop postponing.

Backstory

I worked with a colleague on the Angular front-end part of a project, adding a feature, and wanted to write some tests. I have been working on back-end stuff for some time now, and I’m not used to writing tests for components. We went around with my pair partner, looking at all the test files available, and they all looked quite similar and different at the same time. Some tests were exercising the kind of code we were about to implement but in a way that looked strange to us. It wasn’t convincing, but we couldn’t decide ourselves to write the tests differently. After all, we were just two back-end devs on a front-end codebase new to us. Who were we to change the way to test that code? We probably didn’t understand the choices made by previous developers. Anyway, we were paralyzed.

We needed some sort of documentation explaining the rationale behind the different types of tests, a guide to select which one to write, some guidelines about how to write them.

Test suites were organized by directories, each containing some type of test. As expected, some tests weren’t looking like the others in the same directory, but it was a start. We could have a README file in each directory explaining what tests should look like in there. This is the idea I wanted to talk about. It’s relatively simple and can probably avoid losing time, improve team speed, and remove some frustration.

As an alternative, we could use Architecture Decision Record. In my opinion, having the description file closer to the tests is better because it makes it easy to find when needed.

What should we find in the README file

The README file should be beneficial to newcomers, more junior developers, or even the ones who decided the test strategy, reminding us why we made these choices.

I really like Sandi Metz’s description of the system under test as a capsule we can send incoming commands and queries to and which, in turn, can send outgoing queries and commands to something else.

I think the test suite guide should explain how incoming messages are generated:

Are we allowed to call SUT methods directly?
Or should we go through something that exercises the UI and help us pretend we’re a real user acting on a component?
Should we send an HTTP request to an endpoint?
Should we use framework provided tool to pretend we’ve sent an HTTP request?
Should we call a CLI?

The guide should probably tell us what we should do about the other side of the capsule.

Are we allowed to stub dependencies? If so, which ones?
Should we only stub dependencies doing IOs?
Should we allow some IOs? Are database queries allowed but HTTP calls disallowed?
Should a controller test exercise the domain code, or is it sufficient to know that we called some method or fired some command?

You probably can imagine a lot of variations just around these two themes.

Justin Searl came up with other really interesting points we probably should include as well.

The first one is a description of the confidence each type of test should give us. For instance, end-to-end tests verify that all pieces of the software are working well together. Adapter tests let us know if a contract with a dependency is still respected after an upgrade. Consumption tests prove that the behavior we are responsible for works correctly.

Knowing the gained confidence of each test type helps decide which test to write. If we want to be confident about multiple things, maybe we need to use various kinds of tests.

Note that it also drives, or enforces, the architectural choices. Say you want to ensure that an adapter behaves appropriately. We could use an end-to-end to prove that, and it would be costly to write, maintain, probably slower than needed, and could fail for multiple reasons. Once we decide that we gain confidence around adapter behavior using adapter tests, we will need the adapter to be testable on its own. To do this, we will probably add some boundaries around it.

Another idea expressed in Justin’s talk is to know which understanding each type of test gives us.

Here we’re more interested in feedback on the design of our application:

If our end-to-end tests are individually slow, maybe workflows are too complex for our users.
If our consumption tests are hard to write, our service is probably hard to use, and we could improve the interface.
If our contract tests with other teams are often failing, it might be a signal that our priorities aren’t aligned.
Adapter tests give us clues about our usage of third parties, which in turn help us know what we should look for if we want to switch to another provider.

Justin also talks about indicating who the "user" is in each test. Is it a real user, another piece of code, our application talking to a dependency?
It relates closely to the messages each test can send to the capsule.

He also shares some guidelines and warning about each type of test. These could probably be included in the readme file too.

Another valuable idea would be to keep one test always clean for each suite, following the rules and demonstrating the best practices. It would act as an exemplar we could refer to to get a better understanding of the content of the readme. Of course, the readme file links to this test file. This idea comes from the book "Living documentation" by Cyrille Martraire.

In short

The readme file should guide us when deciding which test to write and how to write them.
Some ideas of useful information to include are:

Which confidence are we getting from it
Who is the user
What are the incoming and outgoing messages
What understanding are we getting from the test
An exemplar of what’s considered a good test

If you need some help with your test I think I can help. Have a look at my video course in French or let's have a chat and see what we can do together.

Comment tester une méthode privée ?

2022-09-20T00:00:00Z

Dans cet article quand j’utilise le terme "méthode privée" cela correspond aux méthodes qui ne sont pas accessibles depuis l’interface publique d’une classe. En PHP, par exemple, il peut s’agir de méthodes private ou protected.

On ne la teste pas. Ça, c’est la réponse courte. La réponse longue demande un peu plus d’effort.

Commençons par le commencement.

La question originale est trompeuse. En réalité on ne cherche pas à tester une méthode, quelle soit private, protected ou public, mais un comportement. On veut savoir ce que fait notre logiciel, pas comment il le fait, et donc la manière dont on implémente ce comportement nous importe assez peu.

Évidemment, on va quand même retrouver une méthode publique comme point d’entrée du comportement, mais la méthode privée, elle, est un détail d’implémentation. Le comportement qui nous intéresse pourrait tout aussi bien être servi par une méthode publique qui contient l’ensemble du code nécessaire, ou une méthode publique qui délègue une partie du travail à une ou plusieurs méthodes privées, ou même encore à une ou plusieurs autres méthodes publiques d’autres classes. En somme, l’existence ou non d’une méthode privée ne nous intéresse pas.

Pour tester le comportement, on va donc se placer à l’endroit où l’on peut le déclencher et vérifier son bon déroulement, c’est-à-dire au niveau de la méthode publique. Au passage cela va "tester la méthode privée", ou plus précisément "exercer le code contenu dans la méthode privée". Encore une fois, on ne veut pas tester une méthode privée mais un comportement qui s’appuie sur une méthode privée. On peut dire que l'on "teste" la méthode privée via une méthode publique.

Oui, mais cette méthode privée est complexe

Ceci étant dit, il se peut que le code présent dans la méthode privée soit assez complexe pour que l’on ait envie de pouvoir exercer ce bout de code sans avoir à passer par la méthode publique. On veut ainsi éviter de devoir faire une mise en place de test complexe, d’avoir à zigzaguer au milieu du reste du code pour pouvoir se placer dans un cas particulier du comportement qui nous intéresse.

Une solution couramment proposée est d’essayer de contourner le problème en rendant la méthode privée accessible. Ce qui peut se faire en utilisant le système de réflexion du langage. L’avantage de cette solution est que l’on n’a pas à toucher au code. En revanche on ajoute de la complexité à la mise en place de notre test et l’on augmente le couplage entre les tests et le code. Cela peut alors nuire à un potentiel futur refactoring, puisque les tests ont alors une connaissance assez fine de l’implémentation.

Une autre solution, bien meilleure selon moi, est d’écouter les tests. Comme très souvent, quand quelque chose est compliqué à tester ce ne sont pas les tests le problème mais la conception du code.
En nous montrant qu’un bout du comportement est complexe, si complexe que nous avons envie de pouvoir nous assurer de son bon fonctionnement séparément du reste les tests nous indiquent qu’il nous manque un concept, qu’il manque un collaborateur.

Il faut alors prendre le temps de trouver quel est ce concept, de le définir, de le nommer. On peut ensuite introduire ce nouveau concept dans notre code à l’aide d’une nouvelle classe et y transférer la méthode privée, que l’on rend publique au passage.

Cela nécessite, certes, de modifier le code, mais permet de faire naitre un vocabulaire plus riche pour discuter du problème que l’on cherche à résoudre. On est capable de documenter cette partie du comportement par elle-même au travers des tests.

On n’a pas non plus besoin d’utiliser d’astuces qui complexifient les tests, un simple appel à la nouvelle méthode publique suffit.

On a réussi à transformer notre problème. Puisqu’il n’y a plus de méthode privée on n’a plus besoin d’essayer de tester de méthode privée.

Voilà pourquoi la réponse longue à la question "Comment tester une méthode privée ?" est aussi "On ne la teste pas".

Et si vous voulez améliorer la manière dont vous écrivez des tests je ne peux que vous conseiller ma formation vidéo sur l'amélioration des tests automatisés. Avec cette formation vous saurez comment écrire des tests super lisibles, rapides, et maintenables. Il vous est aussi possible de prendre rendez-vous et voir ensemble comment je peux vous apporter mon aide ou simplement pour discuter.

Le dummy, un indice pas si con à propos de votre design...

2022-09-28T00:00:00Z

Je l’ai déjà évoqué dans le précédent article, je suis persuadé que nos tests nous fournissent des informations pertinentes quant au design de nos applications. Il faut savoir les entendre.

Pour une raison relativement évidente, je me suis retrouvé à expliquer les différences entre les différents types de doublures il y a peu. Cela m’a permis de me rendre compte que je n’utilisais que très rarement, pour ne pas dire jamais, de dummy, et ça m’a interrogé. Une question en entrainant une autre je me suis assez vite demandé dans quel cas un dummy est utile et ce que l’utilisation d’un dummy pouvait indiquer à propos du design du code.

Qu’est-ce qu’un dummy ?

Très rapidement, un dummy est un type de doublure qui sert à remplacer un collaborateur dont on ne va pas avoir besoin lors de l’exécution du test. On s’en sert souvent pour remplacer un collaborateur pénible à construire mais pourtant nécessaire à la construction du système sous test. Un dummy peut aussi servir en tant que paramètre d’une méthode. Dans les deux cas ce collaborateur ne sert pas et on fait généralement en sorte qu’un appel fait sur le dummy provoque une erreur.

Il existe plusieurs manières de créer un dummy, avec ou sans framework de mock, et les possibilités varient en fonction du langage que l’on utilise.

Pour en savoir plus sur les dummies, je vous invite à consulter la page de description de ce pattern sur le site de xUnit Patterns.

Une chose est sûre, les dummies nous parlent, et à nous d’essayer de les comprendre.

Un dummy comme argument pour la construction

Intéressons-nous au premier cas, celui où l’on utilise un dummy pour remplacer une dépendance injectée à la construction.

Pour avoir de quoi discuter prenons pour exemple une classe Cuisinier qui nous permet d’allumer le gaz et de partir en pause.

class Cuisinier {

    public function __construct(
        private readonly Gazinière $gaziniere,
        private readonly Briquet $briquet,
        private readonly PaquetDeCigarettes $paquetDeCigarettes) {
    }
    
    public function allumerLeGaz() {
        
        $this->gaziniere->ouvrirLeRobinet();
        
        $feu = $this->briquet->allumerLeFeu();
        
        $this->gaziniere->approcher($feu);
    }
    
    public function partirEnPause() {
        $cigarette = $this->paquetDeCigarettes->sortirUneCigarette();
        
        $feu = $this->briquet->allumerLeFeu();
        
        $cigarette->allumerAvec($feu);
        
        //...
    }
}

Cette classe a 3 dépendances :

une Gazinière,
un Briquet,
et un PaquetDeCigarettes.

Lorsque l’on va écrire des tests qui s’intéressent au premier comportement, l’allumage du gaz, seuls 2 collaborateurs vont nous êtres utiles, la Gazinière et le Briquet. Pour eux on va pouvoir utiliser de véritables collaborateurs ou une doublure autre qu’un dummy. Ici sans doute un mock. En revanche, le PaquetDeCigarette ne nous intéresse absolument pas pour ce test et utiliser un dummy est une solution tout à fait acceptable.

Pour le second comportement, celui où notre cuisinier part en pause, c’est pour la Gazinière que l’on peut utiliser un dummy. Pas besoin du fourneau pour s’en griller une.

Deux comportements, deux dummies.

La présence de ces deux dummies devrait selon moi nous pousser à nous interroger sur le design de notre application :

Est-ce que le design ne serait pas mieux en séparant les deux responsabilités de cette classe dans deux classes indépendantes : un cuisinier spécialisé dans l’allumage de gaz et un cuisinier dont l’unique utilité est de prendre des pauses ?
Est-ce qu’avoir une classe Cuisinier pour faire l’orchestration de ces actions a vraiment du sens ? Ne devrait-on pas se débarrasser du Cuisinier et créer un concept de gazinière toujours prête à l’emploi, quitte à ce que celle-ci possède son propre Briquet ? Avoir un paquet de cigarettes qui fournit des cigarettes qui peuvent s’allumer sans besoin de taxer du feu ?

Il n’y a pas de réponse évidente à ces questions. Le code environnant, les préférences personnelles, les habitudes de l’équipe, et sans doute bien d’autres facteurs jouent sur les réponses que l’on peut apporter.

Oui mais le briquet !

De manière intéressante la métrique Lack Of Cohesion in Method version 4, ou LCOM4 de cette classe est de 1, ce qui semble indiquer un bon niveau de cohésion. C’est l’utilisation du Briquet dans les deux méthodes qui permet à cette classe d’être perçue comme cohésive.

La présence de dummies nous offre ici un signal que la métrique n’est pas en mesure de donner et nous permet de poser de nouvelles questions :

Est-ce que l’utilisation du Briquet dans les deux méthodes est une justification suffisante pour placer les deux comportements dans la même classe ?
Est-ce qu’on allume vraiment une cigarette comme on allume une gazinière ?
Est-ce que ce Briquet n’est pas uniquement une possible implémentation de SystèmeDAllumageDeGazinière ? D’autres implémentations pouvant être Alumette ou AllumagePiezoElectrique.
Est-ce qu’on peut allumer une cigarette avec un AllumagePiezoElectrique ? (non).
Est-ce qu’avoir deux systèmes d’allumage n’aurait pas plus de sens ? Un pour la cigarette et un pour la gazinière ? Si oui, le LCOM4 de la classe passe à 2, signal qu’une séparation en 2 de la classe pourrait être intéressante.

Doit-on changer ce code ?

Oui, non, peut-être, je ne sais pas. À vous de voir, finalement.

Ce que je trouve intéressant, c’est que la présence d’un dummy nous offre un signal. Il nous pousse à nous interroger sur la répartition des responsabilités entre les différents collaborateurs :

Doit-on splitter certaines classes en plusieurs ?
Doit-on changer la manière dont les classes collaborent entre-elles ?
Doit-on transférer certaines responsabilités d’une classe à une autre ? Ou dans un nouveau concept ?

Encore une fois, je ne pense pas qu’il y ait de réponse toute faite pour ces questions. La forme des tests, comme les résultats de métriques, n’est qu’un outil pour nous pousser à nous poser des questions mais c’est bien à un développeur ou une développeuse de faire l’effort de choisir ce qui lui semble le meilleur design au moment actuel.

Dans ma formation vidéo sur l'amélioration des tests automatisés nous explorons les différents types de doublures. Il vous est aussi possible de prendre rendez-vous et voir ensemble comment je peux vous apporter mon aide ou simplement pour discuter.

Des tests super lisibles

2022-10-10T00:00:00Z

i_Cet article est une traduction et légère adaptation d’un article précédent écrit il y a déjà plusieurs années sur ce même blog. Les techniques exposées ici me sont toujours utiles et je pense qu’elles peuvent aider un grand nombre de personnes à améliorer leurs tests automatisés. Bonne lecture !_

Avec cet article je souhaite partager certaines des techniques que j’utilise régulièrement lorsque j’écris des tests. Ces techniques permettent de fortement améliorer la lisibilité des tests, les rendant plus faciles à comprendre. Certaines d’entre elles ont également un impact positif sur la maintenabilité de l’ensemble des tests. On verra un exemple dans l’article.

D’autres personnes ont déjà partagé certaines de ces idées et cet article, ainsi que ma manière d’aborder les tests, s’appuie en partie sur les travaux de Sandro Mancuso, Matthias Verraes, Dan North, et également sur des idées exposées par Nat Pryce et Steve Freeman dans Growing Oriented Object Software Guided By Tests

Dans cet article, nous allons refactorer le test suivant une étape après l’autre afin de le rendre davantage lisible.
Ici on utilise PhpUnit mais les idées partagées ici peuvent l’être avec d’autres frameworks (certaines des idées sont d’ailleurs inspirées par PhpSpec qui est également un très bon outil) ou même dans d’autres langages que PHP.

<?php
class SendWelcomeEmailToMemberTest extends PHPUnit_Framework_TestCase {

    protected function tearDown() {
        Mockery::close();
    }

    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        $mailSender = Mockery::mock(MailSender::class);

        $welcomeMailSender = new WelcomeMailSender($mailSender);

        $mailSender->shouldReceive('send')
           ->with(Mockery::any(new Email(
           'charles@test.fr',
           'us@chorip.am',
           'Welcome',
           'Hey! Welcome!')
           ))
           ->once();

        $welcomeMailSender->sendTo(
            new Member('Charles', 'charles@test.fr')
        );
    }
}

Nom du test en Snake_case

Comme vous le voyez dans ce test on n’utilise pas la convention de nommage classique en camelCase, testWhatItsSupposedToDo, mais plutôt du snake-case. De plus le nom de la méthode de test débute par it_, ce qui, nous oblige à utiliser l’annotation @test. Démarrer le nom du test avec it_ aide à formuler une phrase qui décrit le comportement que l’on veut tester dans un langage naturel. Il est alors possible d’utiliser la liste de tous les tests pour générer une documentation, ce qui se fait facilement avec l’option testdox de PhpUnit.

Arrange - Act - Assert

Le premier changement est de réorganiser le test pour qu’il suive la forme "Arrange - Act - Assert". Lorsque tous les tests sont sous cette forme il est très simple de savoir quelles sont les préconditions du test - ce que j’appelle la mise en place du petit monde -, l’action, et les assertions, qui sont toujours à la fin du scénario.

Dans la version actuelle, une des assertions, celle à propos de l’envoi de l’email, est faite avant l’étape d’action.

Pour permettre de déplacer l’assertion à la fin du test on a besoin d’utiliser un Spy à la place d’un Mock. Pour en savoir plus sur les différents types de doublures vous pouvez consulter cet article.

<?php
    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        // Arrange    
        // Utilisation d’un spy à la place d’un mock
        $mailSender = Mockery::spy(MailSender::class); 

        $welcomeMailSender = new WelcomeMailSender($mailSender);

        // Act
        $welcomeMailSender->sendTo(
            new Member('Charles', 'charles@test.fr')
        );

        // Assert
        $mailSender->shouldHaveReceived('send')
            ->with(Mockery::any(new Email(
                'charles@test.fr',
                'us@chorip.am',
                'Welcome',
                'Hey! Welcome!')
                ))
            ->once();
    }

Builder

Faisons maintenant un petit effort d’imagination : la classe Member est une des classes centrales de notre application et on la retrouve dans de nombreux tests. Dans ces tests, on utilise le constructeur de Member pour instancier des membres.

Un jour arrive une nouvelle fonctionnalité qui nous pousse à vouloir ajouter la notion de date de naissance pour les membres. On ajoute alors un nouveau paramètre $birthDate au constructeur de la classe Member. Le problème, c’est que cela va faire échouer tous les tests qui font un appel à ce constructeur. En route pour la correction de dizaines de tests. Quelle joie !

Pour éviter ce problème ma solution favorite est d’utiliser un builder qui va encapsuler l’appel au constructeur de Member. Désormais quand on veut effectuer un changement sur le constructeur il n’est plus nécessaire de passer dans chacun des tests. Seul un changement dans la méthode build du builder est nécessaire.

En utilisant un builder pour Member et Email le test est alors le suivant :

<?php

    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        $mailSender = Mockery::spy(MailSender::class);

        $welcomeMailSender = new WelcomeMailSender($mailSender);

        $charles = (new MemberBuilder)
            ->withFirstName('Charles')
            ->withEmailAddress('charles@test.fr')
            ->build();

        $welcomeMailSender->sendTo($charles);

        $welcomeEmail = (new EmailBuilder)
            ->from('us@chorip.am')
            ->to('charles@test.fr')
            ->withSubject('Welcome')
            ->withContent('Hey! Welcome!')
            ->build();

        $mailSender->shouldHaveReceived('send')->with(Mockery::any($welcomeEmail))->once();
    }

Et voici le code du MemberBuilder dans une version simple :

<?php
final class MemberBuilder {
    private $firstname;
    private $emailAddress;
    private $birthDate;

    public function __construct() {
        $faker = \Faker\Factory::create();
        $this->emailAddress = $faker->email;
        $this->firstname = $faker->firstname;
        $this->birthDate = $faker->date;
    }

    public function build() : Member {
        return new Member(
         $this->firstname,
         $this->emailAddress,
         $this->birthDate
         );
    }

    public function withFirstName($firstname) : static {
        $this->firstname = $firstname;

        return $this;
    }

    public function withEmailAddress($emailAddress) : static {
        $this->emailAddress = $emailAddress;

        return $this;
    }

    public function bornOn($birthDate) : static {
        $this->birthDate = $birthDate;

        return $this;
    }
}

En plus d’encapsuler la création d’une instance de Member a un unique endroit l’utilisation d’un builder à un second avantage. Dans le constructeur du builder une valeur par défaut est assignée pour chacune des informations nécessaires à la création d’un membre. Cela nous permet de n’avoir à spécifier que les informations pertinentes pour le test que l’on est en train d’écrire. Si on a besoin d’un Member et que seul son prénom nous intéresse on peut appeler (new MemberBuilder)->withFirstName('John')->build() et omettre de préciser son adresse email et sa date de naissance. Cela permet de réduire le bruit lors de la mise en place du test, ce qui le rend plus facile à comprendre.

De plus ici on utilise Faker, une library qui permet de créer de fausses données de manière aléatoire. En utilisant des données aléatoires on réduit le risque que notre implémentation ne se base sur des valeurs fixes.

Fonction de création pour les builders

Pour faciliter encore davantage la lisibilité du test, il est possible de masquer la création des builders derrières des fonctions.
En introduisant les deux fonctions suivantes :

<?php

function aMember() {
    return new MemberBuilder;
}

function anEmail() {
    return new EmailBuilder;
}

On peut modifier le test sous cette forme :

<?php
    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        $mailSender = Mockery::spy(MailSender::class);

        $welcomeMailSender = new WelcomeMailSender($mailSender);

        $charles = aMember() // Création d’un Member
            ->withFirstName('Charles')
            ->withEmailAddress('charles@test.fr')
            ->build();

        $welcomeMailSender->sendTo($charles);

        $welcomeEmail = anEmail() // Création d’un Email
            ->from('us@chorip.am')
            ->to('charles@test.fr')
            ->withSubject('Welcome')
            ->withContent('Hey! Welcome!')
            ->build();

        $mailSender->shouldHaveReceived('send')
            ->with(Mockery::any($welcomeEmail))->once();
    }

Méthodes d’aide

Pour nous permettre d’employer le vocabulaire du métier, ou a minima nous permettre d’utiliser un langage plus proche du langage naturel, il est aussi possible d’ajouter des méthodes d’aide :

<?php
class SendWelcomeEmailToMemberTest extends PHPUnit_Framework_TestCase {
    private $mailSender;
    private $welcomeMailSender;

    public function setUp() {
        $this->mailSender = Mockery::spy(MailSender::class);
        $this->welcomeMailSender = new WelcomeMailSender($this->mailSender);
    }

    protected function tearDown() {
        Mockery::close();
    }

    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        // it exists a member with first name 'Charles'
        // and with email adress 'charles@test.fr'
        $charles = $this->it_exists(aMember()
            ->withFirstName('Charles')
            ->withEmailAddress('charles@test.fr')
        );

        $this->welcomeMailSender->sendTo($charles);

        // It should send an email from 'us@chorip.am' to 'charles@test.fr'
        // with subject 'Welcome' and with content 'Hey! Welcome!'
        $this->it_should_send(anEmail()
            ->from('us@chorip.am')
            ->to('charles@test.fr')
            ->withSubject('Welcome')
            ->withContent('Hey! Welcome!')
        );
    }

    private function it_exists(MemberBuilder $member) {
        return $member->build();
    }

    private function it_should_send(EmailBuilder $email) {
        $this->mailSender->shouldHaveReceived('send')
            ->with(Mockery::any($email->build()))->once();
    }
}

Le test se lit alors bien plus aisément et des personnes non techniques serait tout à fait en mesure de comprendre le comportement décrit par elles-mêmes. Le test peut désormais servir de documentation aussi bien pour les développeuses et développeurs mais aussi pour le reste de l’équipe.

Aller plus loin avec une extension pour PhpUnit

On pourrait encore améliorer la lisibilité du test en replaçant l’utilisation de $this pour appeler les méthodes que nous venons d’introduire par given() et then(). J’ai tenté une expérimentation qui fonctionne mais n’ai jamais pris le temps de pousser l’idée jusqu’à créer une extension pour PhpUnit qui permette de le faire facilement.

Voilà ce à quoi ressemble le test en appliquant cette idée :

<?php

    /**
     * @test
     */
    public function it_sends_a_welcome_email_to_a_member() {
        $charles = given()->it_exists(aMember()
            ->withFirstName('Charles')
            ->withEmailAddress('charles@test.fr')
        );

        $this->welcomeMailSender->sendTo($charles);

        then()->it_should_send(anEmail()
            ->from('us@chorip.am')
            ->to('charles@test.fr')
            ->withSubject('Welcome')
            ->withContent('Hey! Welcome!')
        );
    }

Conclusion

Toutes ces techniques peuvent être utilisées indépendamment les unes des autres. On peut les introduire pas à pas, en fonction des problèmes que l’on rencontre ou des améliorations que l’on souhaite apporter, et nous permettent d’obtenir des tests très facilement lisibles une fois combinées.

Comme pour chaque outil il y a bien sûr un cout d’introduction qui doit être mis en regard de l’intérêt du projet sur lequel on est en train de travailler. Je dois admettre que de mon côté, j’essaye de mettre en place des builders le plus vite possible. Ils améliorent énormément la lisibilité des tests, facilitent l’écriture des tests et leur maintenabilité au cours du temps.

Quoiqu’il en soit je vous encourage à garder ces techniques en tête pour pouvoir vous en servir en case de besoin et pour vous permettre d’avoir des tests avec lesquels vous preniez plaisir à travailler pendant un bon bout de temps !

Si vous voulez aller encore plus loin, je vous conseille ma formation vidéo sur l'amélioration des tests automatisés dans laquelle nous allons plus en profondeur sur des techniques permettant d'écrire des tests extrèmement lisibles et facilement maintenables. Il nous est aussi possible de nous rencontrer et voir ensemble comment je peux vous apporter mon aide.

Des solutions pour les conflits au niveau de la base de données pour les tests exécutés en parallèle

2022-10-25T00:00:00Z

On m’a demandé il y a quelques jours si j’avais des ressources parlant de solution aux conflits entre tests joués en parallèle lorsqu’ils touchent à la base de données. J’avais quelques pistes à partager et je me suis dit que ça ferait un bon sujet d’article. Le voici donc.

Le problème

Lorsque l’on joue des tests en parallèle on se retrouve face à des problèmes qui n’existaient pas lorsque les tests étaient joués en série, notamment lorsque les tests partagent une ressource, comme une base de données.

Des tests qui, jusque-là, tournaient de manière régulière et avaient le même comportement à chaque fois qu’ils étaient exécutés commencent à avoir des comportements différents d’un run de test à un autre, parfois ils passent, parfois ils échouent. On parle de flaky tests pour décrire les tests qui vont passer ou échouer en fonction de certaines conditions.

Prenons un exemple avec la base de données :

Nous avons deux tests qui ajoutent un utilisateur dans la base de données lorsqu’ils sont lancés.
Bien sûr, le nom de l’utilisateur doit être unique.
Les choses sont bien faites et après chaque test l’utilisateur est supprimé de la base de données, ce qui permet aux différents tests d’utiliser le même nom d’utilisateur.

Ces tests vont fonctionner parfaitement en série. En revanche en parallèle il est possible que l’un d’eux échoue s’il essaye d’ajouter l’utilisateur à la base de données avant que l’autre test n’ait eu le temps de faire le ménage.

Nos deux tests qui étaient, jusque-là, reproductibles deviennent flaky.

Il existe, bien évidemment, plusieurs solutions à ce problème.

Bien préparer les données

Commençons par la fausse bonne idée.

Une solution est de faire en sorte de préparer les jeux de données qui vont être utilisés dans les tests pour éviter les collisions. Cette solution semble simple à première vue mais risque de poser des problèmes de maintenance des tests sur le long terme.

Il va falloir s’assurer que certaines données ne sont utilisées qu’une seule fois et devenir imaginatif pour créer de nouvelles données.

On peut imaginer mettre en place des outils qui permettent de connaitre quelles sont les données déjà utilisées pour ne pas s’en resservir, mais ces outils risquent de devenir complexes assez vite. Il ne s’agit là, à mon avis, que d’une fuite en avant.

On peut également se dire que l’on peut lancer les tests pour savoir si chaque nouveau test n’entre pas en conflit avec un test précédent mais malheureusement il est possible que le conflit ne soit pas détectable immédiatement et le devienne plus tard.

Bref, cette option est donc clairement à déconseiller.

Faire tourner les tests en séries

Cette solution peut sembler décevante mais a le mérite d’exister : Si les tests n’étaient pas flaky lorsqu’ils tournaient en séquence une solution est d’abandonner la parallélisation.

Dans cette solution on échange du temps d’exécution contre davantage de fiabilité. Si le temps d’exécution sans la parallélisation n’est pas trop important il s’agit sans doute de la solution la plus simple.

Mais lorsque l’on met en place de la parallélisation c’est parce que la durée d’exécution des tests était trop importante et que l’on veut aller plus vite.^[1]

Ce qui fait que cette solution risque de ne pas satisfaire tout le monde et nous mène l’option suivante.

Se passer de la base de données

Puisque le problème vient du partage de la base de données entre plusieurs tests une autre solution est de se passer de la base de données pour les tests. On peut imaginer remplacer les différentes dépendances qui s’appuient sur une base de données par des doublures qui vont avoir le même comportement que la dépendance originale mais ne vont pas avoir besoin de partager de ressource.

Autrement dit il est possible d’utiliser un fake par processus, tel qu’un repository en mémoire, qui va stocker les données dans un tableau. Une instance de ce repository sera créée dans chaque processus, ou même mieux pour chaque test, ce qui va éviter les conflits entre les tests.

Mais là encore cette solution a quelques inconvénients, le premier étant que l’on aura toujours besoin de tests avec la base de données pour s’assurer que l’intégration de notre code avec celle-ci se passe comme voulu (via des tests end-2-end, des tests de contrat qui s’assurent que notre doublure a bien le même comportement que la dépendance remplacée, ou autre...). Ces tests peuvent eux aussi potentiellement rencontrer des problèmes lorsqu’ils sont exécutés en parallèle. Retour à la case départ en quelque sorte.

Sauf que tout n’est pas perdu : si l’on a réussi à réduire le nombre de tests qui dépendent de la base de données il y a des chances que notre suite de tests tourne beaucoup plus rapidement et l’on peut alors faire tourner nos tests en séries. L’option précédente devient viable.

Le second inconvénient est la potentielle difficulté de mise en place de cette solution. Transformer l’architecture d’une application pour permettre de pouvoir remplacer certaines dépendances pendant les tests n’est pas forcément facile et peut prendre beaucoup de temps. C’est pourquoi il peut s’agir de la solution la plus compliquée à mettre en œuvre suivant l’état du système.

Utiliser des transactions

Fort heureusement il y a une solution généralement plus rapide à mettre en place : démarrer une transaction avant le début du test et faire un rollback une fois le test terminé, que ce soit un succès ou non. Chaque test est alors exécuté dans son bac à sable, sur la même base de données, et ne peut pas interagir avec les autres.

Dans le monde PHP la plupart des frameworks proposent une solution pour ça, soit nativement, soit via des dépendances à installer^[2]. Il est également possible de faire sa propre implémentation.^[3] Cette solution nécessite en général beaucoup moins de travail pour pouvoir être mise en place que la précédente.

Au passage, cette solution permet d’ailleurs de régler des problèmes d’interactions entre tests également lorsqu’ils sont lancés en série, sans avoir besoin de se préoccuper du nettoyage d’après test.

Évidemment, vous vous en doutez bien, cette solution a, elle aussi, un inconvénient. Pour fonctionner il faut que le test et le système sous test partagent la même connexion à la base de données. Si ce n’est pas le cas le test n’est pas en mesure démarrer et d’annuler la transaction. En fonction de l’architecture de l’application, il peut être compliqué de réussir à mettre la main sur la même connexion aussi bien dans le code de production que dans les tests. De plus, cette contrainte rend cette solution inefficace pour certains types de tests qui vont nécessairement devoir séparer le processus dans lequel tournent les tests du processus dans lequel tourne l’application - les tests end-2-end par exemple.

Heureusement, il reste encore une alternative...

Utiliser une DB par processus

La dernière solution permet de répondre à la contrainte des processus différents entre le test et l’application, et également à toutes situations dans lesquelles on ne peut pas, ou l’on ne veut pas, mettre en place le système de transaction.

Plutôt que d’essayer de faire interagir les tests des différents processus avec la même base de données on va dédier une base de données à chacun.

En fonction du runner de test la mise en place peut-être plus ou moins compliquée.

Certains runners créent un identifiant incrémental pour chaque processus et donnent accès à cet identifiant via une variable d’environnement.^[4]
Il est alors possible d’utiliser la variable d’environnement pour créer la connexion à la base de données dédiée au processus courant.

D’autres runners de test ne semblent pas fournir de variable d’environnement^[5]. Il faut alors générer un identifiant unique dans un hook du runner de test qui va s’exécuter avant le début de tests et l’ajouter aux variables d’environnement. Et enfin, de la même manière, utiliser la variable d’environnement pour se connecter à la base de données.

Reste encore à créer les bases de données. Dans le cas de la variable d’environnement incrémentée il est possible de créer les différentes bases avant le début des tests pour peu que l’on connaisse le nombre de processus qui vont tourner en parallèle. C’est la solution simple.

Alternativement, si l’identifiant est aléatoire ou que le nombre de processus ne peut être connu apriori, il est possible de créer la base de données avant l’exécution des tests, là encore dans un hook. Particulièrement dans le cas d’un identifiant aléatoire, il faut penser à supprimer la base de données qui vient d’être créée à la fin des tests pour éviter de faire une collection de bases inutilisées.^[6]

À noter qu’il est possible que cette solution soit plus rapide à mettre en place que la solution avec les transactions selon l’état du système. Une fois le système de création de bases de données installé il n’y a pas besoin de faire de nombreux changements. Il faut uniquement modifier la création de la connexion pour indiquer la base de données à utiliser.

Dans le cas des tests end-2-end l’idée va rester la même, une base de données par processus de tests, mais la mise en pratique va devoir être un peu différente. Il va falloir permettre aux tests d’indiquer à quelle base de données ils sont rattachés. Cela peut se faire en utilisant plusieurs instances de l’application, chacune connectée à une base de données différentes. Chaque processus de test va alors pouvoir appeler l’instance qui lui est dédiée (ex.: en changeant de nom de domaine ou de port).

Une autre solution, s’il n’est pas possible d’avoir plusieurs instances de l’application, est que le test communique le nom de la base de données à laquelle il souhaite s’adresser (ex.: au travers d’un header) à celle-ci, et qu’en fonction de l’information transmise elle se connecte à la base de données. Là encore cette solution présente des inconvénients, principalement liés à la mise en place de logique dédiée spécialement aux tests pour pouvoir changer de base de données.

Conclusion

Comme souvent, il n’y a pas une seule solution qui va convenir à toutes les situations. Avec cet article, on a vu différentes pistes de solutions, qui peuvent répondre à différents cas d’usage, et qui sont plus ou moins faciles à mettre en place selon l’état de l’application.

Et donc comme toujours, pas de solution miracle, c’est à vous de voir laquelle de ces options s’applique le mieux à votre contexte.

Il existe peut-être d’autres solutions à ce type de problème. Si vous en connaissez n’hésitez pas à me contacter ici ou sur Twitter.

Et si vous voulez faire en sorte d'avoir des tests avec lesquels vous prendrez un réel plaisir à travailler, vous pouvez accéder à ma formation vidéo sur l'amélioration des tests automatisés. Il vous est aussi possible de prendre un rendez-vous pour que nous travaillions ensemble à faire passer votre équipe au niveau supérieur.

Certainement pas pour tester, pour le fun, ou simplement parce que l’outil le permet. Non, non... ↩︎
Par exemple Laravel l’inclut nativement, côté Symfony il faut installer un plug-in pour Doctrine. ↩︎
Voici un exemple d’implémentation en PHP avec PhpUnit et PDO qui n’est pas de toute beauté, mais qui donne une idée de ce qui peut être fait avec une "encapsulation" de la connexion et une extension PhpUnit. ↩︎
C’est le cas de Paratest côté PHP, et de Jest dans le monde JS. ↩︎
xUnit, côté .Net ne semble pas partager de variable d’environnement avec un ID pour le processus. Cette série d’articles donne une solution en C# avec Entity Framework. ↩︎
Un exemple de hook qui permet de créer et de supprimer une base de données avec un nom aléatoire en PHP. ↩︎

Le tags, la fonctionnalité oubliée des lanceurs de tests

2022-11-02T00:00:00Z

Il y a une fonctionnalité des lanceurs de tests que l’on utilise bien trop peu, moi le premier : les tags.

Avec un peu d’imagination ils peuvent pourtant nous rendre de chouettes services, et voir même changer certaines discussions autour des tests et de leur organisation.

C’est quoi les tags ?

Habituellement pour créer des groupes de tests, on se base sur la position du test au milieu de notre ensemble de tests. Les tests sont groupés car ils appartiennent à la même class de tests, à la même méthode describe, au même fichier, à une même hiérarchie de fichiers.

Les tags sont un autre moyen de créer des groupes de tests sans avoir besoin de savoir où ils sont rangés. Certains outils parlent d’ailleurs de groupes plutôt que de tags.

Généralement, les tags se mettent en place via le mécanisme d’annotations du langage dans lequel sont écrits les tests.

Une fois les tags mis en place, on a un nouveau moyen de décider quels tests on souhaite lancer ou non. Nous verrons des exemples d’usages dans la suite de l’article.

Mon lanceur de tests ne gère pas les tags

C’est pas de bol...

Mais il y a de fortes chances que la communauté propose un plug-in qui permet d’avoir un mécanisme similaire^[1].

Si ce n’est pas le cas tout n’est pas perdu. Il y a de grandes chances que le lanceur de tests permette de filtrer les tests en fonction de leur nom^[2]. On peut alors décider d’inclure les tags à la fin du nom des tests et filtrer sur le pattern qui nous intéresse. Ajouter un tag dans le nom du test rajoute du bruit, de l’information qui n’est pas pertinente dans certains cas, et il est important d’avoir une convention qui permet de facilement savoir si une information est un tag ou non. Commencer l’ensemble des tags par un croisillon (#) est sans doute une bonne idée.

Examples d’usage

Voyons quelques idées de ce que nous permettent de faire les tags.

Type de test

La première idée est la plus évidente, si évidente qu’elle peut sembler inutile, et pourtant...

Les tags peuvent permettre d’indiquer le type du test. S’agit-il d’un test unitaire, d’un test d’intégration, d’un test end-2-end, d’un test d’acceptance, ou d’un autre type de test ?

Avec les tags, on peut alors décider de ne lancer que certains types de tests à certains moments.

Cette idée semble inutile puisqu’il y a des chances que vous vous serviez déjà de test suites rangées dans des dossiers différents pour séparer les types de tests, et qu’il vous est déjà possible de décider quels types de test exécuter.

L’inconvénient du tri des types de tests par dossier est qu’elle nous force à maintenir synchronisé une ou plusieurs hiérarchies de fichiers miroirs de celle du code de production. Et avouons-le franchement, la plupart du temps cette synchronisation n’est pas faite. On a alors des hiérarchies de dossiers complètement différentes de tous les côtés.

Grâce aux tags, on peut abandonner cette contrainte et placer tous les tests dans une même hiérarchie, ce qui nous facilite grandement la maintenabilité.

On peut même envisager d’aller plus loin et de mettre les tests directement à côté du code et quand même avoir un moyen de choisir quels types de tests on veut lancer.

Et là, plus de hiérarchies de dossier à maintenir synchronisées et pourtant il est possible de sélectionner facilement quels types de test on souhaite lancer.

Dépendances

La seconde idée est une continuation de la précédente.

Mais d’abord permettez-moi de vous poser quelques questions :

Combien de fois avez-vous eu un débat pour décider si un test est un test unitaire ? Ou plutôt un test d’intégration ? Ou un alors test d’acceptance ? Peut-être plutôt un test end-2-end ?
Combien de temps avez-vous perdu à vous écharper en équipe pour savoir comment classer un test et dans quel dossier le ranger ?

Et si le plus important ce n’est pas le type du test mais ses caractéristiques ?

Pensez-y 2 minutes : pourquoi tient-on tant à connaitre le type d’un test ?

Est-ce que pouvoir séparer les tests trop lents des autres n’y est pas pour beaucoup ?

Si votre manière de travailler s’appuie beaucoup sur les tests, par exemple si vous pratiquez le TDD, vous n’avez surement pas envie d’attendre un long moment avant de savoir si vos modifications ont eu un impact imprévu sur le système. Il y a des chances que vous lanciez fréquemment les tests les plus rapides, comme un premier filet de sécurité, et de temps à autre l’ensemble de tous les tests, même les plus lents, pour avoir un filet de sécurité plus complet.

Alors plutôt que de débattre sur le type des tests demandons-nous pourquoi ils sont lents. Généralement les tests sont lents dès lors que l’on fait des entrées/sorties. Les tests qui touchent à une base de données, à un système de fichier, qui font des appels à des services externes via le réseau sont bien plus lents que les autres tests^[3].

On tient alors notre nouvelle idée de catégorie de tags: des tags qui indiquent les dépendances.

Lorsqu’un test va avoir besoin d’une dépendance particulière on peut l’indiquer avec un tag, comme database, external-call, network, filesystem, ...

Avec ce type de tags on est plus obligé d’essayer de catégoriser les tests précisément, on a simplement à décrire les dépendances dont ils ont besoin, ce qui va nous éviter des débats sans fin.

On peut désormais sélectionner ou exclure certains tests en fonction de ce dont ils dépendent. Lorsque l’on a besoin de ne lancer que les tests les plus rapides il est facile d’exclure les dépendances pénibles.

En bonus, s’il manque une dépendance dans un environnement, par exemple si l’on n’a pas moyen de faire des appels à des services externes parce qu’on est dans le train, on peut facilement exclure tous les tests qui ne passeront de toute façon pas.

Cette idée de sélection par dépendance, qui est ingérable avec le système de fichier, devient possible avec les tags.

Documentation plus précise

Je parle souvent d’utiliser les tests comme documentation, pour les devs, mais aussi pour le reste de l’équipe, voir même pour le "métier".

En PHP, avec PHPUnit, l’option '--testdox' permet de générer de la documentation en texte ou sous forme d’un fichier HTML, sauf que...

Différentes personnes sont intéressées par différents aspects du système. Plutôt que de générer des documents énormes que personne ne va lire il est possible d’ajouter des tags pour chaque sujet d’intérêt.

Avec ces tags on peut maintenant générer une documentation spécifique pour chacun :

une documentation qui parle des fonctionnalités de paiements en filtrant sur le tag feat-payment
une documentation sur le fonctionnement du backoffice avec tool-bo
...

Là encore on peut s’en sortir avec le système de fichier mais il existe aussi des sujets plus transverses, tels que la sécurité (security), la traduction (i18n), ou les règles métiers qui ne concernent que les clients de certains pays (market-es)...

En se servant des tags on peut mettre des marqueurs sur les sujets qui nous intéressent de manière récurrente.

Lien avec le système de tickets

Les tags peuvent également permettre de faire le lien entre un système de gestion de tickets.

Avec le numéro du ticket en tant que tag on peut facilement retrouver un test de non-régression pour un bug bug-3457 ou trouver tous les tests liés à une user story US-89.

Je ne pense pas que ces tags soient très utiles sur le long terme puisqu’après un certain temps savoir qu’un comportement a été créé en lien avec un ticket n’est plus une information très pertinente selon moi, mais ils peuvent toutefois servir à court terme.

Par exemple, l’outil de build pourrait générer une documentation spécifique basée sur les tests pour chaque user story et l’afficher directement dans un commentaire de pull request^[4] ou vérifier qu’un test de non-régression a bien été posé avant de résoudre le bug.

Nomenclature

Comme pour tout outil, en abuser peut avoir de désagréables conséquences.

Pour éviter que l’utilisation des tags ne devienne un immense foutoir il est important de réfléchir à une nomenclature. Pour cela il faut se demander ce que vous pouvez tirer de l’utilisation des tags, que ce soit pour sélectionner les tests à lancer ou pour extraire de l’information à partir de ceux-ci.

Une fois que vous avez défini ce à quoi vont vous servir les tags et identifié des catégories vous pouvez utiliser des préfixes pour facilement identifier les tags.

On a croisé quelques idées dans cet article :

feat-payment
US-89
bug-3457
tool-backoffice
market-es
domain-delivery

Pensez également à faire du nettoyage de temps à autre et à supprimer les tags qui ne vous servent plus - particulièrement si vous utilisez les tags liés au système de gestion de tickets.

Pour cela, laissez-vous aider par votre lanceur de test. S’il permet de filtrer les tests par groupe il a certainement une commande permettant de lister les tags. Par exemple, PHPUnit a une commande --list-groups. Si le lanceur de test ne vous aide pas vous pourrez probablement vous en sortir à coup de grep.

Certains tags auront des durées de vie plus ou moins longues de par leur nature, vos usages changeront sans doute également, et c’est très bien.

J’espère que cet article vous a donné des idées, celles présentées ou d’autres, sur la manière dont vous pouvez incorporer l’usage des tags à votre quotidien pour tirer le meilleur parti de vos tests.

Par exemple, jest-runner-groups si vous utilisez Jest. ↩︎
L’option testnamepatternregex de Jest peut aider. ↩︎
Oserais-je simplement dire que ces tests sont bien plus lents que les tests unitaires ? ↩︎
Si votre workflow utilise des PRs, bien sûr. ↩︎

Coder ses tests sans les mains grâce aux Live Templates de PhpStorm

2022-11-17T00:00:00Z

Je ne tape pas particulièrement vite, en tous clairement pas avec les 5 doigts des 2 mains^[1] mais j’arrive à compenser parce que j’ai trouvé différentes manières d’écrire le même code que quelqu’un qui taperait vite tout en seulement quelques caractères.

Les Live Templates des IDE Jetbrains, et donc de PhpStorm, font clairement partie de mes fonctionnalités préférées.

Dans cet article je vais vous partager certains des Lives Templates que j’utilise lorsque je travaille avec des tests, ce qui devrait vous faire découvrir la puissance de cet outil.

On va commencer par des Live Templates simples et on verra sur la fin ce que l’on peut faire quand on commence à pousser l’outil presque au maximum^[2] de ce qu’il est capable de faire.

Comment récupérer ces Live Templates

Pour chacun des exemples de cet article je vous partage le snippet XML qui le représente, ce qui va vous permettre de facilement le mettre en place dans votre IDE.

Pour cela :

Copiez le snippet XML
Dans PhpStorm, ouvrez les préférences et cherchez Live Templates
Sélectionnez le dossier dans lequel vous voulez ajouter le Live Template. Sans doute Php ou PhpUnit pour ceux de cet article.
Gardez la sélection sur le dossier et collez, avec CTRL + V ou CMD + V
Adaptez-le si besoin !

Création de cas de test

J’utilise plusieurs Live Templates qui me permettent de créer facilement des cas de tests.

Créer un cas de test et commencer à tester

Le premier, @t, pour "test", est celui que j’utilise le plus souvent. Il aide à créer un test avec l’annotation @test, plutôt qu’avec le prefix test. Cela permet d’avoir des noms de tests plus lisibles.

<template name="@t" value="/**&#10;* @test&#10;*/&#10;public function $NAME$(): void {&#10;    $END$&#10;}" description="Add a test function" toReformat="false" toShortenFQNames="true">
  <variable name="NAME" expression="" defaultValue="" alwaysStopAt="true" />
  <context>
    <option name="PHP Class Member" value="true" />
  </context>
</template>

Créer un cas de test pour plus tard

Quand on fait du TDD on a parfois envie de lister des cas de tests en avance. Il y a plusieurs options: noter sur une feuille de papier, mettre des commentaires, ou créer des méthodes tests.

Pour faciliter la création de méthodes de test "pense-bête" j’ai un autre Live Template, @ts, pour "test skipped".

Notez qu’une fois le nom du test écrit, le curseur se place une ligne sous la nouvelle méthode de test pour permettre d’enchainer la prise de note des idées.

<template name="@ts" value="/**&#10;* @test&#10;*/&#10;public function $NAME$(): void {&#10;&#9;$this-&gt;markTestSkipped('Not implemented yet.');&#10;}&#10;&#10; $END$" description="Add a test function" toReformat="true" toShortenFQNames="true">
  <variable name="NAME" expression="" defaultValue="" alwaysStopAt="true" />
  <context>
    <option name="PHP Class Member" value="true" />
  </context>
</template>

Une grosse partie de ce qui fait la réussite d’un Live Template pour gagner du temps est l’endroit où se place le curseur une fois la modification terminée. Quand vous créer un Live Template vous pouvez spécifier la dernière position du curseur avec la variable $END$ .

Lazy Naming des tests

Un dernier exemple de création de méthode de test.

J’ai envie d’expérimenter avec l’idée de Lazy Naming^[3]. En Lazy Naming on ne va nommer les tests qu’une fois qu’ils sont dans leur forme finale, ou que l’on en est assez satisfait pour daigner faire l’effort de les nommer.

Le Live Template @tl, pour "test lazy", rajoute un nouveau test dont le nom est généré en utilisant le numéro de la ligne et le curseur est directement placé dans le corps de la méthode.

<template name="@tl" value="/**&#10;* @test&#10;*/&#10;public function TODO_RENAME_$LINE_NUMBER$(): void {&#10;    $END$&#10;}" description="Add a lazy named test" toReformat="false" toShortenFQNames="true">
  <variable name="LINE_NUMBER" expression="lineNumber()" defaultValue="" alwaysStopAt="false" />
  <context>
    <option name="PHP Class Member" value="true" />
  </context>
</template>

Builders

Passons maintenant aux choses un peu plus sérieuses !

J’utilise énormément de builders dans les tests. Ils facilitent fortement la lisibilité et la maintenabilité des tests.

Le seul problème des builders c’est leur mise en place qui demande un peu d’investissement. C’est souvent la raison pour laquelle même ceux qui connaissent et aiment cette technique remettent à plus tard leur création.

Et plus tard, c’est en général trop tard. Au moment où on commence à souffrir de leur manque il faut repasser sur tout un tas de tests pour les mettre en place...

Mais les Live Templates peuvent nous aider à mettre en place nos builders super rapidement, si rapidement que la question de créer ou ne pas créer un builder ne se pose plus.

Laissez-moi vous présenter la classe Burrito, qui va nous servir de support d’exemple pour la suite.

<?php

final class Burrito
{
    public function __construct(
        private readonly string $burritoName,
        private readonly Salsa $salsa,
        private readonly array $ingredients
    )
    {}
}

Le constructeur de cette classe à 3 paramètres :

$burritoName de type string
$salsa de type Salsa
$ingredients de type array

Création d’une méthode de paramétrage du builder

Dans la classe BurritoBuilder il faut ajouter des méthodes qui vont nous permettre de paramétrer la construction du Burrito.

Dans l’exemple on ajoute deux méthodes de paramétrage en quelques secondes. L’une pour la sauce et l’autre pour le nom du burrito. Ce Live Template fonctionne aussi bien pour un type natif que pour une classe que l’on a créée.

Voilà ce qui se passe :

Je tape "with"
Tabulation
J’indique le type du paramètre. Salsa ou string ici.
Tabulation
Je dois indiquer le nom du paramètre. Ici l’autocomplete m’aide. S’il s’agit d’une classe il propose directement le nom de la classe comme nom de variable. $salsa par exemple.
Le nom me convient, Entrée. Le nom me convient pas, je le change.
Le nom de la propriété à laquelle est assignée la variable prend directement le nom que je viens de choisir
Tabulation
Le curseur se place au niveau de l’assignation de la propriété
Alt + Entrée, le raccourci à tout faire de l’IDE.
Choisir d’ajouter la propriété manquante
Entrée

Et voilà. La propriété est créée, la méthode pour lui assigner une valeur également.

<template name="with" value="public function with$NAME$($PARAMETER_TYPE$ $$$PARAMETER_NAME$): self&#10;{&#10;&#9;$clone = clone $this;&#10;&#9;&#10;&#9;$clone-&gt;$PARAMETER_NAME$$END$ = $$$PARAMETER_NAME$;&#10;&#9;&#10;&#9;return $clone;&#10;}" description="Create a wither" toReformat="false" toShortenFQNames="true">
  <variable name="PARAMETER_TYPE" expression="" defaultValue="" alwaysStopAt="true" />
  <variable name="PARAMETER_NAME" expression="complete()" defaultValue="" alwaysStopAt="true" />
  <variable name="NAME" expression="" defaultValue="capitalize(PARAMETER_NAME)" alwaysStopAt="false" />
  <context>
    <option name="PHP Class Member" value="true" />
  </context>
</template>

Mais ce n’est pas fini.

Création de la méthode `build`

Qui dit builder dit méthode build. Toutes les méthodes build se ressemblent. Au moins au début.

On construit un objet en lui passant en paramètre les propriétés du builder et on le retourne.

Il est également possible d’utiliser un Live Template pour là aussi gagner du temps. En réalité il s’agit même de deux Live Templates qui se combinent bien ensemble.

Le premier Live Template, build, créer la méthode build, place le curseur au niveau du type de retour, l’autocomplétion est lancée et nous aide à choisir le type.

Automatiquement le constructeur du type est appelé avec le mot clé new.

On fait une tabulation et le curseur se place dans la liste des paramètres. L’autocomplétion est démarrée et nous propose la bonne propriété à utiliser pour le premier paramètre.

C’est là que le second LT entre jeu. Le constructeur de Burrito prend plusieurs paramètres et il est possible de les ajouter très rapidement.

Ce second Live Template est bindé sur ,.

Oui, oui, la virgule.

, + Tabulation démarre automatiquement l’autocomplétion, qui se charge de proposer la bonne propriété pour le paramètre.^[4]

Le snippet du Live Template build:

<template name="build" value="public function build(): $TYPE$&#10;{&#10;    return new $TYPE$($PARAMETERS$$END$);&#10;}" description="Build method" toReformat="false" toShortenFQNames="true">
  <variable name="TYPE" expression="complete()" defaultValue="" alwaysStopAt="true" />
  <variable name="PARAMETERS" expression="completeSmart()" defaultValue="" alwaysStopAt="true" />
  <context>
    <option name="PHP Class Member" value="true" />
  </context>
</template>

et celui de ,:

<template name="," value=", $COMPLETE$" description=", and auto complete" toReformat="false" toShortenFQNames="true">
  <variable name="COMPLETE" expression="completeSmart()" defaultValue="" alwaysStopAt="true" />
  <context>
    <option name="PHP Expression" value="true" />
    <option name="PHP Statement" value="true" />
  </context>
</template>

Conclusion

J’espère que ces quelques exemples d’utilisation des Live Templates vous auront donné envie de créer les vôtres. Si vous cherchez de l’inspiration je vous invite à lire cet article de Marijn Huizendveld^[5] dans lequel il partage ces Live Templates pour créer des Command Handlers, des Values Objects, des collections...

Devenez fainéants !

Et si vous pensez qu'un accompagnement technique pourrait être bénéfique pour votre équipe, rencontrons-nous et trouvons ensemble comment vous faire passer au niveau supérieur.

Si vous voulez prêcher la bonne parole de la fainéantise auprès de vos amis cet article est aussi disponible sous forme de thread Twitter^[6].

Les live templates sont une de mes features préférées de @phpstorm.

Ça me fait gagner énormémement de temps. Faut dire que je tape pas bien vite.

Comme je parle beaucoup de tests, petit thread d'illustration des live templates qui me servent quand j'écris des tests ! 👇 pic.twitter.com/VeZkjJGcg3
— Charles Desneuf - @selrahcd@piaille.fr (@Selrahcd) November 17, 2022

Ça doit avoisiner 4 doigts sur les deux mains. ↩︎
Il est possible de faire tourner des scripts externes, mais je n’ai pas encore eu l’occasion de tester. J’ai quelques idées cela dit. ↩︎
Je tiens cette idée de Lazy Naming de Romeu Moura qu’il présente dans cette conférence ↩︎
Le Live Template bindé sur la virgule est une nouvelle idée et est en cours d’expérimentation. Je ne serais pas étonné d’avoir des mauvaises surprises un de ces jours 😅. Vous êtes prévenus. ↩︎
C’est cet article qui m’a fait connaitre la méthode facile de partage des Live Templates. Merci Marijn. ↩︎
Tant que Twitter existe, tout du moins. ↩︎

Facilitation Nightmares: How to escape a workshop not going on as planned

2023-01-11T00:00:00Z

A few days ago, I hosted a session about Facilitation Nightmares during a Virtual DDD gathering organised as an open forum.

The idea was for the participants to share some of their stories of facilitation going astray and to explain how they remedied the situation or to see if other participants would have propositions on what they could have done.

As a group, we discussed different situations. Here are the three I enjoyed the most and wanted to keep with me because they all had unusual solutions.

Let’s start with you, the facilitator, being excluded.

The "You know nothing, John Snow" Situation

Diana Montalion described the "You know nothing, John Snow" situation. She explained that during some engagements, she was facilitating architectural sessions. I don’t remember if she mentioned what they were doing, but I can imagine it was an activity we love in the DDD community but can feel strange for other people, such as Event Storming. Unfortunately, some people in the room wanted to avoid engaging in the workshop, claiming it wasn’t architecture. Who was she to make them lose their time that way?

Diana shared that the feeling she had was that they wanted to do things their way, as they usually did, and that the message they had was that she didn’t have anything to bring to the table. Later in the discussion, we rephrased the situation as the "You don’t belong here" situation.

Together, with the group, we discussed some ideas on how to avoid that situation.

We pointed out the usual stuff about facilitation. At the beginning of the workshop, share why we are running it and make sure everyone agrees on the goals. Make it really obvious and easy to see, so you can get back to them during the session if needed. You can use a "Purpose slide" or write them on a paper board.

Also, establish the goal of the meeting beforehand. Let people know before they agree to come what we will discuss. It would be best if you made the invitation explicit, that way, only interested people will show up.

If people find the meeting is not for them after all, allow them to leave. Explain the Law Of Two Feet at the start of the workshop: if you’re neither learning nor contributing, you can use your two feet to move to something more interesting to you.

Another idea is to let people know you at the beginning or even before the session. If people know who you are and what you’ve done, it adds to your credibility. Show that you have the right to be here and some experiences to share. Someone introducing Diana as a long-time architect and frequent speaker could have mitigated the situation.

Here you are using your rank, demonstrating that you are a player in the game and know things to mitigate the situation.

Doing your own introduction can be a bit awkward, and having allies before the meeting can significantly help. Find some people who know you and your past work and who will make the introduction. If needed, they will remind hard-to-manage participants that you have something to bring to the discussion too.

While Diana faced people being vocal about not wanting to participate in the workshop, sometimes, as facilitators, we met more quiet people, even non-talkative ones.

People not willing to speak

How can we cope with people remaining silent during the workshop?

First, as facilitators, we must remember that some people are quieter than others. Not all of us are comfortable with the potential mess of the workshops.

One proposed solution is to make yourself available after the workshop. Sometimes people can’t speak during the session: it’s not a good time for them, and the environment is inappropriate.

You have multiple solutions to make yourself available. The obvious one is to share some way to contact you afterwards, like your email address.

Alberto Brandolini proposed an idea that resonated with me: never leave the room on time. Hang around in the room after the end of the workshop. Take your time to tidy the room. Some people might stay and help you. As people are living, you might have the chance to hear some voices you haven’t heard until now.

Alberto shared a story of one participant who didn’t talk during a challenging workshop, staying silent in the corner of the room. Once the workshop ended and the other participants had left, he went to Alberto and explained what was happening during the workshop. He described the company situation, the power games, and the toxic relationships between participants. Suddenly, everything happening during the session became clearer.

Another idea we didn’t discuss that day, but I’m pretty sure I’ve heard Alberto talk about before, is to have a drink with participants after the workshop. Some people might feel more comfortable in a different environment and start to speak. You might get extra information helping to do your job.

An example of the environment not being as safe as it should be for the workshop to go smoothly is when some participants have more power than others. For instance, when the bosses participate.

The boss derailing the workshop

Having the bosses around can make the discussions more difficult than it should be. People might stop talking, look at the boss for approval for everything they say or worst, you could encounter a boss who decides to "show them how things are done" because "he knows better" and "it will be faster if he explains it". In that case, you lose the collaborative part of the workshop and get only one point of view. And more, you get the point of view of someone who is probably not doing the actual job and has some blind spots.

While having the boss in the room can give you a different perspective, it’s not always an ideal situation.

To avoid the boss capturing the mic and not letting other people speak, a solution is to round-robin the timer. Everyone can talk for some amount of time, and once the time is over, we’re moving to the next person. This idea obviously works with anyone, big chief or not, because, let’s be honest, bosses don’t have the monopoly of talking too much.

Another excellent idea proposed by Tobias Goeschel is to occupy the boss. This is easier said than done because you must be at least two facilitators. One remains working with the group, and the other creates a new activity in another part of the room. For this, you can use a paper board and ask him to explain his ideas and vision of the world. As long as you make him feel important, you’ll be able to keep working with the rest of the group.

Alberto reminded us there is no such thing as a boss showing up late to a workshop to be a spectator. At some point, they will want to speak, and change things the group has already discussed. Even if they remain silent, just by being there, they will change the environment, and the room’s dynamic might be different: some people will talk more, others will speak less, discussions will move to another topic, ...

In this blog post, I focused on the three situations we talked about that night I most relate to. We had other exciting discussions, and we only covered some of the ideas we wanted to discuss. Have a look at the board and see if some of the situations and remediation ideas are interesting to you for your facilitation practice.

I think it would be valuable to gather more facilitation stories. First, they can be fun to hear, and also, because we can learn from them. If you like that idea and have stories to share, hit me up, and let’s see what we can do!

And maybe you are looking for a facilitator, that would be a fanstatic opportunity to meet and see what we can do together!

Approval testing: 3 technics to deal with randomness

2023-01-17T00:00:00Z

Cet article est disponible en français.

When dealing with legacy code, randomness is not something you enjoy meeting.

Creating a test harness on top of a system with random results can be pretty challenging.

Obviously, you’d like your tests to be deterministic and to always give you the same result, provided you didn’t change anything in the system.

Let’s see three ideas that can help you deal with the randomness in the system. The first two technics are based on the idea of removing the random part of the system during tests.

Block the random generator

The first solution I wanted to offer in this article is to ensure that the random generator always returns the same results when called. The answer here is to seed the generator with a value you control, to make the generator work in a deterministic manner.

For instance, in PHP, you can call the mt_rand function. Later, any call to the rand function will return the same random numbers in the same order.

This is a fast way to turn the non-deterministic system into something that always gives the same results.

Unfortunately, this solution, while being one of the quickest ways to get the job done, only works in some situations. First, not all languages allow seeding the random number generator. If you can’t, you can’t and need another solution.

Another caveat of this solution is that if you modify the structure of your code and the order of calls to a random method is changed; you’ll get results in the same order but not in the same logical space. For instance, let’s take two calls to rand and assign the result of the first call to variable A and the second call to variable B. After seeding the random number generator, A and B are always set to the same value, say 34 and 1879. During a refactoring, if we decide to invert the order assignation, we have a problem. The code now assigns values to B before A; A is now 1879, and B is 39. You can play with this idea here.

What can we do when seeding the random number generator is impossible, or we might want to change the order of the calls at some point?

Control the randomness

The other alternative is to take back control of the values generation. As often, adding a layer of abstraction is one of the solutions available in our toolbox. Here we want to break the direct dependency to the random value generator and swap it with a dependency we can manipulate.

In more concrete terms, instead of directly calling a function from the language, we encapsulate it, create a stub we can manipulate, and find a way to substitute the original implementation with our double.

The substitution technics available greatly vary depending on the programming language you’re using and the code’s state. If you’re lucky enough to be able to inject dependencies at construction, this is an easy task. If you’re not, some patterns from Working Effectively With Legacy Code by Michael Feathers can prove helpful.

This solution is available in every language. The only requirement is to feel comfortable messing around with code while not having tests to be able to take control back.

Once you have control, you can decide the values generated.

That solution allows us to solve the ordering issue mentioned before. If you invert the order of calls in the code, you can change the configuration of your stub to change the order and keep the tests passing.

Modifying the stub after tests are created needs to be done carefully. If you want greater confidence and want to avoid touching the tests, you have another alternative. Instead of making one abstraction for every call to the random generator, you can create one for each call. Taking back the example from the first part, you can introduce an abstraction for generating A and an abstraction for generating B. Having two implementations gives you more granular control, and you can create two stubs. When you invert the call order, the code still calls the right abstractions and the same stubs and obtains the same final result.

Touching an existing code base to introduce indirections to control randomly generated value is not always straightforward nor necessary. Sometimes we don’t care about the generated value, and we can be totally fine with keeping the randomness as long as we manage to keep our tests deterministic. And this is the point of the third idea.

Hide the randomness

Sometimes the random value is not really important for the result. If we don’t care about it, we can remove it from what is asserted in the test.

The trick here is that you don’t need to make assertions based on the direct outputs of the system under test. Instead, you can get them and transform them to keep only the information you need to be confident enough that you’re not breaking anything.

When doing approval testing, it is common to use a printer, a function or a class that will transform the outputs to make them look easy to understand. If a piece of information is not relevant, the printer’s role is to remove it and create a clean output. You can then use that clean output in the assertion.

What could be simpler than removing what you don’t care about?

Sometimes you don’t care about the actual value, but you want to ensure that a value is here. Instead of removing the random part, you can replace it with something else. Say you don’t care about the value of a UUID but want to be sure that it is displayed. Using a regular expression, you can replace all UUIDs with something else, like a UUID string.

Another common use case is to detect that a value is repeated in several places in the output.

Let’s continue with the previous example. You still don’t care about the value of the UUID, but you want to assert that the UUID used in a link to access a product is the product’s UUID, not some other one.

Here the printer has to be more clever.

When a UUID is detected using a regex, the scrubber generates a new string, usually made from a base string and an increment (UUID_1). Then it replaces all the subsequent occurrences of that UUID with the generated string, increments the counter and repeats for each UUID found in the output. That way, even if you can’t predetermine the result of the system, you can turn it into something deterministic that you can use as your golden master.

The good thing about this solution is that it doesn’t require touching the legacy system. You don’t need to worry about breaking something while creating the tests. This is why it is a very convenient and relatively rapid method to build confidence. Also, you might be very lucky and work with a language where approval testing tools come with scrubbers doing part of the cleaning work for you.

Discovering a system using randomness and needing to create a test harness doesn’t have to be scary or take an awful lot of time. With the three ideas exposed in this article, you should be able to attack that piece of legacy code you avoided refactoring for a while.

If you feel like you could use a hand to gain control over a legacy codebase, let's chat and see how I can help.

3 hacks for a magical code kata in PHP

2023-02-07T00:00:00Z

When creating content for people to learn from, having a story as a backbone can be a good idea. In my improving tests class, I’ve constructed most chapters with a story. To make a story more powerful, sometimes you need to add some magic. In this article, I’ll show you how I’ve created dishonest code to make a production-like environment with painful library upgrades, slow tests that aren’t really that slow, and lying code.

Use these ideas for more common use cases

The techniques I share here are useful for storytelling purposes but are, more than anything, ideas you can apply with legacy code or code organization. For each technic, I share more conventional usages.

Some background

This article is quite long, feel free to skip if you’re only interested in the 3 hacks!

First, let me describe a typical chapter in my testing class. In the beginning, we have some tests. They’re not always terribly bad, but they have room for improvement. They serve as a demonstration of a problem we can see in a lot of test suites. Usually, the next part of the story is about making a change, either in the tests or in the code, and seeing how these tests are getting in the way of that change. Then, we introduce an idea and demonstrate how to change our tests to remove that roadblock. It’s time to try again to make the change or make another related change and see how easier it is now. That story form should show students what these ideas can help them with when they struggle with their tests.

One chapter of the course is about the "Don’t mock what you don’t own" advice. I’ve used the same story for years, and I like it. When I created the course, I chose to use a Burrito restaurant as the backstory for all examples, and I was lucky enough to reuse my favourite story one more time.

That story is about cooking something using an oven. In that case, we’re cooking some peppers and toasting some bread. AKEI, a well-known Swedish company, manufactured our oven, a Kulinarisk. We directly use the Kulinarisk in our own code. At some point, a new version of the Kulinarisk oven is published and, obviously, breaks our code because AKEI finally decided that publishing code in Swedish was a disservice to its customers. In the V2 of the Kulinarisk library, method names are in English, which causes a BC break.

I like this example because it shows a few different things:

When you use external code, you must conform to its vocabulary. In the test class, we have some code in French mixed with some Swedish words. Creating an abstraction on top of the external code gives you the power to select what terms you want to use in your code. You are free to make the vocabulary used in the code match the concept you are manipulating. No more "laga" when you’re supposed to talk about "baking".
Using the library code in many places in the code and the tests shows how painful it can be when a library makes a BC break. We need to make multiple changes in our code and our tests. Touching at the tests is clearly not the best way to make a refactoring with confidence.

I’ve recently decided to create a kata version of this chapter. From what you’ve read, you already know that I needed to have at least two versions of the Kulinarisk library. I also want the learners to discuss the need to use a double instead of the real Kulinarisk. As often, an excellent reason for using a double in place of the real thing is that the real thing is slow. In my story, baking peppers to get nice and soft peppers takes 25 minutes. Clearly, that’s a bit too long to use in a kata. You don’t want people to stare at their screen for 25 minutes while waiting for the test runner to tell them if the test passed or not.

Could we have our tests be slower when we’re using the real Kulinarisk, to demonstrate that using a double is a good idea, but not to slow to avoid wasting everyone’s time? Yes, we can, and we can even display that the test took 25 minutes to run, even if it actually took a few seconds. These are special effects applied to deliberate practice!

Let’s see how I’ve recreated an almost real environment for that kata.

Using an almost external library and upgrading it

As mentioned before, we need multiple library versions, and students should be able to upgrade to the next version using composer, the PHP package manager.

The first trick here is to use a built-in functionality of composer. Instead of hosting the package online, I kept it inside the repo. The kata comes with all the akei/kulinarisk library versions.

With composer, you can register custom repositories for packages. One of these custom repository types, path, allows you to map a local directory to a package at a specific version.

The composer.json of the kata repositories points to all the necessary versions of akei/kulinarisk. They are stored in the same repository as the kata code.


{
  //composer.json
  // ...
  "repositories": [
    {
      "type": "path",
      "url": "./libs/Kulinarisk_V1.0",
      "canonical": false,
      "options": {
        "versions": {
          "akei/kulinarisk": "1.0"
        }
      }
    },
    {
      "type": "path",
      "url": "./libs/Kulinarisk_V2.0",
      "canonical": false,
      "options": {
        "versions": {
          "akei/kulinarisk": "2.0"
        }
      }
    }
  ]
}

Participants install the project and run composer install when the kata starts. They get akei/kulinarisk:1.0. Later, they can run a composer require akei/kulinarisk to upgrade to the V2 and feel the pain of having code coupled with vendor code.

Keeping libraries in the same repository

Aside from creating katas, this feature is useful with the modular monolith and mono repo approach. Maybe some modules could use some shared code. Instead of creating a library on another repository, pushing it to packagist (PHP libraries repository), and trying to keep the library private, you could make your life simpler and keep that library inside the same repository as your application code.

The second magic trick I use in the kata allows me to hide some code.

The fake `sleep` method trick

At some point, I hope that participants will look at the Kulinarisk code to understand why it’s so slow. They will notice that some maths is involved in transforming a duration in minutes into seconds, followed by a call to the sleep function.


namespace AKEI;

class Kulinarisk {

        public function laga($maträtt, int $varaktighet)
        {
            $varaktighetISekunder = $varaktighet * 60;
    
            sleep($varaktighetISekunder);
    
            return $this->värme($maträtt, $varaktighetISekunder);
        }
    }

As said before, in the story we’re supposed to wait for 25 minutes before getting our warm peppers, but as a facilitator, I don’t want to see people waiting that long each time they run their tests. Here we need to replace a function everyone knows with something else but make it look like we are calling the original one.

The trick is to monkey patch with namespaces. Alongside the file containing the Kulinarisk class, I have another file declaring a sleep function in the same namespace as the Kulinarisk class.


namespace AKEI 

function sleep($duration)
{
    TimeLogger::setExecutionTime($duration);

    if($duration > 5 * 60) {
        \sleep(intdiv($duration,300));
    }
    else {
        \sleep(intdiv($duration, 60));
    }

}

Because they are in the same namespace, the fake sleep function takes precedence over the original one. Hidden in that fake sleep function, we can add some code to make the tests run slowly but not as slow as the initial sleep implementation.
Note that we’re calling the original sleep function from the global namespace thanks to the \ prefix.

Monkey patching

Using a namespace to replace language’s functions can be helpful when creating a test harness on top of legacy code.

You can quickly introduce a double without touching the code with that technic. This is sometimes called monkey patching and is an example of the Link Substitution pattern described by Michael Feather’s "Working effectively with legacy code".

As you can see above in the code of the fake sleep function, we are passing the expected real sleep time to the TimeLogger::setExecutionTime method. It’s the first step for our next trick.

Lying about the test execution time

So far, we’ve seen how to have a consistent story, saying that we will bake something for 25 minutes, calling the sleep method like we’re going to wait for 25 minutes, but wait only a few seconds, to give a feeling of slowness. To make our story consistent, we need to ask: If our tests are calling a method that is expected to run in 25 minutes, how long should they take? 25 minutes, right? How is it that the test output shows that it only took seconds to execute? That doesn’t make for a compelling story. Of course, we can do something and push our story telling a little further!

In the end, we would like our test report to display that the test took 25 minutes like so:

Look at the TimeLogger class, which is responsible for storing each theoretical sleep duration. That class keeps the value in a static field when we call the setExecutionTime static method. TimeLogger also offers a static method to access the stored value and a static method to reset the value.

final class TimeLogger {

    private static ?int $executionTime = null;

    static public function resetTimer() {
        static::$executionTime = null;
    }

    static public function getExecutionTime() {
        return static::$executionTime;
    }

    static public function setExecutionTime(int $executionTime):void {
        static::$executionTime = $executionTime;
    }
}

We’ve seen where the value is stored. Let’s now discuss where we need to read and reset the value.

So, where? Well, directly inside the test framework. Yes, that’s really hacky, but it does the job.

Inside the TestResult::run method from PHPUnit, which runs every test, we add an if statement. If the TimeLogger contains a non-null value, we use that value instead of the value recorded by the PHPUnit timer. Also, we reset the recorded value to null before running each test.

Now, when a test exercise the Kulinarisk::laga method, which in turn calls our fake sleep function, a value is recorded, and we use that value for the duration time display. If a test doesn’t exercise the Kulinarisk::laga method, the recorded value stays null, and we keep the real test duration time.

We found a way to display the information we want. One little problem still needs to be solved. The two modifications we’ve made are on vendors' files, which aren’t versioned in the repository. That means that when participants get the repo and run composer install to get the dependencies they don’t have that dirty hack.

Patching dependencies

It’s now time for the final interesting trick: modify the PhpUnit code when the library is installed.

To do this, we use cweagans/composer-patches, a package that applies patches when we install another package.

We need to create a patch file containing the difference between the code from the repo and the code we would like to have. I’ve cloned the PHPUnit repository, made the change I needed, ran git diff, stored the result in a file in the kata repository.

diff --git a/src/Framework/TestResult.php b/src/Framework/TestResult.php
index 4fde29fac..6be437905 100644
--- a/src/Framework/TestResult.php
+++ b/src/Framework/TestResult.php
@@ -37,6 +37,8 @@
 use SebastianBergmann\ResourceOperations\ResourceOperations;
 use SebastianBergmann\Timer\Timer;
 use Throwable;
+use AKEI\TimeLogger;
+
 
 /**
  * @internal This class is not covered by the backward compatibility promise for PHPUnit
@@ -693,6 +695,8 @@ function_exists('xdebug_start_function_monitor');
             xdebug_start_function_monitor(ResourceOperations::getFunctions());
         }
 
+
+        TimeLogger::resetTimer();
         $timer = new Timer;
         $timer->start();
 
@@ -772,7 +776,8 @@ function_exists('xdebug_start_function_monitor');
             $error = true;
         }
 
-        $time = $timer->stop()->asSeconds();
+
+        $time = TimeLogger::getExecutionTime() ?? $timer->stop()->asSeconds();
 
         $test->addToAssertionCount(Assert::getCount());

I also configured cweagans/composer-patches to apply the patch file every time Phpunit is installed in the vendor directory like so:

{
  // composer.json
  // ...
  "extra": {
    "patches": {
      "phpunit/phpunit": [
        "./patches/patch.diff"
      ]
    }
  }
}

Patching dependencies

Patching dependencies can be necessary when you have some legacy code.

Maybe you’re blocked with an old language version and would like to use a library that could be available to you if you changed a few things or your code relies on a bug that is now fixed in the library.

With a patch, you can keep your code running and use vendor code without needing to maintain your own fork and see it diverge from the original library.

After applying all these tricks, kata participants can clone the kata repository, run composer install, get the patched version of PHPUnit, the V1 of akei/kulinarisk, a fake sleep method they don’t see, tests pretending to run in 25 minutes even if they actually run in a few seconds. This is magic!

Even if creating special effects for kata is fun, the interest of this article is in sharing technics and tools you can apply in your daily life. The monkey patching technic and the patching package are helpful if you’re dealing with legacy code. Keep in mind that it’s not because you can use these ideas that you should stop improving your code. The composer loading library from a local path can help if you use a mono-repo approach and remove the pain of working across many repositories.

I hope you enjoyed that story!

Anyway, if you have trouble dealing with a legacy codebase and think that you could use some help let's have a chat and see what we can do together to help you out of that situation.

Display a message when you enter a directory

2023-02-20T00:00:00Z

A few days ago, I was doing something I do from time to time with a CLI tool. The command requires a bunch of parameters, and I usually go through my command history to find the previous usage, change only the part that needs to be changed and run the command. That’s efficient, and that works, but every time I do this, I think to myself: "I should have something simpler, just an easy command that takes only one argument; you should create a bash script."

Last time I finally decided to create that batch script. I opened a new file in my text editor, copied and pasted the command from my bash history, and looked one more time at how to use arguments in a bash script and what is the proper environment-based shebang. Relatively easy. I tried to save my new script, "download" would be a good name for that command, so I entered "download" in the file name input and tried to save.

Error.

Another file already existed with that name. I opened the file and compared it to my new file; they looked identical. It turns out I already automatized my task away and made it simpler but always failed to remember that I did it. This means every time I used the original script, I was losing time and feeling bad about not taking care of it despite having already dealt with the issue.

I already have that script but keep forgetting about it. What could be the next step to avoid the issue and fix the problem once and for all?

I usually entered that directory only to run that one task^[1]. Is it possible to be reminded that I already have a script and can use it directly without needing to go through the bash history?

It turns out you can.

The solution is to alias the cd command to a custom command that enters the directory, looks for a file, and displays its content. That’s a dead simple solution to my problem.

Here is the code I took from StackOverflow for the new command and alias creation that you should place in your alias file.

reminder_cd() {
    builtin cd "$@" && { [ ! -f .cd-reminder ] || cat .cd-reminder 1>&2; }
}

alias cd=reminder_cd

After that, I added a .cd-reminder file in my project directory with a message explaining that I could use the download command instead of searching and editing past commands.

As a freelancer, I can think of a few usages for that tool. Because I go from project to project, I constantly need to remember which toolkit is used or what command I need to start a project. Now, I can place the information I usually need when I enter the directory in the .cd-reminder file, and boom.

By the way, it’s a good idea to add the .cd-reminder in a global .gititgnore.

I think we can find other usages of that tool for teams. From the top of my mind, I think the quick how-to can work here as well - even if teams in a company should invest in standardizing their tooling to reduce cognitive load. I also believe it could be used to display some messages about the project. Just another way to broadcast information to team members.

It’s still in the experiment phase on my side, but let me know if you find that idea useful and what you’re using it for.

No, I can’t automate it further and run it automatically as I can’t possibly know the parameter before. ↩︎

Quickly jump to periodic file in Obsidian

2023-05-13T00:00:00Z

I'm currently working with a client to unstuck a team blocked by a legacy codebase and complicated work processes. I send a weekly report of things we've done and things I think we should work on to make help us move forward to a few people in the company.

I'm using Obsidian as my note-taking application, trying to build some sort of a second brain, and I, of course, write my reports there, to keep track of them.

Working with reports was a bit annoying because as I have a report per week, I have a new file name every week made from the date of my last day working with a client, and I had trouble quickly accessing it to add new information. I could use the week number for the filename, but I'm the kind of person who keeps track of time in weeks, which means I never know what the current week is.

I wanted to find a way to open the current week's report quickly, and I did an experiment using Templater, an Obsidian plugin, and it works. Even better, the first time I access the weekly report and it doesn't exist yet a new report is created in the correct directory with my basic weekly report template.

I know that other plugins help to manage recurrent notes, such as Periodic Notes. Still, I'm already using the weekly periodic note system for something else, and it would lose all the fun of the experiment.

The template

The idea is to use a Templater template, which comes with a few interesting functions, to make everything work. When I'm somewhere in the application, I can use a shortcut to introduce the template, the template is processed, and the logic is executed. In case the report doesn't exist, it creates it, and in any case, takes me to it.

Here is the template with some comments

<%*
// Generate the file name. I'm not working on Friday's with that client
// so I end my week on Thursdays => start of week + 3 days
const fileName = moment()
    .startOf('Week')
    .add(3, 'd')
    .format('[W]WW-YYYY-MM-DD');

// Where Weekly reports are stored
const weeklyReportDirectory = '🦺 Projects/MY-CLIENT/Weekly reports';
const fullFileName = weeklyReportDirectory + fileName;

// Try to access the file
const file = tp.file.find_tfile(fileName)

if(file !== null) {
	// The file exists, open it
	return await app.workspace.getLeaf(false).openFile(file);
}

// The file doesn't exist
// Get the report template
const template = tp.file.
    find_tfile('🦺 Projects/MY-CLIENT/Templates/Weekly Report.md');
// Create a new file based on the template
const newReport = await tp.file.create_new(template, fileName, true);
const folder = app.vault.getAbstractFileByPath(weeklyReportDirectory);
await tp.file.create_new(template, fileName, true, folder);
%>

Making it work from everywhere

Unfortunately, Obsidian's template system can't append a template when no file is opened. This is clearly a minor issue as I have almost always one file opened, but I wanted to access the report quickly in any situation. Also, even if the solution is based on a template, using the template shortcut to select a template to go to another file felt weird. In my opinion, it would be better to have that quick access right in the Obsidian main action menu.

The solution to my problem is to use the QuickAdd plugin.
With QuickAdd, you can capture elements to a page, format these elements using a template, and run the capture macro from the main menu.

Here is the recipe:

In QuickAdd, create a capture macro
Select Capture to active file
Select Capture format
Copy-Paste the template created before in the text area
In Quick Add main menu, click the lightning strike icon to add a command to the main menu

Now, I'm able to open the Obsidian main action menu, select the Quick Add command and access the weekly report.

I don't need to think about the date and try to guess the filename to access my weekly reports. A little brain power saved each time!

Causal Loop Diagram Asymmetrical Level Pattern

2023-06-06T00:00:00Z

I participated in a workshop on System Thinking hosted by Kent Beck (Yep, that Kent Beck) and Jessica Kerr (Yep, the real Jessitron) during NCraft this year.

We draw a lot of causal loop diagrams all together and in sub-groups.

In some diagrams we drew, I started spotting a pattern I found strange, something bothering my appetite for symmetry.

---
className: diagram-25
---
strict digraph { 
  A -> B
  B -> C
  A -> C
}

From time to time, we drew diagrams indicating that :

more of A leads to more of B, and more of B leads to more of C, and
more of A leads to more of C

What felt strange to me is that these diagrams described in two ways that C influences A. One directly and another indirectly, via B.

It looks like path 1, A -> B -> C, and path 2, A -> C, are not on the same level of abstraction. Why do you, on one side, care about that intermediate B step and not on the other?

Seeing that pattern pushes me toward two options, Zooming Out and Zooming In.

Zoom Out

The first option is that the diagram still conveys enough information for our discussion if we remove B. We are then left with a simpler diagram, showing that more of A leads to more of C. We can avoid the clutter and acknowledge that B wasn’t that relevant to the story we’re telling. We don’t need that level of detail, and we can zoom out.

---
className: diagram-25
---
 
strict digraph { 
  A -> C
}

But that’s not always that easy. Maybe B is interesting for our story, and we really don’t want to remove it from the picture.

Zoom In

The other option is to start asking questions and try to zoom in. That imbalance prompts us to look for more details. How does A influence C in other ways? What parts of this story are we missing?

---
className: diagram-25
---

strict digraph { 
  A -> B
  B -> C
  A -> D
  D -> C
}

After a few questions, we might discover that A influences C via D, or even via even more complex relationships and have a different look at our system. By noticing that asymmetry, we might gain a deeper knowledge of the problem we are dealing with, and have greater discussions.

Zoom In and Zoom Out later

Choosing between Zooming Out and Zooming In is not an obvious choice.

Why did we include B in the first place? It must have made sense for our discussion and understanding of the system. Before getting rid of it, I feel like we should give a chance to the discovery process and try to Zoom In first. After completing the diagram with the missing parts, we can decide that B and the new details we’ve uncovered are not worth keeping, and finally, Zoom out.

Drawing as many diagrams as needed and not being afraid to throw them away, to try different forms and see what works for your current discussion, was one the first pieces of advice Jessica and Kent shared with us.

It’s about the direction...

Zooming out is only possible if in both paths, A -> C and A -> B -> C, A influences C in the same direction.

more of A leads to less of C, but
more of A leads to more of B, and more of B leads to more of C (so more of A leads to more of C)

---
className: diagram-25
---

strict digraph { 
  A -> B
  B -> C
  A -> C [label="-"]
}

more of A leads to more of C, and
more of A leads to more of B, but more of B leads to less of C (so more of A leads to less of C)^[1]

---
className: diagram-25
---

strict digraph { 
  A -> B
  B -> C [label="-"]
  A -> C 
}

You definitely should avoid the Zooming Out option and keep that part of the diagram as it is. There is something interesting in the story here. Zooming In could probably be worthwhile to explain how more of A can lead to more of C and to less of C at the same time. You might find an interesting intervention point in your system if you are trying to increase (or decrease) C.

I find it really interesting how noticing a visual pattern of one direct and one indirect path can push us to clarify our drawing or to ask more questions. It is similar to our reflexes while looking at code, noticing patterns that are good opportunities for code improvements or should lead to more inquiry. Being able to pattern match quickly is one of the things you gain with experience (so yay me, more system thinking experience!) and also greatly helps when you are helping others. You quickly notice something and can start asking questions to facilitate the discussion. Maybe that pattern I noticed will be useful to others!

The same idea applies to more of A leads to less of B, but less of B leads to more of C. ↩︎

Remove all services arguments from Symfony config with a script

2023-07-05T00:00:00Z

I'm helping a team migrate from Symfony 3.4 to a newer version of the framework. One of our objectives is to move away from using the container to get access to services and use dependency injection instead. Our first step was enabling auto wiring, which should simplify the rest of the work because the framework will automagically inject every new service we add to the constructor directly into the service—no need to edit any configuration.

The next step we plan is to use Rector to transform every call to Container::get to dependency injection.

Rector will save us tremendous time by refactoring the code but will not touch the configuration. In the configuration, when a service definition declares the service arguments, it takes precedence over the auto wiring. We need to clean all the configuration files by removing the arguments declaration.

I decided to see if I could use awk to script that and avoid the manual work requested on many different config files, and I managed to do so in less time than expected. Practicing some awk katas was worth it, after all.

Here is the script. It looks for arguments: and sets a variable indicating that we are in an argument block. Once in that block, we look for the first line not starting with - to say we are out of the argument block. The script prints every line outside arguments blocks, when ARGUMENT_BLOCK == 0.

BEGIN {  
    ARGUMENT_BLOCK == 0  
}  
  
ARGUMENT_BLOCK == 1 && !/[[:blank:]]*-/ {  
    ARGUMENT_BLOCK = 0  
}  
  
/arguments:/ {  
    ARGUMENT_BLOCK = 1  
}  
  
ARGUMENT_BLOCK == 0 {print $0}

Here is an example of a diff once the script was used on a services.yaml file. Note that the script deals with the single and multiline YAML array format.

@@ -2,29 +2,12 @@
     sdk.entity.list:
         class: XXX\ListsRepository
         factory: [ '@doctrine.orm.default_entity_manager', 'getRepository' ]
-        arguments:
-            - 'SDKListBundle:Lists'
     sdk.service.list:
         class: XXX\ListBundle\Services\Lists
-        arguments:
-            - '@service_container'
-            - '@event_dispatcher'
-            - '@sdk.entity.list'
 
     sdk.entity.list_access_set:
         class: XXX\Repository\ListAccessSetRepository
         factory: [ '@doctrine.orm.default_entity_manager', 'getRepository' ]
-        arguments: ['SDKListBundle:ListAccessSet']
 
     sdk.service.list_access_set:
           class: XXX\Services\ListAccessSet
-          arguments:
-              - '@sdk.entity.list_access_set'
-              - '@sdk.entity.mapping_list_access_set'
-              - '@logger'
\ No newline at end of file

To run the script on a single file you can use awk -i inplace -f script file.

And if you have a lot of services.yaml files to edit at once, you can use a combination of find, xargs and awk:
find ./src -name services.yml | xargs awk -i inplace -f script.

That step is now automated; we don't need to edit hundreds or more service definitions manually. A few minutes of work and a lot of time saved.

Expected, actual, and super fast assertion creation

2023-07-23T00:00:00Z

If you follow this blog^[1] you know that one thing I enjoy is improving my workflow inside my IDE. The feeling of productivity of being able to code something in seconds instead of minutes brings me joy and lets me keep my focus on other things. I recently had a new improvements idea for test creation^[2].

I think that a great idea to use in tests is to prefix variables with a special role and have a convention for it. My personal preference is to prefix the variable containing the expected result with ‘expected’ and the one containing the value you want to check with ‘actual’.

I decided to create two custom postfix templates that, for some value, create a variable with the wanted prefix and trigger auto-completion to generate the end of the variable name.

The flow goes like this:

Create a value: new Pony(‘Eole’)
Type .ev
Get $expected<cursor> = new Pony(‘Eole’);
Let the autocompletion guide you. Here it will typically suggest something interesting based on the value, like Pony in our example. If you’re dissatisfied with the suggestion, type something else.
Tab ⇥
Done.

See it in action here:

I let you imagine how it works with the av postfix template to get an actual prefixed variable.

It took me a few minutes to get a variable name in camelCase because the auto-completion returns a capitalized name. Thankfully, you can use GroovyScript inside templates, which makes it easy to lower the generated string.

You can avoid the work of recreating those two postfix templates by copying the following snippet in the plugin configuration file.

.ev : Create an expected variable
	ANY -> \$expected$var:capitalize(groovyScript("if(_1) { _1.substring(1) } else { _1 }", phpSuggestVariableName()))$ = $expr$;$END$

.av : Create an actual variable
	ANY -> \$actual$var:capitalize(groovyScript("if(_1) { _1.substring(1) } else { _1 }", phpSuggestVariableName()))$ = $expr$;$END$

But that’s just the first part of the improvement.

Now that I had a convenient way to generate my expected and actual value, it unlocked a new idea: let’s generate assertions with the variable already there.

For this, I created a new live template, asseq which adds the code for an equality assertion, $this->assertEquals($expected, $actual); and triggers autocomplete after the two prefilled parameters. Getting an assert equal in a test is now as easy as typing asseq, Tab ⇥, Tab ⇥.

See it here how fast it goes :

And here is the live template snippet to add to your configuration if you want to use that trick to.

<template name="asseq" value="$$this-&gt;assertEquals($$expected$VAR$, $$actual$VAR2$);" description="" toReformat="true" toShortenFQNames="true">
  <variable name="VAR" expression="complete()" defaultValue="" alwaysStopAt="true" />
  <variable name="VAR2" expression="complete()" defaultValue="" alwaysStopAt="true" />
  <context>
    <option name="PHP Statement" value="true" />
  </context>
</template>

As a bonus, here is the snippet for a second live template that allows you to select what assertion you want because you sometimes need something other than an equality assertion. It does the same thing as the previous one but adds an extra stop and autocomplete step to select the assertion.

<template name="assert" value="$$this-&gt;assert$ASSERT$$END$($$expected$VAR$, $$actual$VAR2$);" description="" toReformat="true" toShortenFQNames="true">
  <variable name="VAR" expression="complete()" defaultValue="" alwaysStopAt="true" />
  <variable name="VAR2" expression="complete()" defaultValue="" alwaysStopAt="true" />
  <variable name="ASSERT" expression="complete()" defaultValue="" alwaysStopAt="true" />
  <context>
    <option name="PHP Statement" value="true" />
  </context>
</template>

I hope this blog post will give you other ideas on how to use live templates and postfix templates to improve you workflow and bring more joy to your work life.

As you can see, I enjoy teaching about testing or help dev teams being more productive and working in a safer way. If you feel like your team could enjoy imrpving the way it works let's have a chat and see what we can do together.

You don’t, let’s be honest, no one does 😅 ↩︎
If you followed that blog, you would know that I spend a lot of time speaking about testing. If that’s one of you’re areas of interest, go there. ↩︎

PipDecks Iphone Widget

2023-08-23T00:00:00Z

I started using Readwise a few months ago. The app allows you to highlight articles or books you read and aims to help memory retention by resurfacing content regularly. One way to do this is by displaying a widget on your phone screen. I installed the Readwise widget on my iPhone and found the idea to be brilliant. It could be part of the solution for me reading articles on the same topic repeatedly.

I wondered what other content I could display on widgets. My first choice was to randomly display a note from my Obsidian vault. I decided to try it using Scriptable, an iOS automation app that uses Javascript. However, I needed markdown parsing and I had no idea how to easily accomplish it. This led me to another idea.

I had purchased some PipDecks (this is an affiliate link) in the past, and very recently with the addition of the new Strategy Deck, it gave me an idea. Would it be possible to display a random card in a widget? The goal here is the same as with Readwise: resurface content, remember it, and make new connections.

Using Scriptable, I created, with the help of ChatGPT, a script displaying the front of a card, and when clicked, it would take you to the back of the card image inside the iPhone’s file manager. I encountered a problem where my widget had an ugly white background, but after some googling, I discovered that Maxwell Zeryck created a base Scriptable script to create a transparent background widget. After merging my script with Maxwell’s code, voilà!

Here’s how the widget looks and works:

How to Install

If you want to install this widget on your phone, you can download the script. The code may not be perfect, but it gets the job done.

Create a directory where it makes sense to you in the File app on your iPhone.
Download your PipDecks cards from PipDecks Vault.
Copy some PipDecks cards into that directory. You need both the front and back of all cards, and all cards must be directly within that directory. You must remove every intermediate directory (see bonus below).
Install Scriptable, if you haven’t already.
In the Scriptable configuration, create a bookmark named "PipDecks" pointing to the directory containing the cards. Don’t change that name, it’s hardcoded in the script.
Create a new Scriptable script. Pick a name you like. "PipDecks Card Widget" is a good one.
Copy my script and paste it inside the Scriptable editor and save.
Create a screen capture of your iPhone on an empty Home Screen Page. We will use this to generate the widget background.
Go back to Scriptable, run the script. It will ask you to provide the screen capture you just took, and to select if you want to place your widget at the top or the bottom of the screen.
Add a new widget on one of your Home Screen Pages and select the script you’ve just created.
Enjoy!

Bonus: Installing Decks with a Directory per Card

When you download cards from PipDecks Vaults, you’ll notice that some decks are organized differently from others. Some have all cards in the same directory (which is what we want) and others have a sub-directory for each card. Instead of undergoing the manual and tedious task of going inside each card directory to copy the front and back images to the bookmarked "PipDecks" directory, I created an iPhone shortcut which prompts for an origin directory, loops through all cards directories, and moves the images to the bookmarked directory. It’s much easier.

If you prefer a simpler approach, you can download the shortcut.

I hope you’ll enjoy the widget and that it will help you remember all the great ideas from your PipDecks.

How test frameworks work: A basic mental model

2023-09-13T00:00:00Z

Understanding how your test framework works is a good start to avoid making mistakes. In this post, I share my basic mental model of how most test frameworks work. This is not 100% accurate, but it is sufficient to understand a few things about writing better tests.

In the following posts, we will continue growing that mental model and see why some features of testing frameworks exist.

A loop over functions...

You can view most test frameworks as a loop over functions. Because it’s a loop, some tests are run before others. That’s not a super impressive statement, but it has some consequences we’ll see later. For now, remember that we’ve got something that loops and that it’s the first component of our test framework.

In pseudo-code, here is the main loop for a test framework:

tests = [
	test1() {},
	test2() {},
	test3() {},
];

foreach(tests as test) {
	test()
}

... throwing exceptions...

The second part of the test framework is assertions. We use assertions to validate that everything went as expected while running the code.
To communicate if the result was ok or not, assertions use exceptions. If something is okay, assertions do nothing; if something goes wrong, they throw an exception.

Here is some pseudo-code for the superstar assertEqual assertion :

assertEqual($a, $b) {
	if($a != $b) {
		throw TestAssertionException("$a and $b are not equal")
	}
}

You get a green bar if no assertions throw an exception inside the loop, meaning that your system behaved according to your tests. Hurray!

The basics are as simple as that: a loop over functions that may throw exceptions.

But keeping it like this would not provide a great developer experience because the loop would stop at the first exception thrown. You can only know how many of your tests are failing by fixing them individually and each time relaunching your test runner to discover if you still have errors or have just fixed the last problem.

... and a try/catch.

To solve that problem, the test framework encapsulates the test function inside a try/catch block. Now, when something doesn’t go as expected, the assertion still throws an exception, which bubbles up the test function and is caught by the try/catch block inside the loop.

There, the exception is stored in a list of exceptions caught for every test, and that list will later be used to display all error messages.

Even if one or more tests fail, all tests are run. Also, the test framework is now able to display one error for each of the failing tests.

The pseudo-code for the enhanced test framework is as follows. We’ve added a try/catch encapsulating the test execution and a second loop for exception display.

tests = [
	test1() {},
	test2() {},
	test3() {},
];

failedTestExceptions = []

// Test loop
foreach(tests as test) {
	try {
		test()
	} catch(exception) {
		failedTestExceptions[] = exception
	}
}

// Error display
if(count(failedTestExceptions) === 0) {
	displayGreenBar()
}
else {
	foreach(failedTestExceptions as exception) {
		displayErrorFor(exception)
	}
}

And now, we have an understanding of how basic test frameworks work. Obviously, real test frameworks are more complex than that simple mental model, and they offer more functionalities.

In the following posts, we will see why following the Arrange-Act-Assert pattern is impossible when we verify that the system under tests throws an exceptionopen . We will also understand why the tearDown functionality is more important than the setup one and discuss the "one assertion per test" rule from a technical point of view.

As you can see, I enjoy teaching about testing. If you are looking to improve your test suites I think I can help. Have a look at my video course in French or let’s have a chat and see what we can do together.

How test frameworks work: Expecting that an exception is thrown and the Arrange-Act-Assert pattern.

2023-10-15T00:00:00Z

In the previous article, we built a basic mental model of how test frameworks work. Now, it’s time to expand on that model to understand why it’s not possible to follow the Arrange-Act-Assert test arrangement pattern for tests that expect an exception to be thrown.

Let’s look at the code of our simple test framework in the state we left it previously.

tests = [
	test1() {},
	test2() {},
	test3() {},
];

failedTestExceptions = []

// Test loop
foreach(tests as test) {
	try {
		test()
	} catch(exception) {
		failedTestExceptions[] = exception
	}
}

// Error display
if(count(failedTestExceptions) === 0) {
	displayGreenBar()
}
else {
	foreach(failedTestExceptions as exception) {
		displayErrorFor(exception)
	}
}

Let’s throw some exceptions.

Imagine now that you want to assert that the system under test throws an exception under certain conditions. We can add a new test to the test list, which throws an exception to mimic the fact that the system under test throws one, like so:

test4() {

	throws MySuperException
	
}

and add it to the test list

tests = [
	test1() {},
	test2() {},
	test3() {},
	test4() {

		throws MySuperException
	
	}
];

When the test framework executes the test, the exception is thrown. This is the behavior we expect. The exception is caught by the test framework and added to the collection of failedTestException. Even if the system under test behaved as expected and correctly threw an exception, the test is marked as failing.

OK, so we need to tell the framework that we expect this to happen.

We need to find a way to tell the test framework that we expect an exception so that he can avoid considering the test as failing, and we need to do this before the exception is thrown. Otherwise, the test will fail. We can achieve this by storing the expected exception in a variable. Now, when the SUT throws the exception, we can compare it with the expected one, if set, and avoid storing the exception as an evidence of a failing test.

tests = [
	test1() {},
	test2() {},
	test3() {},
	test4() {
		expectedException = MySuperException

		throws MySuperException
	}
];

failedTestExceptions = []

// Test loop

foreach(tests as test) {
	expectedException = null
	try {
		test()
	} catch(exception) {
		if(exception != expectedException) {
			failedTestExceptions[] = exception
		}
	}
}

// Error display
if(count(failedTestExceptions) === 0) {
	displayGreenBar()
}
else {
	foreach(failedTestExceptions as exception) {
		displayErrorFor(exception)
	}
}

Note that expectedException variable is reset to null before every test to ensure that we will stop tracking that that exception is thrown for the following tests to be run. Without that part, the error would be masked if MySuperException is thrown by the SUT incorrectly in a subsequent test. Also, we are storing information about what exception is expected and not only that some exception is expected because we want to avoid masking failure if an exception that is not MySuperException is thrown.

As you can see, we have to tell the framework that an exception is thrown before the moment it will be thrown because it wouldn’t be able to compare them otherwise and decide what to do. This explains why you, unfortunately, have to stay away from the AAA pattern when testing that the SUT throws an exception.

The other important information you get with that mental of model of test frameworks expect exception is that you have to be super suspicious when you see a test where the expected exception is declared after the Act part of the test. The test is probably not testing what you think and will likely never fail because the exception goes missing ^[1].

And what if the expected exception is not thrown after all?

So far, our basic test framework doesn’t fail when the expected exception is thrown. That’s only the first half of the problem. Suppose now that the exception is not thrown. That would mean that our SUT doesn’t behave as expected, and this is something we would like to know.

We need to complement our test framework so that it can tell us that an expected exception wasn’t thrown. To achieve this, we add a continue statement in the try/catch block. If an exception is thrown, expected or not, we deal with it in the catch block and move on to the next test—nothing more to do here.

If no exception was thrown, we must ensure we weren’t expecting one. After the try/catch block, a place we should not reach if an exception was thrown, thanks to the continue instruction, we can check if an exception was expected. If we expected an exception, if expectedException variable is not null, we add to the list of errors that we never received the expected exception.

The pseudo-code looks like this:

tests = [
	test1() {},
	test2() {},
	test3() {},
	test4() {
		expectedException = MySuperException

		throws MySuperException
	
	}
];

failedTestExceptions = []

// Test loop

foreach(tests as test) {
	expectedException = null
	try {
		test()
	} catch(exception) {
		if(exception != expectedException) {
			failedTestExceptions[] = exception
		}
		continue;
	}
	// Did we get our expected exception?
	if(expectedException) {
		failedTestExceptions[] = NeverReceivedExpectedException(expectedException)
	}
}

// Error display
if(count(failedTestExceptions) === 0) {
	displayGreenBar()
}
else {
	foreach(failedTestExceptions as exception) {
		displayErrorFor(exception)
	}
}

Our mental model of the inner workings of test frameworks continues growing as we add more features to it. We now understand how choosing exceptions as the communication mechanism to tell that an assertion failed blocks us from using the Arrange-Act-Assert pattern when we want to assert that an exception is thrown.

That suite of articles on building a mental model about test frameworks will continue. In the next one, we will dive into the necessity of the tearDown method.

And was probably written after the code. ↩︎

Change the problem to write simpler tests.

2023-11-15T00:00:00Z

Sometimes, writing tests for a piece of code is difficult but can be simplified by changing the problem we are trying to solve.

A very good example of this I know is about testing something time-related. Controlling time is something we cannot do. No matter what we try, time passes. We could use tools to control the system's clock and see where it goes, but we sometimes can find a simpler solution just by having another perspective about the problem we're trying to solve.

Imagine that you're working on a food inventory system that can tell if a dish is edible right now.

For instance, Panna Cottas can be eaten two days after they've been cooked. One possible way to implement that logic is the following code:

class PannaCotta {

	public function __construct(
		private readonly DateTimeImmutable $cookingDate
	)
	{}

	public function isEdible(): bool {
		$now = new DateTimeImmutable('now');

		$twoDaysAfterCookingTime = $this->cookingDate->modify('+ 2 days');

		return $now <= $twoDaysAfterCookingTime;
	}
}

That code is pretty straightforward, but the dependency on time with the call to new DateTimeImmutable('now') makes it difficult to test.

A test that ensures that a Panna Cotta cooked less than two days ago is edible will start failing at some point (in two days max, actually) if we hard-code the value of the cooking date. We could write our test to use a relative date, but they make tests harder to understand and can cause flakiness. That's not a satisfying solution either.

A better solution is to look for a solution to control the date used for the comparison, and this is where changing the problem can help us to write more straightforward test. Instead of trying to know if a panna cotta is edible right now, we can change the problem to ask if a panna cotta is edible at some date. After all, you might want to know if the panna cotta you have will still be edible tomorrow and, if so, choose something else for dessert right now.

Once we decide to move to that new problem, we can modify the PannaCotta class code to this:

class PannaCotta {

	public function __construct(
		private readonly DateTimeImmutable $cookingDate
	)
	{}

	public function isEdibleOn(DateTimeImmutable $potentialEatingDate): bool {
		$twoDaysAfterCookingTime = $this->cookingDate->modify('+ 2 days');

		return $potentialEatingDate <= $twoDaysAfterCookingTime;
	}
}

We've renamed the isEdible method to isEdibleOn and have introduced a DateTimeImmutable parameter representing the date we might eat the panna cotta, $potentialEatingDate.

Testing that code is now a piece of cake, because we control the cooking date and the date at which we might eat the panna cotta:

class PannaCottaTest {

	/**
	 * @test
	 */
	public function is_edible_up_to_two_days_after_cooking(
		DateTimeImmutable $potentialEatingDate
	): bool {
		
		$pannaCotta = new PannaCotta(new DateTimeImmutable('2017-05-18'));

		$this->assertTrue(
			$pannaCotta->isEdibleOn(new DateTimeImmutable('20127-05-19')
		);
	}
}

As you can see, we didn't have to use some fancy tooling to create the test. Just a change of perspective and reframing the problem.

Dan North wrote an article about rethinking the problem we're trying to solve and discovering a missing concept in the domain you might be interested in as well. Funnily enough, it's also about time.

Forcing us to think harder about the problem is a really lovely side-effect of testing. When a test is difficult to write, seems odd to read, forces us to instantiate things we are not going to use anyway - something we can detect when we see a dummy (an article by me about that, in French)- it's a good clue that we should take a few minutes to rethink the problem or our design.

If you need some help with your test I think I can help. Have a look at my video course in French or let's have a chat and see what we can do together.

Creating databases when starting the Postgres Docker container

2023-11-21T00:00:00Z

One of the first things we usually ask people to do when they set up a project is to create the needed databases. While we often directly use the default database for development to avoid that problem, it remains that we would like at least one other database for testing purposes. Using the same database for development and testing is a nightmare of data inconsistencies.

The PostgreSQL Docker container allows a nice trick to avoid any manual steps. It’s entry point, the script that is run when the container starts, looks for scripts inside a specific directory, /docker-entrypoint-initdb.d, and runs them.

You can mount an SQL script to create the testing database if it doesn’t exist yet.
To create a ‘testing’ database automatically, you can use the following script:

SELECT 'CREATE DATABASE testing'  
    WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'testing')\gexec

And here is an example of what to change in à docker-compose.yaml file:

db:  
    image: postgres:latest
    volumes:    
        - ./infrastructure/database/pgsql/create-testing-database.sql:/docker-entrypoint-initdb.d/10-create-testing-database.sql

The entry point only runs the scripts when the storage volume is empty

Adding a script to create the testing database on an existing project where the database container already has a volume will not work. Two solutions here: run the script by hand or drop the volume and restart the container.

I learned that trick working on a project using Laravel Sail, so thank you to them!

Testing that generated PDFs are visually the same

2023-12-03T00:00:00Z

We need to generate various PDF files for the project I’m currently working on. Some of them are around 50 pages long. With such large documents, missing that we made a change that messes with the output has a high probability of happening. We wanted to automate discovering mistakes and avoid the tedious task of looking at no-so-interesting PDFs - most of them are contracts - after every change and decided to add some automated tests. Bonus point if our tests are capable of discovering that the newly created PDFs are not the same as the expected ones but also to show us what is different. And we did it.

We created a custom assertion we can use inside PhpUnit. That assertion is in a trait, which means we can use it in multiple testing classes.
That assertion uses diff-pdf, a CLI tool for visually comparing PDFs. If you want to use the assertion, you must install it.

The assertion is inspired by Approval Testing and only takes the path to the new PDF as a parameter. It then compares that file with a stored PDF with the new one. If they are not visually similar the assertion fails.

Here is an example of its usage:

class GeneratePDFTest extends TestCase  
{  
use PdfAssertion;

    /**  
     * @test  
     */  
    public function generate_the_PDF_document_with_the_information_we_want(): void  
    {  
        $PDFGenerator = new PDFGenerator();  
  
        $pdfFilePath = $PDFGenerator->generateFor('Charles');  
  
        $this->verifyPDF($pdfFilePath);  
    }
}

First, use the PdfAssertion trait, and then, in your tests, generate a PDF and call the verifyPDF method with the path of the newly generated PDF.

When you run the test, the generated PDF is copied to a file ending with .received.pdf in the approval directory at the same level as the test file. That file is compared with the file ending with .approved.pdf in that same directory. If they match, the test passes; if they don’t, the test fails. You must keep the .approved.pdf as it is the reference for future test runs. The *.received.pdf pattern can be added to your .gitignore.

The first time you run the test, you’ll get an error explaining that no file was previously approved and that you need to review the generated file and create an approved file if you are satisfied with its content. The error message even gives you the copy command to run to create the approved file.

If the two files do not match and the test fails, the tool will produce a file ending with .diff.pdf showing you the differences between them.
That’s super convenient. There is no need to read all that long contract to discover what changed!
These *.diff.pdf should also be excluded from versioning.

Below is a diff of a PDF supposed to render “Bravo Charles” that actually rendered “Bravo Champion.” The end of “Charles” is red colored, and the end of “Champion” is in cyan.

Here is the code of the trait:


trait PdfAssertion  
{  
    /**  
     * @var int[]  
     */
     private array $PDFVerifiedInTestCount = [];  
  
    private function verifyPDF(string $pdfFilePath): void  
    {  
        self::assertFileExists($pdfFilePath, 'The PDF file doesn\'t exist');  
  
        $this->incrementPDFVerificationInTestCount();  
  
        $this->copyToReceivedFile($pdfFilePath);  
  
        $this->ensureApprovalDirectoryExists();  
          
        $this->ensureApprovedFileExists();  
  
        $this->verifyThatPDFsAreMatching();  
    }  
  
    private function verifyThatPDFsAreMatching(): void  
    {  
        $approvedPDFFilePath = $this->approvedFileName();  
        $receivedPDFFilePath = $this->receivedFileName();  
        $diffedPDFFileName = $this->diffedFileName();  
  
        $command = sprintf(  
            'diff-pdf --output-diff="%s" "%s" "%s" 2>&1',  
            $diffedPDFFileName,  
            $approvedPDFFilePath,  
            $receivedPDFFilePath  
        );  
  
        exec($command, $output, $resultCode);  
  
        self::assertEquals(  
            0,  
            $resultCode,  
            sprintf(  
                <<<EOS  
The generated PDF file is not the same as the control pdf. Diff is visible here: %s  
EOS,  
                $diffedPDFFileName  
            )  
        );  
    }  
  
    private function ensureApprovedFileExists(): void  
    {  
        $approvedPDFFilePath = $this->approvedFileName();  
  
        $receivedFileName = $this->receivedFileName();  
  
        self::assertFileExists($approvedPDFFilePath, sprintf(<<<EOS  
No approved file exist for PDF %1\$s.  
Please review that file and if you are satisfied with its content, copy it to %2\$s.  
cp "%1\$s" "%2\$s"  
EOS,  
            $receivedFileName,  
            $approvedPDFFilePath  
        ));  
    }  
  
    private function ensureApprovalDirectoryExists(): void  
    {  
        $approvalDirectory = $this->getSnapshotDirectory();  
  
        if (!is_dir($approvalDirectory)) {  
            mkdir($approvalDirectory);  
        }  
    }  
  
    private function incrementPDFVerificationInTestCount(): void  
    {  
        $testName = $this->cleanTestName();  
  
        if (!array_key_exists($testName, $this->PDFVerifiedInTestCount))          {  
            $this->PDFVerifiedInTestCount[$testName] = 0;  
        }  
  
        $this->PDFVerifiedInTestCount[$testName]++;  
    }  
  
    private function currentPDFVerificationCount(string $testName): int  
    {  
        return $this->PDFVerifiedInTestCount[$testName];  
    }  
  
    private function copyToReceivedFile(string $pdfFilePath): void  
    {  
        copy($pdfFilePath, $this->receivedFileName());  
    }  
  
    private function approvedFileName(): string  
    {  
        return $this->fileName('approved');  
    }  
  
    private function diffedFileName(): string  
    {  
        return $this->fileName('diff');  
    }  
  
    private function receivedFileName()  
    {  
        return $this->fileName('received');  
    }  
  
    private function fileName(string $suffix): string  
    {  
        $approvalDirectory = $this->getSnapshotDirectory();  
  
        $cleanTestName = $this->cleanTestName();  
  
        return $approvalDirectory . ‘/’. $cleanTestName . ‘_’. $this->currentPDFVerificationCount(  
                $cleanTestName  
            ) . ‘.’. $suffix . ‘.pdf’;  
    }  
  
    private function getSnapshotDirectory(): string  
    {  
        return dirname((new ReflectionClass($this))->getFileName()).  
            DIRECTORY_SEPARATOR.  
            ‘approval’;  
    }  
  
    private function cleanTestName(): string  
    {  
        return self::cleanFilename($this->nameWithDataSet());  
    }  
  
    private static function cleanFilename(string $raw): string  
    {  
        $file = preg_replace("([^\w\s\d\-_~,;\[\]\(\).])u", '', $raw);  
  
        $file = preg_replace("([\.]{2,})", '', $file);  
  
        return $file;  
    }  
}

Part of the code is stolen from Spatie’s Snaphots Assertion library. Thank them!

If you look at the code, you’ll notice that the error messages are super helpful. They give you the link to the diff.pdf when the PDFs are not matching and the copy command call you should make after reviewing your first PDF generation.

I hope this code will help you save some precious time you’re currently spending looking at boring generated documents!

Cleaning some legacy code is intellectually rewarding; it can even be fun when you use your automated refactoring tools, not to mention that it greatly reduces the risk of mistakes. If you need some help to get started or to deal with your legacy codebase, let's chat and see how I can help.

Process Behavior Charts - A Christmas Tale

2023-12-17T00:00:00Z

Last year, around the same period, I was thinking about Process Behavior Charts (PBC), a tool that helps in figuring out when you should or should not react when looking at data. I don’t remember what the trigger was (probably somebody saying to try harder at something), but I thought that this tool should be more widely known.

Because Christmas was approaching, I decided to create a small Christmas Tale to introduce people to PBCs. And it worked because I know some people now use them at a past company where I worked.

PBC, the short version

Every process, or system, exhibits variation. You don’t finish the same amount of User Stories every week, the defect rate of that machine producing widgets is not always 0.2, and the duration of an appointment with a doctor is not always exactly 15 minutes...

Variations come in two sorts: noise and signal. Noise is the normal variation you can expect from a stable system. Signal, on the contrary, are variations that demonstrate that your system is unstable, that there is something unknown about your system, and that you should investigate.

Some people (like, a lot of people) react to every data point they see, no matter if that data point is something that we should expect from the stable behavior of the system or if it’s something more interesting we need to look into. A lot of time is lost looking at numbers, reacting to every variation going up or down. This leads to finger-pointing, time spent trying to understand something you won’t be able to explain, and management helping by asking to work harder...

PBCs help in sorting what is signal and what is noise. They tell you if your system is stable and predictable. They tell you when you should look into why that data point is better or worse than another to understand what you can improve.

My Christmas Tale

So, I wrote a Christmas Tale, in French, more as a joke than anything, and published it on LinkedIn. I added a mention to Mark Graban, who is an expert in the Lean community and who wrote "Measures of Success", the book that introduced me to PBCs. Mark sent me a message right away to ask me if we could translate the story into English. I, of course, agreed. We exchanged a few emails, improved the story a little, translated it, and published it. Mark even bought the domain name of the fictional Santa Claus company, North Pole Box - that redirects to his website where he sells the "Measures of Success" book.

A few weeks after Christmas, I even received a printed copy with the French and English version of the story in my mailbox. Thanks, Mark!

Process Behavior Charts - The Christmas Tale

I’m sure that by now you would like to immerse yourself in that fairy land of PBC and Christmas.

Here is the English version:

And here is the French one:

Want to Know More About PBCs?

If you want to know more about PBCs, I encourage you to read Mark’s book. The book explains PBCs but also demonstrates why most of the time we use numbers quite poorly, leading to overreaction, pressure, and no real improvement. The downside of the book is that once you’ve read it, a lot of meetings where your colleagues are sharing numbers will feel terribly unproductive to you...

Also, Daniel Vacanti recently published "Actionable Agile Metrics Volume II" that dives into PBC for flow metrics. I’m currently reading it, and this is a good book as well to understand what PBCs are and how you can use them.

Of course, the book on that subject is "Understanding Variation" by Donald J. Wheeler, who developed the idea of PBCs and closely worked with Edward Deming. I’ve recently ordered that book, so I can’t tell if it’s an easy read or not. From what I’ve read about it, it appears to be.

Speaking about Dr. Wheeler, Mark told me that he sent the link to the ebook to him. It’s also a nice gift to know that my silly idea reached the eyes of a legend in statistics and process improvement.

Quickly check for duplication with your IDE refactoring tools

2024-01-05T00:00:00Z

This week, I was looking at some code that I suspected contained duplication, presenting a good refactoring opportunity. I wanted to know if I could eliminate the duplication in the multiple conditionals, or even go a step further and consolidate the conditional logic in one place, where I could edit it to make it simpler.

Here is that code:


public function getDataAboutSomething(ASomethingId $somethingId): DataAboutSomething  
{  
    // Some other code  
  
    if ($draftEloquent) {  
        if ($draftEloquent->validation) {  
            $draft = new PieceOfData(false, true, $draftEloquent->validation);  
        } else {  
            $draft = new PieceOfData(true, false, $draftEloquent->validation);  
        }  
    } else {  
        $draft = new PieceOfData(false, false, null);  
    }  
  
    if ($preMarketEloquent) {  
        if ($preMarketEloquent->validation) {  $preMarket = new PieceOfData(false, true, $preMarketEloquent->validation); } else {  
            $preMarket = new PieceOfData(true, false, $preMarketEloquent->validation);  
        }  
    } else {  
        $preMarket = new PieceOfData(false, false, null);  
    }  
  
    if ($deliveryEloquent) {  
        if ($actableEloquent->validation) {  
            $delivery = new PieceOfData(false, true, $deliveryEloquent->validation);  
        } else {  
            $delivery = new PieceOfData(true, false, $deliveryEloquent->validation);  
        }  
    } else {  
        $delivery = new PieceOfData(false, false, null);  
    }  
  
    if ($archivedEloquent) {  
        if ($archivedEloquent->validation) {  
            $archived = new PieceOfData(false, true, $archivedEloquent->validation);  
        } else {  
            $archived = new PieceOfData(true, false, $archivedEloquent->validation);  
        }  
    } else {  
        $archived = new PieceOfData(false, false, null);  
    }  
  
    return new DataAboutSomething($draft, $preMarket, $delivery, $archived);  
}

By now, you probably notice the duplication too and are thinking about how to refactor the code.

But, should you trust your eyes completely? Initially, I trusted my judgment and moved forward with the code. However, let's pause for a moment here.

How can you be certain it's truly duplication?

Searching for one occurrence to match it with others isn't effective, as the variable names are different. You could use a regex search, but it's not straightforward, especially if the indentation varies (as I intentionally did in one of the occurrences).

So, you're left with meticulously reading the code.

Or...

You could proceed as I did: try extracting a method and rely on your IDE to handle it correctly. If the code is essentially the same, except for variable names, the IDE will extract the method and offer to replace all occurrences of the duplicated code.

This approach worked, except for one instance. If you examine closely, you'll notice that the variables in the two if conditions aren't the same. Did you catch that initially? I didn't.

Here is a gif of the process:

Without the IDE, I might have inadvertently altered the logic of the code. The IDE revealed that what I thought was duplicated four times was actually only three. This discovery led me to discuss the inconsistency with my coworkers, who suspected it was likely a bug and should have been coded like the others.

So, I uncovered a bug by leveraging my IDE's automated refactoring tools and being somewhat lazy.

Additionally, the IDE, by analyzing the syntactic form of the code, recognized that the second block, despite its irregular indentation, had the same meaning as the method I extracted.

Dealing with legacy code often uncovers insights when experimenting with refactoring tools. Try things out, see where it leads. If you're not satisfied with the results, you can always revert the changes. Assess if the refactoring aligns with your expectations. It's an effortless and risk-free approach.

When you spot potential duplication in code, try extracting a method. See if the IDE applies the extraction in multiple places or not. You'll gain insight. And who knows, you might just eliminate the duplication with a few keystrokes.

3 techniques pour gérer l'aléatoire en Approval Testing

2024-03-23T00:00:00Z

This article is available in English.

Lorsqu'on est confronté à du code legacy, se retrouver face à de l'aléatoire n'est pas ce qu'il y a de plus sympa.

Mettre en place des tests pour couvrir un système qui produit des résulats aléatoires peut s'avérer assez vite difficile.

Comme tout le monde, vous voulez que vos tests soient déterministes et vous donnent toujours le même résultat, quand vous n'avez rien changé dans le système.

Voici trois idées qui peuvent vous aider à gérer l'aléatoire dans le système. Les deux premières techniques sont fondées sur l'idée de supprimer la partie aléatoire du système pendant les tests.

Bloquer le générateur d'aléatoire

La première solution que je souhaite partager dans cet article consiste à garantir que le générateur aléatoire renvoie toujours les mêmes résultats lorsqu'il est appelé. La réponse ici est de configurer le générateur avec une valeur que vous contrôlez, afin de rendre le générateur déterministe.

Par exemple, en PHP, vous pouvez appeler la fonction mt_rand. Plus tard, tout appel à la fonction rand renverra les mêmes nombres aléatoires dans le même ordre.

C'est une manière rapide de transformer un système non déterministe en quelque chose qui donne toujours les mêmes résultats.

Malheureusement, cette solution, tout en étant l'une des façons les plus rapides de résoudre le problème, ne fonctionne que dans certaines situations. Tout d'abord, toutes les langages ne permettent pas de configurer le générateur de nombres aléatoires. Si ce n'est pas possible il vous faute une autre solution.

Un autre inconvénient de cette solution est que si vous modifiez la structure de votre code et que l'ordre des appels à une méthode aléatoire est changé vous obtiendrez toujours les mêmes valeurs dans le même ordre mais pas dans le même espace logique. Par exemple, prenons deux appels à rand et assignons le résultat du premier appel à la variable A et du second appel à la variable B. Après avoir configuré le générateur de nombres aléatoires, A et B ont toujours les mêmes valeurs, disons 34 et 1879. Lors d'un refactoring, si nous décidons d'inverser l'ordre d'assignation, nous avons un problème. Le code assigne maintenant la valeur de B avant celle de A ; A vaut maintenant 1879 et B vaut 39. Vous pouvez expérimenter cette idée ici.

Que pouvons-nous faire lorsque la configuration du générateur de nombres aléatoires est impossible, ou si nous souhaitons peut-être changer l'ordre des appels à un moment donné ?

Contrôler l'aléatoire

L'autre alternative est de reprendre le contrôle sur la génération des valeurs. Comme souvent, ajouter une couche d'abstraction est l'une des solutions disponibles dans notre boîte à outils. Ici, nous souhaitons briser la dépendance directe envers le générateur de valeurs aléatoires et le remplacer par une dépendance que nous pouvons manipuler.

En termes plus concrets, au lieu d'appeler directement une fonction du langage, nous l'encapsulons, créons un stub que nous pouvons manipuler, et trouvons un moyen de substituer l'implémentation originale avec notre doublure.

Les techniques de substitution disponibles varient grandement en fonction du langage de programmation utilisé et de l'état du code. Si vous avez la chance de pouvoir injecter des dépendances lors de la construction, c'est une tâche facile. Si ce n'est pas le cas, certains patterns issus de Working Effectively With Legacy Code par Michael Feathers peuvent s'avérer utiles.

Cette solution est disponible dans tous les langages. La seule exigence est de se sentir à l'aise avec la manipulation de code sans avoir de tests pour pouvoir reprendre le contrôle.

Une fois que vous avez le contrôle, vous pouvez décider des valeurs générées.

Cette solution nous permet de résoudre le problème d'ordre mentionné précédemment. Si vous inversez l'ordre des appels dans le code, vous pouvez changer la configuration de votre stub pour changer l'ordre et maintenir les tests valides.

Modifier le stub après la création des tests doit être fait avec prudence. Si vous voulez plus de confiance et éviter de toucher aux tests, vous avez une autre alternative. Au lieu de créer une abstraction pour chaque appel au générateur aléatoire, vous pouvez en créer une pour chaque appel. Reprenant l'exemple de la première partie, vous pouvez introduire une abstraction pour générer A et une autre pour générer B. Avoir deux implémentations vous donne un contrôle plus granulaire, et vous pouvez créer deux stubs. Lorsque vous inversez l'ordre des appels, le code appelle toujours les bonnes abstractions et les mêmes stib, on obtient donc le même résultat final.

Modifier une base de code existante pour introduire des indirections afin de contrôler des valeurs générées aléatoirement n'est pas toujours simple ni nécessaire. Parfois, la valeur générée ne nous importe pas, et nous pouvons tout à fait accepter l'aléatoire tant que nous parvenons à maintenir nos tests déterministes. Et c'est le point de la troisième idée.

Masquer l'aléatoire

Parfois, la valeur aléatoire n'est pas vraiment importante pour le résultat. Si cela ne nous importe pas, nous pouvons l'éliminer de ce qui est vérifié par une assertion dans le test.

L'astuce est que vous n'avez pas l'obligation de faire des assertions basées sur les sorties du système sous test telles quelles. À la place, vous pouvez récupérer ces sorties et les transformer pour ne conserver que les informations dont vous avez besoin pour être suffisamment confiant que vous ne cassez rien.

Lors de la création de tests d'approbation, il est courant d'utiliser un printer, une fonction ou une classe qui transformera les sorties pour les rendre faciles à comprendre. Si une information n'est pas pertinente, le rôle de l'imprimante est de la supprimer et de créer une sortie propre. Vous pouvez alors utiliser cette sortie propre dans l'assertion.

Quoi de plus simple que de supprimer ce dont on ne se soucie pas ?

Parfois, vous ne vous souciez pas de la valeur réelle, mais vous voulez vous assurer qu'une valeur est présente. Au lieu de supprimer la partie aléatoire, vous pouvez la remplacer par autre chose en utilisant un scrubber. Si vous voulez vous assurer qu'un UUID est affiché mais que sa valeur ne vous importe pas, vous pouvez utiliser une expression régulière qui va remplacer tous les UUID par autre chose, comme une chaîne UUID.

Un autre cas d'utilisation courant est de détecter qu'une valeur est répétée à plusieurs endroits dans la sortie.

Continuons avec l'exemple précédent. Vous ne vous souciez toujours pas de la valeur de l'UUID, mais vous voulez vérifier que l'UUID utilisé dans un lien pour accéder à un produit est bien celui du produit, et non un autre.

Pour cela il faut rendre notre scrubber un peu plus intelligent.

Lorsqu'un UUID est détecté à l'aide d'une regex, le scrubber génère une nouvelle chaîne, habituellement à partir d'une chaîne de base et d'un incrément (UUID_1). Ensuite, il remplace toutes les occurrences suivantes de cet UUID par la chaîne générée, incrémente le compteur et répète l'opération pour chaque UUID trouvé dans la sortie. De cette façon, même si vous ne pouvez pas prédéterminer le résultat du système, vous pouvez le transformer en quelque chose de déterministe que vous pouvez utiliser comme référence.

Le mieux avec cette solution est qu'elle ne nécessite pas de toucher au système legacy. Vous n'avez pas besoin de vous inquiéter de casser quelque chose en créant les tests. Cette méthode est donc très pratique et relativement rapide pour créer de la confiance. Avec un peu de chance, vous pourriez même travailler avec un langage où les outils d'approval testing viennent avec des scrubbers qui font une partie du travail de nettoyage pour vous.

Comprendre le fonctionnement d'un système utilisant de l'aléatoire et devoir créer un filet de test n'est pas obligatoirement effrayant ou ne doit pas forcément prendre énormément de temps. Avec les trois idées exposées dans cet article, vous devriez être en mesure d'attaquer ce code legacy que vous évitez de refactoriser depuis déjà trop longtemps.

Si vous souhaitez un coup de main pour attaquer un chantier de reprise de code legacy, discutons-en et voyons comment je peux vous aider.

Tester les extraits de code dans un livre

2024-04-19T00:00:00Z

Pour ma formation sur l'amélioration des tests j'ai mis en place des fiches récapitulatives pour chacun des chapitres. Ces fiches récapitulatives contiennent pas mal de code et je voulais m'assurer que le code soit fonctionnel. Quoi de mieux que des tests pour s'assurer que le comportement est tel qu'attendu ? Plutôt logique dans une formation sur les tests.

Les fiches sont à destination des étudiants mais j'ai également écrit un livre blanc partangeant 5 manières d'améliorer la lisibilité et la maintenabilité des tests en utilisant cette même technique. Récupérez-le pour vous donner une idée du résultat.

Asciidoc et inclusion de code

Pour écrire les fiches, j'ai décidé d'utiliser Asciidoc, un format textuel présentant des similarités avec le Markdown, principalement pour sa capacité à inclure des fichiers à l'intérieur du texte, et plus particulièrement du code. Même mieux, il est possible de n'inclure qu'une partie d'un fichier, ce qui permet de ne montrer que le code utile à la compréhension.

Chaque chapitre est stocké dans un dossier différent et le fichier principal pour la fiche s'appelle fiche.adoc. Il y a également un dossier code dans ce même dossier qui va contenir les fichiers contenant le code.

Pour inclure du code, il est possible d'utiliser l'instruction asciidoc suivante:

[source,php]  
----  
include::code/code-01.php[tag=class-facture]  
----

Cette instruction va inclure la portion de code entre les tags tag::class-facture[] du fichier code-01.php.

Ce fichier ressemble peu ou prou à ça:

<?php  
  
namespace PasDeCalculDansLesTests\Code_01;

class UneClasseQuOnNeVeutPasMontrerMaisDontOnABesoin {

}

// tag::class-facture[]  
final class Facture  
{  
 // ...
}  
// end::class-facture[]  
  
// tag::class-produit[]  
final class Produit  
{  
}  
// end::class-produit[]  
  
// tag::testfile[]  
class FactureTest extends \PHPUnit\Framework\TestCase {  
  
	// tag::test_calcule_le_total[]
     /**
     * @test  
     */  
    public function calcule_le_total(): void {  
    }  
    // end::test_calcule_le_total[]  
}  
// end::testfile[]

Quelques informations importantes:

Comme vous le voyez, ce fichier contient tout le code qui lui est nécessaire afin d'être indépendant. C'est primordial car chaque chapitre du cours est présenté comme une histoire au travers de laquelle on va modifier le code ou les tests. On veut donc pouvoir avoir plusieurs versions du code en parallèle.
Chaque version du code est contenue dans son propre namespace, ici PasDeCalculDansLesTests\Code_01, ce qui permet d'éviter les conflits entre les différentes versions du code.
Le fichier de code contient le code qu'on ne souhaite pas montrer dans la fiche mais qui est pourtant nécessaire au bon fonctionnement du code.
Il est possible d'utiliser plusieurs tags différents pour montrer plusieurs sections du code. Le snippet précédent ne vas inclure que la classe Facture.
Les tests se trouvent également dans ce fichier, ce qui veut dire qu'ils sont aussi versionnés.

Lorsque l'histoire avance et qu'une autre version du code est nécessaire il suffit alors de copier le fichier et de créer le fichier code-02.php et de changer le namespace. On a alors un nouveau terrain de jeu indépendant.

Lancer les tests

Pour lancer les tests facilement, j'ai créé une suite de test PhpUnit qui permet de lancer les tests contenus dans n'importe quel fichier .php dans le dossier content, comme on le voit dans cet extrait du fichier phpunit.xml.

<testsuites>  
    <testsuite name="All">  
        <directory suffix=".php">./content/*</directory>
    </testsuite>
</testsuites>

Je peux donc désormais lancer les tests pour m'assurer que les exemples contenus dans les fiches sont corrects.

Exclure des tests

Dans certains chapitres, il est intéressant de montrer du code qui fait échouer les tests. Je ne veux pas que ma suite de test échoue pour ces tests à chaque lancement, j'ai donc fait en sorte de pouvoir les exclure facilement.

J'ai expérimenté avec deux idées.

La première est d'exclure les fichiers de code contenant les tests cassants de la suite de test, en utilisant la balise <exclude> de la configuration PhpUnit.

<testsuites>  
    <testsuite name="All">  
        <directory suffix=".php">./content/*</directory>
        <exclude>./content/sections/02-AAA/06-BBB/code/code-01.php</exclude>
    </testsuite>
</testsuites>

La seconde est d'utiliser une annotation @group disable sur les tests que je souhaitais pas lancer et de configurer PhpUnit, là encore dans phpunit.xml, pour exclure ces tests.

<groups>  
    <exclude>  
        <group>disable</group>  
    </exclude>  
</groups>

Il s'agit d'un des usages des tags dans ce projet. J'ai également utilisé les tags pour pouvoir aisément exclure les tests s'appuyant sur la base de données pour ne pas avoir besoin d'avoir un serveur disponible en permanence.

Avec ces idées, il est possible de facilement mettre en place des tests pour s'assurer que les exemples de code présent dans vos articles, livres, ou même documentations sont corrects.
D'ailleurs, inclure des fichiers de code ou des fichiers générés par du code permet de créer une documentation vivante, toujours à jour avec le fonctionnement actuel d’un système (en utilisant de l’approval testing comme l’explique Sébastien Fauvel.

Malheureusement, j’utilise du markdown

Si vous utilisez du Markdown, qui ne permet pas l’inclusion de fichier comme le fait Asciidoc, il est possible d’utiliser MarkdownSnippets. Ce programme va créer une copie du fichier markdown original en incluant le code, ce qui peut être fait dans un pipeline de build.

Je suis vraiment content du rendu des fiches dans la formation avec cette technique. Si vous avez besoin d’aide pour mettre en place ce genre d’idées dans vos équipes, parlons-nous).

Generating a file without blocking form submission

2024-05-20T00:00:00Z

Last week I was discussing with one teammate at my current client about an issue they had.

The application generates a document based on some data provided by the user.
He explained to me that we needed to wait for the document to be generated to be able to redirect to the next page, just after form submission, because there is a link to download the document there, which means we need to know where the document is, which means we need to have already created the document.

At first, it may seem like an intractable issue, but it’s actually not the case.
There is a solution that doesn’t require the document to be generated before redirecting the user to the confirmation page.
That solution, in its most basic form, is relatively simple and doesn’t require a lot of code.

As we’ll see, we can add some complexity to the solution to improve the user experience.

You can have access to the code in the code repository.

Background story

Here is what the story looks like:

The user enters some information
The server starts generating a file based on that information asynchronously.
The user can click a link to download the file.

We want the user to have a pleasant experience, so we don’t want to block the form submission until the file is created.
To provide that nice experience, when the form is submitted, the information is stored, an asynchronous process is started for file generation, and the user is redirected to a confirmation page containing a download link.

If we’re lucky, the file was created by the time the user gets to the confirmation page, and we can show a download link pointing directly to it.
But what if the file was not created yet?
We still want to display the confirmation page as soon as possible. The first trick is that we actually don’t need the file until the user clicks on the download link.
Not needing the file until the user requests the download gives us some extra time to generate it in the background.

“That’s great, you may say, but we still need a URL to download the file. How can we know the file URL if the file hasn’t been generated?”
This is a good question, and the answer is to use a second trick: You don’t need the file URL; you need a promise to be able to get that URL at some point.
You need a way to ask the server if the file is available and how to get it when it’s ready.

This repository aims to show how to implement that, evolving the solution from a super simple solution with pure HTML pages to a click-to-download link with a spinner.
Seeing the iterations from the first to the final one helps to understand the pattern to solve that problem.

Installation and run

The demo is a node application using the express framework.

Start with a git clone from the code repository.

Everything should be properly installed with:

npm install

Run the application with:

node index.js

Using the demo app

You are the worker

⚠️ The background process is faked; you need to manually create the file.
Because implementing an asynchronous job is not in the scope of this demo, there is no worker.
To make things convenient, this repo includes two commands to create and delete the file.
Said differently, you are the worker.
This gives you the advantage of experimenting at each step and observing the behavior of each click when the file is there or not.
You decide.

To create the file, run:

npm run create-receipt

To remove the file, run:

npm run clean-receipt

There is no form

Because there is no worker, there is no need to trigger the worker, which allows us to skip building the form.
The confirmation page is the first page, served at the / route.
In a real application, you are expected to see this page right after submitting the form.
It displays a link to download a file.

Iterations

I’ve created this demo iterating on the code, starting with the simplest solution.
The cool thing is that it demonstrates how the solution works without JS.
Even better, because it was built gradually, it continues working even if the user browser deactivates JS.
JS is only there to improve the UX, but the application is still functional without it.

Play with the demo

Once you’ve cloned the repository, you can see the code for each iteration with a checkout of the correct tag.
I’ve provided a list of things you can experiment with to see how the system behaves for each iteration in experiment block such as this one.
To get things more interesting, look at the network tab of your browser development tools.

Aside discussions

In blocks such as this one, I’ve included aside discussions to extend on some of the ideas.

Iteration 1: User polling

Get the code

git checkout human-polling

or view on Github

This iteration is super raw but really helps understand how this works as it requires the user to manually ask (and ask again, if needed) for the file.
Future iterations are just UX improvements on top of it.

sequenceDiagram
actor User
participant confirmation_page as Confirmation page
participant waiting_page as Waiting page
participant receipt_controller as Receipt controller
participant file_download_controller as File download controller
confirmation_page-->>User: 👁️ Display "Download the file" link
User->>receipt_controller: 👇 Click on "Download the file" [GET]
loop
    alt File doesn’t exist
        receipt_controller-->>waiting_page: returns
        waiting_page-->>User: 👁️ Display "Wait a few seconds and click that link to get the file"
        User->>receipt_controller: 👇 Click the link [GET]
    else File exists
        receipt_controller->>file_download_controller: Redirects to [with Location header]
        file_download_controller-->>User : 📄 File
    end
end

app.get('/', (req, res) => {
    const htmlContent = `
    <!DOCTYPE html>
    <html lang="en">
      <head>
        <title>Form submitted</title>
      </head>
      <body>
        <h1>Your form was submitted !</h1>
        <p>Thank you for your information.</p>
        <a href="/receipt">Get your receipt</a>
      </body>
    </html>
  `;
    res.send(htmlContent);
})

app.get('/receipt', (req, res) => {

    fs.access(filePath, fs.constants.F_OK, (err) => {

        if (!err) { // File exists
            res.redirect('/download');
            return;
        }

        // File doesn't exist yet
        const htmlContent = `
    <!DOCTYPE html>
    <html lang="en">
      <head>
        <title>File Download</title>
      </head>
      <body>
        <h1>File download</h1>
        <p>We are creating your file.</p>
        <p>Please wait 5 seconds and click the download link below:</p>
        <a href="/receipt">Download</a>
      </body>
    </html>
  `;
        res.send(htmlContent);
    });
});

app.get('/download', (req, res) => {

    res.download(filePath, (err) => {
        if (err) {
            console.error(`Error during download: ${err}`);
            res.status(500).send('Error downloading the file.');
        } else {
            console.log(`File downloaded: ${filePath}`);
        }
    });
});

The confirmation page now displays a download link to the /receipt route.
The /receipt route controller acts as a traffic cop.
If the file is available, it will redirect to the real file download URL.
Otherwise, it returns a page explaining that the file is still being created and asking the user to wait a few seconds before clicking on a link to download it.

How can we deal with the file not being available when the user clicks on that second link?
The solution is to repeat that same process and show a page asking to wait and a link to download the file.
Fortunately, we already have a route that behaves this way, the /receipt one.
Here the solution is that the link on the waiting page is a link to the waiting page.
Users are manully polling to see if the file is available when they click on the link.

Experiment

Create the file and click on the link on the confirmation page
Create the file once you are on the waiting page and click the link
Go to the waiting page, click the link, create the file, and click the link again

Why redirect and not send the file immediately from the receipt route?

You might think that the redirection overcomplicates the flow, and you want to send the file directly from the receipt route.
Sure, it saves from creating a new route and removes one HTTP request, but you should probably keep that as it is.

First, the file can be stored elsewhere, on a file storage like AWS S3.
In that case, you don’t want the file to pass through your server and want the file storage to serve it.
A redirection makes a lot of sense here.

Second, using the same URL for the traffic cop and serving the file will make caching harder.
Having a dedicated URL for the file will help the caching mechanisms detect that they can avoid fetching the file from the server again and again.

Iteration 2: Blinking polling

Get the code

git checkout blinking-polling

or view on Github

The next improvement introduced is to avoid the need for the user to click on the second link on the waiting page.

sequenceDiagram
    actor User
    participant confirmation_page as Confirmation page
    participant waiting_page as Waiting page
    participant receipt_controller as Receipt controller
    participant file_download_controller as File download controller
    confirmation_page-->>User: 👁️ Display "Download the file" link
    User->>receipt_controller: 👇 Click on "Download the file" [GET]
    loop
        alt File doesn’t exist
            receipt_controller-->>waiting_page: returns
            waiting_page-->>User: 👁️ Display "Wait a few seconds and click that link to get the file"
            rect rgba(172, 226, 225, 0.4)
                activate waiting_page
                waiting_page->>waiting_page: ⏱️ Wait for the delay specified in refresh meta
                waiting_page->>receipt_controller: GET
                deactivate waiting_page
            end
        else File exists
            receipt_controller->>file_download_controller: Redirects to [with Location header]
            file_download_controller-->>User : 📄 File
        end
    end

app.get('/receipt', (req, res) => {

    fs.access(filePath, fs.constants.F_OK, (err) => {

        if (!err) { // File exists
            res.redirect('/download');
            return;
        }

        // File doesn't exist yet
        const htmlContent = `
    <!DOCTYPE html>
    <html lang="en">
      <head>
        <title>File Download</title>
        <meta http-equiv="refresh" content="5">
      </head>
      <body>
        <h1>File download</h1>
        <p>We are creating your file.</p>
        <p>Please wait 5 seconds and click the download link below:</p>
        <a href="/receipt">Download</a>
      </body>
    </html>
  `;
        res.send(htmlContent);
    });
});

The simplest solution is to reload the waiting page every N seconds, which is what the user was manually doing by clicking on the link.
Once the file is ready, the download starts.

This is achieved by using a meta tag to the header:

<meta http-equiv="refresh" content="5">

Still no JS.

An information for the HTTP client

One interesting thing to note with the introduction of that meta tag is that we start differentiating information for the client and the user.
The meta tags is for the browser, the HTTP client, which knows how to interpret it.
The text is for the user, who knows how to interpret that as well.
The two pieces of information don’t have to match.

Experiment

Go to the waiting page and wait until the page reloads, then create the file

Iteration 3: Polling without blinking

Get the code

git checkout server-retry-after

or view on Github

The next improvement is to poll the server to see if the file has been created without reloading the page.
Now is the time for JS to come into play.

sequenceDiagram
    actor User
    participant confirmation_page as Confirmation page
    participant waiting_page as Waiting page
    participant receipt_controller as Receipt controller
    participant file_download_controller as File download controller
    confirmation_page-->>User: 👁️ Display "Download the file" link
    User->>receipt_controller: 👇 Click on "Download the file" [GET]

    alt File doesn’t exist
        receipt_controller-->>waiting_page: returns
        waiting_page-->>User: 👁️ Display "Wait a few seconds and click that link to get the file"
        rect rgba(172, 226, 225, 0.4)
            activate waiting_page
            loop
                waiting_page->>receipt_controller: GET
                alt File doesn’t exist
                    receipt_controller-->>waiting_page: Retry-After
                    waiting_page->>waiting_page: ⏱️ Wait for the delay specified in Retry-After
                else File exists
                    receipt_controller-->>waiting_page: Content-Disposition: Attachment Location:/download
                    waiting_page->>file_download_controller: GET
                    file_download_controller-->>waiting_page :  📄 File
                    waiting_page-->>User: 📄 File
                end
            end
            deactivate waiting_page
        end
    else File exists
        receipt_controller->>file_download_controller: Redirects to [with Location header]
        file_download_controller-->>User : 📄 File
    end

Once the waiting page is displayed, it starts polling the server to get information about the file.
It keeps polling the same route as before using JS fetch function. We are moving from a space where the browser is the HTTP client to a world where we are creating our own client.

app.get('/receipt', (req, res) => {

    fs.access(filePath, fs.constants.F_OK, (err) => {

        if (!err) { // File exists
            res.redirect('/download');
            return;
        }

        // File doesn't exist yet
        const htmlContent = `
    <!DOCTYPE html>
    <html lang="en">
      <head>
        <title>File Download</title>
      </head>
      <body>
        <h1>File download</h1>
        <p>We are creating your file.</p>
        <p>Please wait 5 seconds and click the download link below:</p>
        <a href="/receipt">Download</a>
      </body>
      <script>

      async function retryWithDelay(taskFn, maxRetries = 5) {
        let attempt = 0;
        let delayMs = 0;
    
        while (attempt < maxRetries) {
            console.log('Attempt');
            // Attempt to run the task function
            const result = await taskFn();
            
            attempt++;
            if (attempt >= maxRetries) {
                throw new Error('Max attempts')
            }
            
            if(result.type !== 'retry-after') {
               return result.value;
            }
            
            delayMs = result.value * 1000;
            await new Promise(resolve => setTimeout(resolve, delayMs));
        }
    }
    
        // Function to perform a GET request to the current page
       async function pollForFile() {
          const response = await fetch(window.location.href);
          
          if(response.headers.get('Retry-After')) {
              return {
                type: 'retry-after',
                value: response.headers.get('Retry-After')
              };
          }
          
          const contentDisposition = response.headers.get('Content-Disposition');
          
          if(contentDisposition && contentDisposition.includes('attachment')) {
           console.log('The content is marked for attachment download.');

            // Extract filename (if any) from Content-Disposition
            const filenameMatch = contentDisposition.match(/filename="?(.+)"?/);
            const filename = filenameMatch ? filenameMatch[1] : 'default-filename';

            // Process the response as a Blob and initiate the download
            const data = await response.blob();
            const downloadUrl = window.URL.createObjectURL(data);
            const a = document.createElement('a');
            a.href = downloadUrl;
            a.download = filename;
            document.body.appendChild(a);
            a.click();
            a.remove();
            window.URL.revokeObjectURL(downloadUrl);
          }
          
          return {
                type: 'success',
                value: true
              };
      }
    
    retryWithDelay(pollForFile, 10, 5000)
        .then(result => console.log(result))
        .catch(error => console.error(error.message));
       
      </script>
    </html>
  `;
        res.header('Retry-After', 5);
        res.send(htmlContent);

    });
});

Until the file is ready, a Retry-after header is added to the response.
It indicates to the client how long it should wait before asking again if the file is ready.
This is the information for the HTTP client.
The JS code interprets that header and retries after the delay proposed by the server.

Once the file is ready, the waiting route will issue a redirect to the file’s location.
The fetch function follows the redirection and gets a response containing Content-Disposition: attachment; header.
This is the trigger for doing some JS vaudoo to download the file.
That’s sad because we need to use some clever tricks to emulate something that was working natively until that point when the browser was the HTTP client.

Experiment

Go to the waiting page, wait a little, then create the file

Why not hard-code the retry delay?

The server provides the delay for each retry using the Retry-After header.
An alternative could be to hardcode the delay in the client.
Using a header provided by the server adds some interesting possibilities, though.
First, it’s our mechanism to tell the client that it still needs to wait.
Without that we would need another way to communicate that information.
Alternatively, we could have a longer hardcoded delay, but that would mean potentially asking the user to wait longer than necessary.

Secondly, because the server can communicate how long the client should retry, it can change that waiting time.
Maybe the server can predict how long it will take until the file is ready and ask the client to wait just that long.
Or the server is serving many requests at the moment and would like every client to slow down with the amount of requests they send.
In that case, it can increase the delay, using that tool as a back pressure mechanism.

Is there a less hacky way to start the download ?

Hi, JS folks. Is there a better way to start the download of a file when the request is made via fetch?
This is one of the solutions I found online, but I feel like it should be more straightforward.

Iteration 4: Clean and get a clever HTTP client

Get the code

git checkout clean-fetch-decorator

or view on Github

This iteration doesn’t bring any functional improvements, but cleans the code by bringing some functional composition.

app.get('/receipt', (req, res) => {

    fs.access(filePath, fs.constants.F_OK, (err) => {

        if (!err) { // File exists
            res.redirect('/download');
            return;
        }

        // File doesn't exist yet
        const htmlContent = `
    <!DOCTYPE html>
    <html lang="en">
      <head>
        <title>File Download</title>
      </head>
      <body>
        <h1>File download</h1>
        <p>We are creating your file.</p>
        <p>Please wait 5 seconds and click the download link below:</p>
        <a href="/receipt">Download</a>
      </body>
      <script>

      async function fetchWithRetryAfter(fetch, maxRetries = 5) {
        let attempt = 0;
        let response;
    
        while (attempt < maxRetries) {
            attempt++;
            
            response = await fetch();
            
            if(response.headers.get('Retry-After')) {
              let delayMs = response.headers.get('Retry-After') * 1000;
              await new Promise(resolve => setTimeout(resolve, delayMs));
              continue;
            }
            
            return response;
        }
        
         return response;
    }
    
        // Function to perform a GET request to the current page
       async function fetchDownloadAttachment(fetch) {
          const response = await fetch;
          
          const contentDisposition = response.headers.get('Content-Disposition');
          
          if(contentDisposition && contentDisposition.includes('attachment')) {
            // Extract filename (if any) from Content-Disposition
            const filenameMatch = contentDisposition.match(/filename="?(.+)"?/);
            const filename = filenameMatch ? filenameMatch[1] : 'default-filename';

            // Process the response as a Blob and initiate the download
            const data = await response.blob();
            const downloadUrl = window.URL.createObjectURL(data);
            const a = document.createElement('a');
            a.href = downloadUrl;
            a.download = filename;
            document.body.appendChild(a);
            a.click();
            a.remove();
            window.URL.revokeObjectURL(downloadUrl);
          }
          
          return response;
      }
    
    fetchWithRetryAfter(() => fetchDownloadAttachment(fetch(window.location.href)), 10)
        .then(result => console.log(result))
        .catch(error => console.error(error.message));
       
      </script>
    </html>
  `;
        res.header('Retry-After', 5);
        res.send(htmlContent);

    });
});

The JS code is refactored to have specific functions that are able to interpret an HTTP response and act accordingly.

fetchWithRetryAfter looks for a Retry-After header contained in a HTTP response and performs the retry logic.
fetchDownloadAttachment looks for an attachment and does the download trick.

You can create a clever HTTP client by combining all functions and reusing that for all your HTTP requests:

const cleverFetch =  (fetchParams) =>  fetchWithRetryAfter(() => fetchDownloadAttachment(fetch(fetchParams)))

The HTTP response handling logic is now contained in one place, and these functions can be tested independently.

Iteration 5: Spinner

Get the code

git checkout spinner

or view on Github

Let’s now move to the modern world of SPA.
In a SPA, you probably wouldn’t want the user to move from the confirmation page to the waiting page if the file isn’t ready but displays a spinner.

This is what this iteration is about. We want to stay on the confirmation page and avoid moving to the waiting page, even if the file is not there.

sequenceDiagram
actor User
participant confirmation_page as Confirmation page
participant waiting_page as Waiting page
participant receipt_controller as Receipt controller
participant file_download_controller as File download controller
confirmation_page-->>User: 👁️ Display "Download the file" link
rect rgba(172, 226, 225, 0.4)
User->>confirmation_page: 👇 Click on "Download the file" [GET]
activate confirmation_page
confirmation_page->>confirmation_page: 🔄 Display spinner
confirmation_page->>receipt_controller: GET
loop
    alt File doesn’t exist
        receipt_controller-->>confirmation_page: Retry-After
        confirmation_page->>confirmation_page: ⏱️ Wait for the delay specified in Retry-After
        confirmation_page->>receipt_controller: GET
    else File exists
        receipt_controller-->>waiting_page: Content-Disposition: Attachment Location:/download
        confirmation_page->>file_download_controller: GET
        file_download_controller-->>confirmation_page :  📄 File
        confirmation_page-->>User: 📄 File
    end
end
confirmation_page->>confirmation_page: Hide spinner
end

app.get('/', (req, res) => {
    const htmlContent = `
    <!DOCTYPE html>
    <html lang="en">
      <head>
        <title>Form submitted</title>
        <style>
        .spinner {
            display: none;
            border: 4px solid rgba(0, 0, 0, 0.1);
            width: 16px;
            height: 16px;
            border-radius: 50%;
            border-left-color: #09f;
            animation: spin 1s linear infinite;
        }

        @keyframes spin {
            0% {
                transform: rotate(0deg);
            }
            100% {
                transform: rotate(360deg);
            }
        }

        .hidden {
            display: none;
        }
    </style>
      </head>
      <body>
        <h1>Your form was submitted !</h1>
        <p>Thank you for your information.</p>
        <a href="/receipt" class="download">Get your receipt<div class="spinner"></div></a>
      </body>
      <script>
      
      const downloadLinks = document.getElementsByClassName('download');
      
      for(let downloadLink of downloadLinks) {
          downloadLink.addEventListener('click', function(event) {
              event.preventDefault();
              
              const spinner = event.currentTarget.querySelector('.spinner');
              spinner.style.display = 'inline-block';
              
              fetchWithRetryAfter(() => fetchDownloadAttachment(fetch(this.getAttribute('href'))), 10)
            .then(() => {
                spinner.style.display = 'none'
            })
            .catch(error => console.error(error))
          })
      }

      async function fetchWithRetryAfter(fetch, maxRetries = 5) {
        let attempt = 0;
        let response;
    
        while (attempt < maxRetries) {
            attempt++;
            
            response = await fetch();
            
            if(response.headers.get('Retry-After')) {
              let delayMs = response.headers.get('Retry-After') * 1000;
              await new Promise(resolve => setTimeout(resolve, delayMs));
              continue;
            }
            
            return response;
        }
        
         return response;
    }
    
        // Function to perform a GET request to the current page
       async function fetchDownloadAttachment(fetch) {
          const response = await fetch;
          
          const contentDisposition = response.headers.get('Content-Disposition');
          
          if(contentDisposition && contentDisposition.includes('attachment')) {
            // Extract filename (if any) from Content-Disposition
            const filenameMatch = contentDisposition.match(/filename="?(.+)"?/);
            const filename = filenameMatch ? filenameMatch[1] : 'default-filename';

            // Process the response as a Blob and initiate the download
            const data = await response.blob();
            const downloadUrl = window.URL.createObjectURL(data);
            const a = document.createElement('a');
            a.href = downloadUrl;
            a.download = filename;
            document.body.appendChild(a);
            a.click();
            a.remove();
            window.URL.revokeObjectURL(downloadUrl);
          }
          
          return response;
      }
      
      </script>
    </html>
  `;
    res.send(htmlContent);
})

For this, we attach an event listener to the link, prevent it from moving to the next page, display a spinner, and start polling the receipt route.
We keep the same logic as before and mostly the same JS code.
If the response contains a Retry-After header, the file is not there yet, and a future request is scheduled.
If the response contains a Content-Disposition: attachment; header, we do the JS download trick and hide the spinner.

Also, the URL used for polling is the one specified in the href attribute of the link.
We didn’t need to change that template part to make it work.
Even better, it means that if JS is disabled, the default behavior will take over from our code and will issue a request to /receipt, which will display the waiting page or redirect to the file, depending on the file’s existence.
That’s progressive enhancement.

Experiment

On the confirmation page, click on the link, see the spinner and create the file
On the confirmation page, create the file, then click on the link
On the confirmation page, click on the link, and do nothing. (I agree, I should have included an error message here.)
Disable JS, and try whatever you want to.

Iteration 6: Cleaning

Get the code

git checkout main

or view on Github

The next iteration is cleaning the code, reducing page duplication, using a templating system.

Help me with Handlebars inlined partials and ExpressJs

I want to use two regions in the layout, body and scripts.
The idea is that every page template is able to provide its content and its own script.
At the moment, the script is pushed right into the <body>.
It works, but I’m curious how to do that with Handlebars and ExpressJs.
I actually spent more time on this issue than on the rest of that demo...

That looks like a lot of work, should I do that?

Maybe, I don’t know, it depends.

Maybe your users are fine with waiting for the file to be generated, and you don’t need to do that at all.
As we’ve seen, if you take the asynchronous road, you have multiple stops.
Do you really need to stop at the spinner stop?

You can start with the first solution, without any JS, and track how many times the file is not available at the moment the user clicks the link to download the file.
If that doesn’t occur very often, because the system is faster at creating the file than user is at clicking the link in the majority of case, that solution may be good enough.

If this is something common you can start iterating on the solution as we did here.

Alternatively, you can also try to make file generation faster.

Alternatives

That solution based on an intermediary status resource is not the only solution to that problem.

Instead of polling to get the status of the file creation, you could use sockets to push the information to the client.
If you’re working in a server-to-server mode, you could also use a webhook mechanism to notify the other server that the file is ready.

Resources

Derek Comartin from the Code Opinion YouTube channel, made a nice video on that subject:
📺 Avoiding long running HTTP API requests. - Code Opinion.

The "RESTful Web Services Cookbook" covers the solution of the status resource as well.

So, here we are.
We are now able to display a confirmation page even when a file is still being generated.
This improves users experience as they don’t have to wait.
As we’ve seen, the solution, in its simplest form, doesn’t require a lot of code.

Embed external code via HTTP on an eleventy blog

2024-05-27T00:00:00Z

When I was writing my last article about generating a file without blocking form submission, I wanted to embed some code samples to explain the solution, but I wanted to avoid copy-pasting the code from the demo repository to the blog article. Inspired by Asciidoc partial inclusion, which I described in a previous article (in French), I decided to implement a similar method to retrieve some code via HTTP directly from a GitHub repo.

Well, I say "implement"... I should probably say "ask Chat GPT to do that for me." I wanted to see if the LLM could save me some time in implementing the solution. I thought the solution would be relatively easy to describe and create. I even created the prompt on my phone while commuting, just after having the idea. I tasked ChatGPT to build an Eleventy shortcode that takes a URL and a tag name, fetches the content, and returns the content between the start and end tags.

I had previous experiences where it took more time doing it with Chat GPT’s help than without it, but this time it wasn’t the case. Success on the first try, the generated code looked like what I imagined.

I tried the produced code when I was back on my laptop, and it worked. I just edited it a bit.

Here is the piece of code that you can include to configure 11ty in your .eleventy.js file:

 eleventyConfig.addAsyncShortcode("externalContent", async function(url, tag) {
    // Fetch the content from the provided URL
    const response = await fetch(url);
    const text = await response.text();

    // Define the start and end tags based on the input tag
    const startTag = `// tag::${tag}[]`;
    const endTag = `// end::${tag}[]`;

    // Extract the content between the tags
    const startIndex = text.indexOf(startTag) + startTag.length;
    const endIndex = text.indexOf(endTag);

    if (startIndex < 0 || endIndex < 0) {
      throw Error(`Tag ${tag} not found in the document at ${url}`);
    }

    // Return the substring between start and end indices
    return text.substring(startIndex, endIndex).trim();
  });

And here is how you can use it with the new externalContent shortcode:


{% externalContent "https://gist.githubusercontent.com/SelrahcD/a0d1e2295f4195c51836e36b86b8232b/raw/a4d44b9ad70ca9595f149541b8d69a47574d64cf/somePHP.php", "MyCode" %}

// Code that is inside the MyCode tag block
$a += $a;

Which fetches the raw content for a file in a Gist and displays:

// Code that is inside the MyCode tag block
$a += $a;

You can even use that shortcode inside a code block:

```php

{% externalContent "https://gist.githubusercontent.com/SelrahcD/a0d1e2295f4195c51836e36b86b8232b/raw/a4d44b9ad70ca9595f149541b8d69a47574d64cf/somePHP.php", "MyCode" %}

```

To get a nicely displayed PHP Code

// Code that is inside the MyCode tag block
$a += $a;

It’s a very raw solution and far from being quite as good as the Asciidoc implementation.
It deals only with comments starting with // and doesn’t handle tags inside tags.
But it’s good enough for my needs, and sometimes that just we don’t need to work more than that.

Getting your secret password from 1Password in a script

2024-05-29T00:00:00Z

A few weeks ago, I need to create a script that required one of my password.
I didn’t want to hardcode my password in the script and decided to use one super nice functionality from 1Password : access secrets stored in vaults using 1Password CLI, via the op command.

Steps

First, you’ll need to install 1Password CLI that will give you the ability to talk to 1Password right from your scripts.
Then, open the 1Password application, go to the item where your secret is stored. For instance, here I needed my Gmail password. ^[1]
Once copied, go to you script and get the secret using the op command.

gmail_pass=$(op read "<PASTE THE SECRET REFERENCE HERE>")

Use that secret in your script.

Now, when you’ll run your script, 1Password will ask you to identify and will give access to the secret to your script.

This as a couple of advantages.
The obvious one is that you don’t need to hardcode sensitive information in your scripts.
Secondly, that means its easier to rotate password because you don’t need to edit your scripts when you change your password, and you can do this easily even if your in a team setup.

A nice little trick!

Actually, you can’t really use your Gmail password for scripts but will need an application password. ↩︎

How I divided a PhpUnit test suite execution by 5

2024-06-07T00:00:00Z

During my last mission, I was working with a team that invested in testing to secure a Symfony legacy application and find ways to prevent some of the regressions. This was a really good move.

They mainly had two types of tests:

Some tests were written with CucumberJs, automating real user workflows on the user interface via Puppeteer. Execution of these tests involved a lot of HTTP requests, database calls, code logic execution, and wait time for rendering.
Some tests used PhpUnit, written with Symfony test clients, going all the way down to the database and back again.

These two test suites were slow. The first one took around 50 minutes on the build machine. The team decided to avoid running it in the CI on every push and ran them only at merge time. The second one took a little more than 20 minutes. Suffice it to say that this was too long to run the tests after every code change.

The company decided to invest a few days of my presence to speed up the tests. I worked on the PhpUnit test suite and managed to reduce the time by a factor of 5.

Having an impact on the entire test suite meant looking for a global solution and avoiding trying to edit every test one by one to make them faster. That would just take too long.

Parallelization with Paratest

So I went with parallelization. Instead of running each test one after the other, let’s run multiple tests at the same time, taking advantage of the multiple processors of our computers.

In PHP, we can use Paratest, a tool that will run multiple tests from PhpUnit at the same time.

The first step was to install Paratest with composer require --dev brianium/paratest. Then, run it with vendor/bin/paratest.

It worked; tests were running in parallel, but this was just a first step because a lot of tests started to fail.

The vast majority of tests were tied to the database, which is acting as a shared resource. This was the main cause of the new failures. Because the database is shared and multiple tests are running in parallel, they interact with each other via the database. Imagine that test A is trying to read some data from the database while test B deletes that data. If A passes before B, the test will pass, but if B passes before A, test A will fail.

I've explored multiple solutions to the problem of the shared database for parallel tests in a past article (in French). Some of them were not applicable in our case.

Removing the dependency on the database during testing would require too much work because the code had a tight coupling with the database via heavy usage of an ORM. Using a fake for the database wasn't possible.

Decorating each test with a transaction wasn't possible either because the framework was tweaked and used multiple connections to the database side-by-side.

The solution left was to use one database for each process. Each one of Paratest's runners will have its own database, which will prevent tests from interacting via the database.

We will return to the creation of a database for each test process later, but for now, let's focus on another issue: how can the application know which database to connect to?

Connecting to the right database

When Paratest starts a new process, it injects an environment variable named TEST_TOKEN which contains an identifier for the process. This is just an integer, starting from 1 up to the number of started processes.

Let's say that test databases are created using the original test database's name suffixed with the test token. For instance, if the original test database name is test_db, the first process database is named test_db_1.

The database name is passed to the application via an environment variable DATABASE_NAME. It's now a matter of combining the information coming from the two environment variables, DATABASE_NAME and TEST_TOKEN.

To do this, I created a Symfony Environment Variable Processor, which allows us to manipulate the environment variable used in parameters.

In the application config, a database_name parameter exists and is reused for every service definition in need of knowing which database to connect to. To read the DATABASE_NAME environment variable, it was originally defined as:

parameters:  
    database_name: '%env(DATABASE_NAME)%'

I changed it to:

parameters:  
    database_name: '%env(testdb:DATABASE_NAME)%'

The addition of testdb: indicates to the Symfony container to use the TestDBEnvVarProcessor to process the environment variable.

Here is the environment variable processor class:

final class TestDBEnvVarProcessor implements EnvVarProcessorInterface  
{  
    public function getEnv($prefix, $name, \Closure $getEnv)  
    {  
        $env = $getEnv($name);  
  
        $testToken = $getenv('TEST_TOKEN');  
        if($testToken === false) {  
            return $env;  
        }  
  
        return $env . '_' . $testToken;  
    }  
  
    public static function getProvidedTypes()  
    {  
        return [  
            'testdb' => 'string',  
        ];  
    }  
}

The getEnv method is there to transform the original environment variable to the test database name.

$getEnv($name) gives us the value of DATABASE_NAME, test_db in our case.

Then, we try to see if a TEST_TOKEN environment variable is defined. If it's not the case, we keep the database name as it is. If we have a TEST_TOKEN, we concatenate the original name with the token.

And we need to tell the framework where to look for the environment variable processor with a new service declaration.

services:
	XXX\YYY\TestDBEnvVarProcessor:  
	    tags: ['container.env_var_processor']

The process-scoped database will now be used when running with Paratest, and the original test database will keep being used if tests are run with PhpUnit.

A simpler and better way to achieve this

While I was writing this article, it occurred to me that a simpler solution to this problem exists: we can rewrite the DATABASE_NAME environment variable right in the PhpUnit bootstrap file.

$testToken = getenv('TEST_TOKEN');  
  
if($testToken !== false) {  
    $db = getenv('DB_NAME');  
    $testDB = $db . '_' . $testToken;  
    putenv("DB_NAME=${testDB}");  
}

I don't know why I didn't use that solution at that time. Maybe I tried it, missed something, failed, and found the other solution or just didn't think about it.

Anyway, I think this is a better solution for multiple reasons:

It's less and simpler code.
It doesn't require modifying the application; the environment variable modification is made outside of it. This is better because the application should just take an environment variable and trust that it was provided with the right information.
Because the environment variable modification code is written in the test framework bootstrap, it is run only in test mode. Only when it's needed and no risk of issues in production.

If you have to choose between the two solutions, I think you should pick this one. I included the other solution in the article because this is how I actually did it for that project and to share the existence of Symfony's Env Var Processors. Using them for this use case looked like a good idea but maybe isn't after all.

Creating the test databases

So far, the only test database was created using a migration script and some data seeders during PhpUnit bootstrap.

We could reuse that data setup mechanism for every database. The only issue is that it was slow. When you launched the tests you had to wait for a little while - I don't recall exactly but I would say around 1 minute - before the tests started to be executed.

We wanted to avoid each process from waiting for its own migration to be made, and I found a solution to that problem.

But first, a little detour.

Paratest, PhpUnit, and the bootstrap file

When tests are run with Paratest, the PhpUnit bootstrap is called multiple times:

Once before starting Paratest processes
Once per process

You can see that by adding this line to your PhpUnit bootstrap:

file_put_contents(__DIR__ . '/log.txt',  "In phpunit bootstrap.php ${testToken} \n", FILE_APPEND);

After running Paratest with two processors, using ./vendor/bin/paratest -p 2, and 3 test classes, you'll see the following in log.txt:

In phpunit bootstrap.php    
In phpunit bootstrap.php 1   
In phpunit bootstrap.php 2
In phpunit bootstrap.php 1

It took me a while to discover that. I was facing weird behaviors that I didn't understand. I tried adding some echo statements, but because they were in other processes, I couldn't see them. I tried debugging, which is a super fun thing to do because you need to pass the debug settings to processes started by Paratest^[1].

Why was this detour interesting?

We've learned that the bootstrap file is run once per process, which means we can set up each database's data in its own process. This has a couple of advantages:

We don't need to know how many processes Paratest will use prior to creating the databases. We will always create the right number of databases.
The setup for each database is made in parallel, which means faster setup than a sequential one.

Also, we've learned that the bootstrap file is called once per test class. We can see that thanks to the repeated In phpunit bootstrap.php 1 line.

PhpUnit's bootstrap seemed to be a good place to create the databases for each processor. Even better, because we were going through the bootstrap for each test class, we could have a new database for each one of them. This is great because I was also on a side quest of removing test flakiness due to coupling via the shared resource that is the database. But, first, let's create multiple databases.

Creating multiple test databases

Below the pseudo code of my original try at creating all the databases:


function testDBName(): string {  
    return 'test_db_' . getenv('TEST_TOKEN');  
}

if (getenv('TEST_TOKEN') === false) {
	// Migrate and seed with original database name
}
else {
	// Migrate and seed with testDbName()
}

If we had no TEST_TOKEN set, either because we were not using Paratest or because we were in the first bootstrap, we run the migration and seeding as we did before. If the TEST_TOKEN was specified, migrate and seed the specific database.

This solution comes with a couple of issues:

Even when we are using Paratest, we are creating the original database we won't need during the test execution. This is a loss of time.
Worse, because the migration and seeding step is slow, the test suite execution time is now longer than it was before as that step is repeated for each and every test class. Running that in parallel doesn't compensate for that at all.

From slow to fast...

So here we are, with tests running in parallel and a test suite slower than it was before because of database seeding.

By chance, the other test suite, the one running on CucumberJs, offered me a solution. To improve the setup of tests for that test suite, the team had already tried an idea to improve database setup time.

Instead of running migration and data creation for each test, they created a dump of the database and reused it for every test. I could totally steal that idea and reuse it to speed up the PhpUnit test suite as well.

I changed the PhpUnit bootstrap file to migrate and seed the database and dump its content to a file when the TEST_TOKEN environment variable is missing. This ensures that we always have an up-to-date database including the latest schema and data before running the test. When the TEST_TOKEN environment variable is set, instead of going through the original database process, I used the dump to populate the test process dedicated database.

if (getenv('TEST_TOKEN') === false) {
	// Migrate and seed with original database name

	// Dump the content of the database
	$dumpCommand = "PGPASSWORD='${POSTGRES_PASSWORD}' pg_dump --no-owner -h ${POSTGRES_HOST} -U ${POSTGRES_USER} -d ${POSTGRES_DB} -Fc > ${DUMP_FILE}";
	exec($dumpCommand);
}
else {

	$testDbName = testDBName();
	
	// Drop the database if it exists, this prevents error when trying to create an already existing DB
	$createDabataseCommand = "PGPASSWORD='${POSTGRES_PASSWORD}' psql -h ${POSTGRES_HOST} -U ${POSTGRES_USER} -c 'DROP DATABASE IF EXISTS ${testDbName}'";  
	exec($createDabataseCommand);

	// Create the database
	$createDabataseCommand = "PGPASSWORD='${POSTGRES_PASSWORD}' psql -h ${POSTGRES_HOST} -U ${POSTGRES_USER} -c 'CREATE DATABASE ${testDbName}'";  
	exec($createDabataseCommand);

	// Restore the dump
	$restoreDumpCommand = "PGPASSWORD='${POSTGRES_PASSWORD}' pg_restore -h ${POSTGRES_HOST} -U ${POSTGRES_USER} -d ${testDbName} ${DUMP_FILE}";  
	exec($restoreDumpCommand);
}

The dump restoration trick combined with the parallel run reduced the test duration by a factor of 5. Still a bit slow to my taste to be able to TDD, but a vast improvement nevertheless. Running tests on local machines multiple times per hour started to be possible.

As you can see, while some ideas are applicable to multiple codebases, this is purely an ad-hoc solution. The solution was created with a lot of trial and error, sweat, hurray moments, and painful setbacks.

The point here is that a super slow test suite is something you can escape. You probably won't have a super fast test suite without a lot of test rework if your tests are all making database calls, but you can improve your daily life up to some point. The next steps are to decouple business logic code from infrastructure, use unit tests for the former and integration tests for the latter. This is more work and depending on the state of your codebase can be a difficult thing to do. Sometimes it can be helpful to get some assistance to go through these kinds of challenges. If you feel like your codebase or your test suite could benefit from some rework to make your life easier and speed up your delivery, let’s chat!

According to my shell history, you probably can do this via Paratest --passthru-php option. ↩︎

Tags, the Forgotten Feature of Test Runners

2024-07-23T00:00:00Z

Cet article est disponible en français.

There is a feature of test runners that we use far too little, myself included: tags.

With a bit of imagination, they can provide great services, and even change some discussions around testing and test cases organization.

What are tags?

Usually, to create groups of tests, we rely on the position of the test within our test suite. Tests are grouped because they belong to the same test class, the same describe method, the same file, or the same file hierarchy.

Tags are another way to create groups of tests without needing to know where they are stored. Some tools even refer to them as groups rather than tags.

Generally, tags are implemented through the annotation mechanism of the language in which the tests are written.

Once tags are in place, we have a new way to decide which tests we want to run or not. We will see examples of usage in the rest of the article.

My test runner doesn’t support tags

That’s unfortunate...

But chances are, the community offers a plugin that provides a similar mechanism^[1].

If not, all is not lost. There is a good chance the test runner allows filtering tests based on their name^[2]. We can then decide to include tags at the end of test names and filter on the pattern that interests us. Adding a tag to the test name adds noise, information that is not relevant in some cases, so it’s important to have a convention that makes it easy to know if information is a tag or not. Starting all tags with a hashtag (#) is probably a good idea.

Usage examples

Let’s look at some ideas of what tags can do for us.

Test type

The first idea is the most obvious, so obvious that it may seem unnecessary, yet...

Tags can indicate the type of the test. Is it a unit test, an integration test, an end-to-end test, an acceptance test, or another type of test?

With tags, we can then decide to run only certain types of tests at certain times.

This idea seems unnecessary because you may already be using test suites arranged in different folders to separate test types, and you can already decide which types of tests to run.

The downside of sorting test types by folder is that it forces us to maintain one or more file hierarchies mirroring that of the production code. And let’s be honest, most of the time this synchronization is not done. We end up with completely different folder hierarchies everywhere.

Thanks to tags, we can abandon this constraint and place all tests in the same hierarchy, greatly facilitating maintainability.

We can even consider going further and placing tests directly next to the code and still have a way to choose which types of tests to run.

And now, no more synchronized folder hierarchies to maintain, yet we can easily select which types of tests to run.

Dependencies

The second idea is a continuation of the previous one.

But first, let me ask you a few questions:

How many times have you had a debate about whether a test is a unit test? Or rather an integration test? Or perhaps an acceptance test? Maybe an end-to-end test?
How much time have you wasted arguing as a team about how to classify a test and which folder to put it in?

What if the most important thing is not the type of test but its characteristics?

Think about it for 2 minutes: why do we care so much about the type of a test?

Isn’t being able to separate slow tests from others a big part of it?

If your workflow relies heavily on tests, for example if you practice TDD, you probably don’t want to wait a long time to find out if your changes had an unexpected impact on the system. Chances are you frequently run the fastest tests as a first safety net, and occasionally run all the tests, even the slowest ones, for a more comprehensive safety net.

So rather than debating the type of tests, let’s ask ourselves why they are slow. Generally, tests are slow when they involve I/O operations. Tests that touch a database, a file system, or make calls to external services via the network are much slower than other tests^[3].

We then have a new idea for tag categories: tags that indicate dependencies.

When a test needs a particular dependency, we can indicate it with a tag, such as database, external-call, network, filesystem, etc.

With this type of tag we no longer have to try to precisely categorize tests; we just describe the dependencies they need, avoiding endless debates.

We can now select or exclude certain tests based on what they depend on. When we need to run only the fastest tests, it’s easy to exclude troublesome dependencies.

As a bonus, if a dependency is missing in an environment, for example if we can’t make calls to external services because we’re on a train, we can easily exclude all tests that won’t pass anyway.

This idea of selection by dependency, which is unmanageable with the file system, becomes possible with tags.

Link with the ticketing system

Tags can also link to a ticketing system.

With the ticket number as a tag, we can easily find a non-regression test for a bug bug-3457 or find all tests related to a user story US-89.

I don’t think these tags are very useful in the long term because after a while knowing that a behavior was created in connection with a ticket is no longer very relevant in my opinion, but they can still be useful in the short term.

For example, the build tool could generate specific documentation based on the tests for each user story and display it directly in a pull request comment^[4] or check that a non-regression test has been added before resolving the bug.

Nomenclature

As with any tool, overuse can have unpleasant consequences.

To prevent the use of tags from becoming a huge mess it is important to think about a nomenclature. To do this, ask yourself what you can get from using tags, whether to select tests to run or to extract information from them.

Once you have defined what tags will be used for and identified categories you can use prefixes to easily identify tags.

We encountered some ideas in this article:

feat-payment
US-89
bug-3457
tool-backoffice
market-es
domain-delivery

Also, remember to clean up occasionally and remove tags that are no longer useful - especially if you use tags linked to the ticketing system.

For this, let your test runner help you. If it allows filtering tests by group, it probably has a command to list tags. For example, PHPUnit has a --list-groups command. If the test runner doesn’t help, you can probably manage with grep.

Some tags will have more or less long lifespans by their nature, your usage will likely change as well, and that’s okay.

I hope this article has given you ideas, whether the ones presented or others, on how you can incorporate the use of tags into your daily routine to get the most out of your tests.

And if you want to make sure you have tests you truly enjoy working with, you can access my video training on improving automated tests, in French. You can also schedule a meeting so we can work together to take your team to the next level.

For example, jest-runner-groups if you use Jest. ↩︎
Jest’s testnamepatternregex option can help. ↩︎
Dare I simply say that these tests are much slower than unit tests? ↩︎
If your workflow uses PRs, of course. ↩︎

3 and a half heuristics to avoid putting business logic in the wrong place when doing event sourcing

2024-08-09T00:00:00Z

One mistake that is pretty common to people new to event sourcing is putting business logic in the code responsible for rebuilding the state that will serve to make a decision ^[1]. Any decent introduction to event sourcing will warn people against that, but still, this is a common mistake.

The issue here is that you still need some logic to rebuild the state, and it’s easy to confuse that logic with business logic. We tend to fall into the trap of confusing the two when the computation seems stable and doesn’t change often. After all, why should we put that piece of information in an event when we can recompute it during the rebuild?

I want to offer three ideas that can help distinguishing business logic from state rebuilding logic.

Did a business expert tell you something about that piece of logic?

Is the code that you’re about to write interesting for a business expert? Is this something that you discussed with them? Did they tell you something about that specific rule / number / ... ?

In that case, this is business logic, and it should go in the command part of your system, and the results of any calculation should be stored in an event.

Let’s take two examples. The first one is a common one, the second one comes from a team I’m working with.

The first one concerns tax rates. For now, to get the price with all taxes included, you need to multiply every price by 1.2. Easy peasy, why not include that in the rebuilding state?

Well, what if the tax rate changes? Does this mean that suddenly, all products you sold with a 20% VAT were actually sold with a 10% VAT? Of course not!

The reconstruction phase will have to include some more logic to deal with the tax rate change, such as figuring out the right tax rate based on the date of the event.

The second example concerns an expiration date. Imagine that buyers can add items to their cart for 20 days after the basket was created. Since we know the date of the BasketWasCreated event, we can easily compute the expiration date in the reconstruction phase. This is as simple as taking one date and adding a duration.

But here again, what if the business experts come to you and ask you to change the duration, reducing it from 20 days to 10 days? Should you say no to all users who believed they could act after 20 days, only after 10 days, now that the rule has changed? Probably not.

These rules come from our business experts. They told us about the tax rate and duration. The computation should go into the command part of the system, and the result should be stored in the event.

With the two examples, we also saw that other interesting questions to ask yourself are: Is it possible that this business rule might change at some point? And if so, what is the impact? But as I said in the introduction, the trouble starts when we believe that the logic is stable, which makes this guideline less interesting than just looking at who told you about that logic.

Let’s move to another idea.

It should be about state representation, nothing more.

Now that we’ve seen a heuristic for detecting what should go in the handling command part of our event-sourced system, let’s look at what should go in the reconstruction phase.

The reconstruction phase consists of applying all past events to rebuild a state that will help to make future decisions. This is nothing more than a particular live projection.

The logic in the rebuild state should only be about data structures. If you can find multiple ways to represent the information you are probably on track.

For instance, let’s imagine that we have a (probably stupid) business rule that prevents people from buying more than 10 products at once. In the rebuild phase we could take the number of products added in the basket when we apply each ProductAddedToBasket event and sum that, having only an integer in the state. Alternatively, we could rebuild the entire basket with a collection of all products and use products.length when we handle the command to add a new product to the basket.

We found two ways of representing the state. And, by the way, business experts don’t care which one we use. This is clearly reconstruction logic.

Are you doing computation on one event only?

Another idea that can help prevent the mistake of putting business logic in the wrong place is to look for areas where computation is being performed in the rebuild phase based on data contained in only one event.

Let’s retake all previous examples:

To compute the prices, including taxes, we would only need data from one ProductAddedToBasket event. We are not making calculations across multiple events, just from one.
To get the expiration date, this is the same thing. We are only doing calculations using data coming from the BasketStarted event. That logic should go in the command handling part of the system.
To have the entire quantity of products added to the basket, we need to look at multiple ProductAddedToBasket events. This looks good, and the logic belongs to the rebuild phase.

Be careful; this heuristic comes with some limitations. Depending on how you represent the information in the events, you could need to do some computation based only on the data from one event.
For instance, you could store the tax-free price and the tax rate in the ProductAddedToBasket event. In that case, you would need to recompute the price, including VAT, based on these two pieces of data.
The same is true for the other example. Instead of storing the expiration date, you could store the start date and the duration before expiration.

As we can see, this is not a definitive rule; it is just something to look for.
That heuristic could be refined to watch out for logic using data not coming from events.
Or, in yet another form: Be very suspicious about every constant value in the rebuilding logic.
Look for hardcoded values, even hidden behing constants and ask you if it that could go in the command handling part.

We probably can find other clues that we are performing business logic in the rebuild state. These are mine - I might even have other unconscious ones; who knows? - and I hope that they will be useful to you.

When rebuilding the aggregate, if you will. ↩︎

Reading a story across multiple causal links without losing your mind

2024-08-30T00:00:00Z

I often find it difficult to read multiple causal links in a row in causal loop diagrams, a tool used in System Thinking to describe the relation between elements (or stocks).

Unfortunately for me, best stories often include more than one causal link between two stocks, or we wouldn’t need a tool like causal loop diagrams to understand systems. I want to share the trick I use to tell stories that make sense, involving multiple jumps without losing my mind and inverting some relationships.

Recap on reading causal links

Let’s start with a quick recap on reading one causal link in diagrams.

A line between two stocks represents an impact.
One line from one stock to another means that both stocks move in the same direction.

---
className: diagram-25
---
strict digraph { 
  rankdir="LR"
  A -> B
}

Here, more of A leads to more of B, or less of A leads to less of B.

But sometimes stocks move in opposite directions. Multiple notation exists, some with a dot (●), others with a minus (-). For the rest of the article, I’ll keep with the dot notation as I find it easier to read.

So, a line with a ● means the stocks move in opposite directions.

---
className: diagram-25
---
strict digraph { 
  rankdir="LR"
  A -> B [label="●"]
}

Here, more of A leads to less of B, or less of A leads to more of B.

Chaining multiple causal links

When reading a causal loop diagram with multiple links, the trick is to stop trying to be clever and read each link on its own, one after the other, and, instead, read them all in one go by switching from "more" to "less" every time you encounter a ● as you are telling the story.

---
className: diagram-25
---
strict digraph { 
  A -> B
  B -> C [label="●"]
  C -> A
}

You can read the diagram above as "more of A leads to more of B, more of B leads to *switch* less of C, and less of C leads to less of A" or "less of A leads to less of B, less of B leads to *switch* more of C, and more of C leads to more of A."

That trick even works when the loop goes across multiple inversions.

---
className: diagram-25
---
strict digraph { 
  A -> B
  B -> C [label="●"]
  C -> A [label="●"]
}

Here, "more of A leads to more of B, more B leads to *switch* less of C, and less of C leads to *switch* more of A" or "less of A leads to less of B, less of B leads to *switch* more of C, and more of C leads to *switch* less of A."

I hope you’ll find this helpful the next time you try to tell a story involving feedback loops!

Collecting data via Google Form and Apple Shortcuts

2024-09-06T00:00:00Z

A while ago, I started tracking my weight using a Process Behavior Chart (PBC) but stopped, partly because the weighing machine was out of battery, I didn’t get the result I wanted, and because the app I built with AppSheet took too long to load, which was annoying.

Now that I have repurchased some battery, I’m up again for the challenge of losing some weight, and I want to give another try using a PBC to see any change, but I don’t want to use AppSheet again and decided to use an Apple Shortcut instead to collect the data easily.

So here we are: how to set up Google Form and an Apple Shortcut to collect measurements right from your phone to a spreadsheet.

Step 1. Create a Google Form

Head to Google Form homepage, click "Access to Forms", then create a new form.

Next, add as many answers as you need.

In my case, I needed one single answer, a decimal number greater than 0.

Depending on your phone language settings, the Apple shortcut might send decimal numbers formatted using a comma but Google Form won’t accept that as an answer. Instead of using a number input, use a Regular Expression with ^-?[\d*\,{1-9}]*$ as validation rule.

Update: Alternatively, you can change the spreadsheet settings to simplify the process. In Google Sheet, go to "Files > Parameters" and select "United-States" for Regional Parameters.

Part 1, done!

Step 2. Prepare the form submission URL

In the form configuration panel, click on the three dots on the top right of the page and click "Get a Pre-Filled link". A page opens. A Pre-Filled form is a way to share a form pre-filled with some information to someone else, to help them input the form faster or for reviewing data before submission.

You will see the form displayed. Enter some data easily, distinguishable. I entered 100. This data will not be added to the results when you submit.

Click "Get the link" and copy it. This is the URL we will use to submit new data from shortcut to the form.

Here is the link I got:

https://docs.google.com/forms/d/e/[REDACTED]/viewform?usp=pp_url&entry.1061001024=100

Look closely, we can see the 100. This is the value we will replace with the each measurement.

Edit the URL to be able to send the new measurements:

Replace viewform with formResponse
Add &submit=Submit at the end
Replace the dummy value with ``

You should get something like https://docs.google.com/forms/d/e/[REDACTED]/formResponse?usp=pp_url&entry.1061001024={value}&submit=Submit

Step 3. Setup the Apple Shortcut

On your phone, or directly from laptop, if you have a mac, open the Shortcut app and create a new shortcut.

Then:

Add the "Ask for input" step and set it up to receive numbers. In the options tab, allow for decimal numbers.
Add the "Text" step and copy the form submission URL template created in Step 2.
Add the "Replace text" step and replace {value} with the data from the input on the text.
Add the "Get content at URL" step and use the updated text as the URL.

Your shortcut should look roughly like this

And you should be all set. For convenience, add the shortcut to your home screen for direct access.

To input a measurement, click the shortcut icon, enter a value, press "OK", and you’re done.

A simple trick, just what it takes of development to make life easier. This is a nice alternative solution.

In the following articles, I’ll explain how to determine if data variation is meaningful (whether you actually lost or gained weight) using a Process Behavior Chart, and how to build a PBC.

Create a new file at the right place with prefilled content thanks to Jetbrains IDEs

2024-09-13T00:00:00Z

It’s been a while since I felt that pain...
I had to manually create a file for every blog post in the correct directory, repeating information based on the title in snake case, using the current date, with the same structure that I copied from one old post to the new one and manually edited.

That’s just too long, or at least super interesting and error-prone.

I wanted to have a template for my blog post article. My articles are written in an index.md stored in directories including the current dates and using the article’s title in snake case.

For instance, the article you’re currently reading is stored in /articles/2024-09-13-create-a-new-file-at-the-right-place-with-prefilled-content-thanks-to-jetbrains-ides/index.md

Creating the directory and file, setting up some information about the permalink, and linking to the social image—all this after converting the article title to snake case—could probably be easier.

For my Burritalks project, I created a small CLI in node that helps me set up pages faster. I could do the same here, but a question popped into my mind:

Could Jetbrains IDE automate part of this?

Of course!

Here is what you can get:

I tested that on Webstorm, but I’m pretty sure it would work on all their other IDEs. You can try that on Intellij, Rider, PhpStorm, PyCharm, Goland, ...

Here is how it works:

In your IDE settings, go to Editor > File and Code Templates.

Create a new template by clicking on + and giving it a name. In my case, I need to create a new blog post article so I went with Article.

Automating the content of the article

The File template system makes use of Apache Velocity template, and comes with a set of prepared variables.

In my case, I needed ${YEAR}, ${MONTH}, and ${DAY} for the first part of my article path.

You can also add custom variables. If the IDE finds a variable it doesn’t know when the template is applied, it will ask for a value. For example, I defined a $TITLE variable to get the title of the article.

Even, better, you can run some functions on your variable. A good thing when you want to turn the title to snake-case !

I defined a new variable, $TODASH, based on the $TITLE one, which makes it lowercase and replaces spaces with a hyphen: #set( $TODASH = $TITLE.toLowerCase().replaceAll(" ", "-")).

My complete template looks like so:

#set( $TODASH = $TITLE.toLowerCase().replaceAll(" ", "-"))
---
title: ${TITLE} 
description: ""
permalink: /articles/${TODASH}/
socialImage: /articles/${YEAR}-${MONTH}-${DAY}-${TODASH}/social.png
language: en
tags:
  -
---

This is already super cool and saves some time, but I still needed to create the directory with the date and a snake-case version of the title.

It turns out that can be automated as well!

Automating the directory creation

The File Name input runs the template systems, which means we can use Apache Velocity variables and functions there too.

Here is the file name I defined :

src/articles/${YEAR}-${MONTH}-${DAY}-$TITLE.toLowerCase().replaceAll(" ", "-")/index

See how it uses the same ideas as in the body template

The extension, .md, must be set in the Extension input to avoid doubling it.

Now, I just need to add a new "Article", type the name, and I get the index.md file right at the right place and part of the front matter data already setup.

And it’s time to start typing!

A quick introduction to Process Behavior Charts

2024-09-26T00:00:00Z

On this blog, I’ve already spoken about Process Behavior Charts (PBC), mostly to share that I wrote a little Christmas story to explain what they are and how they are used, but I never explained how to build one. That was the original plan for this article, but as I explained what PBCs are and why you might want to use them, the article turned into a very long piece, and I think it’s better to split it into two. The article about building the PBC is here.

What are Process Behavior Charts and Why Are They Useful?

Have you ever looked at some data and seen it going up or down?
What do you do when it goes up? What do you do when it goes down?
Do you react whenever a data point doesn’t look like what you hoped it would be?

Here’s the good news: you should stop reacting to every data point.

In every data set you look at, you’ll see some variation, but not all variations are equal. In every process, some variation is expected; this is routine variation. Reacting to routine variation is just a waste of time because you can’t do anything about it. If you are unhappy with what you see, you must change the system producing that data. On the other hand, from time to time, something unexpected occurs—something outside the usual way of working pops up. Here, you should react and figure out why something unfamiliar just happened. You want to learn about it. Did it produce something better than usual? Why is that? Can we change our system to have more of that? Was it bad? How can we change the system to have less of that?

This leads to a question: How can you sort routine variation from special-cause variation? How can you know when to react?

Process Behavior Charts are tools designed to do just that. They help distinguish signals (special cause variation) from noise (common cause variation).

These charts were pioneered by Walter A. Shewhart in the 1920s and later popularized by W. Edwards Deming and Donald J. Wheeler, who further refined their use in quality management.

How Do They Work?

The idea is to draw a graph (actually two, but I’ll only show you how to build the first one) displaying your data alongside two limits: the Upper Natural Process Limit (UNPL) and the Lower Natural Process Limit (LNPL). You don’t get to choose what these limits are; they are calculated from the data available. They are an indication of how your system is working, not how you would like it to be.

With these charts come a few rules for detecting signals. Something interesting is happening if you see:

Rule 1: Any single point outside of the natural process limits.
Rule 2: Eight consecutive points on the same side of the central line.
Rule 3: Three out of four consecutive data points that are closer to the same limit than they are to the central line.

These are the rules shared by Mark Graban in his book Measures of Success. Note that other rules exist and that you might encounter them if you continue digging into this subject.

How to Compute the Data

Collect the data. You want to make sure that you have a defined process for data collection to ensure that it’s always done the same way. You should probably describe that process alongside your graph or at the place you are collecting the data.
Make sure the data is time-ordered.
Choose a baseline period of at least 6 points and less than 20 that you think properly characterizes your process. Avoid using data if you know something uncommon occurred at the moment of measurement.
Plot the data as a running record.
Plot the central line. This is the arithmetic mean of your baseline point values (BaselineAverage).
Calculate the moving range. This is the absolute value between two successive points (BaselineMovingRangeAverage).
Calculate the Natural Process Limits:
- UNPL = BaselineAverage + (2.66 * BaselineMovingRangeAverage)
- LNPL = BaselineAverage - (2.66 * BaselineMovingRangeAverage)
Plot the NPLs on the Running Record.

And here is the kind of chart you should see:

I’ve started working on automating signal detection, and you can see a Rule 1 signal with one point above the UNPL.

In the next article, I’ll share how to build one in Google Sheets based on the data collected via Google Forms and Apple Shortcuts.

Want to Know More About PBCs?

This is a super, super short introduction to PBCs. You can find some really good resources out there that will dig deeper than I will, with better explanations and details. I was primarily interested in showing my entire workflow from data collection on my phone, PBC calculation, and PBC displayed on the phone, and raising a little awareness about this tool.

Donald Wheeler’s Understanding Variation is still sitting on my bookshelf, but I’ve heard it’s an interesting read and the source of learning for the authors of the next resources I’m about to share.
I learned about PBCs from Mark Graban’s book Measures of Success. This is an easy read that clearly demonstrates why most of the time we use numbers quite poorly, leading to overreaction, pressure, and no real improvement.

If you’re working in an Agile setting or using Kanban, Dan Vacanti’s Actionable Agile Metrics Volume II shows how to use PBCs for flow metrics.

If you are looking for a super good free resource, the team at CommonCog created a tool to create PBCs and offers a free one-week course over email that is super interesting along with articles.

Building a PBC in Google Sheet

2024-10-04T00:00:00Z

In the previous article, I shared what Process Behavior Charts (PBC) are and promised to explain how to build one in Google Sheets, using the data collected from an Apple Shortcut. This article will cover that. Let’s get started.

The article on collecting the data left us with a Google Sheet containing one sheet of data with all the measurements collected from the shortcut. You can access that spreadsheet from Google Forms by clicking on the "Responses" tab.

You can rename the first sheet, containing the measurements, to something that suits you better than "Responses." I decided to rename it "Measurements."

You won’t need to touch that sheet anymore; we will work on another sheet for the rest of this article.

Do not work on the sheet made for collecting data

When working with spreadsheets, having a sheet that contains raw data helps with maintainability since you won’t be tempted to move columns around and break formulas elsewhere. If you’re into Software Architecture, you can view that as your write side in the CQRS pattern, and we will use other sheets for the read sides.

Fetch data from the form submissions sheet to build the PBC

Create another sheet where we will compute the PBC data for the measurements. I named mine "PBC." You can’t make it more straightforward than that.

In that sheet, get the data from the other sheet with the ={Measurements!A2:B} formula in the A2 cell.

In the first row, in cells A1 and B1, add titles for the two columns. I went with "Date" and "Measurements."

Why aren’t we getting all the data from the reference data sheet ?

Why not get the value directly from the A1 cell to get the column’s name from the reference data sheet?

In the A1 cell, we could use ={Measurements!A:B} to get all the data directly from the Measurements sheet. That seems like a good idea, but it actually isn’t.

Google Forms keeps track of how many submissions were received and adds new submissions to the current count + 1 row. This is probably to avoid overwriting past values.
During the setup process of the Apple Shortcut, you will be tempted to test it and see if everything works as expected. Unfortunately, due to Google Forms' ways of working, you won’t be able to insert real, non-test data starting from the first row.

Using the trick above, you can select the row at which your PBC starts reading real values.
If the real values submitted from the form are inserted in row 17, you can edit the formula in the A2 cell to reflect that, using ={Measurements!A17:B} to get only the data you really care about in that sheet.

You should now see values from the form in that spreadsheet. Each new value submitted via the form will end up in that sheet.

Compute the moving range

In the first row of the C column, write "Moving Range."

The moving range is the absolute difference between one measurement and the one before. We can’t have a moving range value for the first measurement, as no previous measurement was made. In the C3 cell, add this formula: =IF(ISBLANK(B3), "", ABS(B3-B2)).

ABS(B3-B2) computes the difference between the second value and the first one and makes it a positive number.

The other part of the formula prevents incorrect values from being displayed in rows without any measurement yet by leaving them blank. We will use this trick for all the other columns.

Use the blue dot and drag down to set the formula for the following rows^[1]. I went down to row 100.

Compute the average

In the first row of the D column, write "Average."

I decided to go with 20 points for my baseline. To do the same, add =IF(ISBLANK(B2), "", AVERAGE(B$2:B$22)) to the D2 cell.

Again, use the blue dot and drag down to set the formula for the following rows.

Compute the average moving range

In the first row of the E column, write "Average moving range."

For a baseline of 20 points, we need to compute the average of the first 20 moving ranges.

To do this, add =IF(ISBLANK(B2), "", AVERAGE(C$2:C$22)) in the E2 cell and use the blue dot once more to apply the formula for more rows.

Compute the Lower Limit

In F1, write "Lower limit", add the =IF(ISBLANK(B2), "", D2 - (3 * E2 / 1.128)) in F2, and use the blue dot.

Compute the Upper Limit

In G1, write "Upper limit", add the =IF(ISBLANK(B2), "", D2 + (3 * E2 / 1.128)) in G2, and use the blue dot one final time.

Now, you have all the data you need to draw your XChart, the first of the two types of charts usually included in PBCs, and the only one we will build together.

The other chart, the mRChart, helps detect signals of unusual variation between two consecutive measurements.

You’ll often see the pair of the two charts referred to as XmR Charts.

Draw the graph

Select columns A (Date), B (Measurement), D (Average), F (Lower Limit), and G (Upper Limit), and insert a new graph.

You’ve drawn your first XChart with Google Sheets. Congrats!

It’s probably unreadable, though. Please take a few minutes to adjust the settings and improve it.

First, zoom in to view your data. In the vertical axis settings, add a minimum value.
Also, change the graph title to give it a better name.

Now that we see more than just a group of compacted lines, we can customize the series.

For the Measurement series:

Make it black
Add the data labels

For the Average series:

Make it blue

For both the Upper and Lower Limits series:

Make them red

And now you have a nicely displayed PBC, or more precisely, the XChart part of it.

If someone knows a better way to set a formula for all following cells in a column, I’d be more than happy to hear it. ↩︎

Run Laravel Artisan Tests directly in PhpStorm test view

2024-10-11T00:00:00Z

One of the teams I work with uses Laravel and enjoys running tests via PhpStorm. Unfortunately, it’s not possible to run Laravel tests using php artisan test directly within the IDE’s test interface. This forces us to run tests slower than with the CLI because they don’t run in parallel.

You might think it’s possible to run Laravel tests using Paratest directly in PhpStorm, but this doesn’t work either due to how the database cleanup system is implemented—at least, that was the case the last time I tried.

Since we prefer PhpStorm’s interface for running our tests, I wondered if it was possible to run Laravel tests directly inside PhpStorm’s test interface, and it turns out that it is—with a bit of a hack.

Get the Wrapper

Copy and paste the following code into a file named phpunit somewhere in your project. In my case, I put it in tooling/PhpstormLaravelTest/phpunit. The file needs to be named phpunit because Paratest, used when you want to run multiple test classes at once (from the same directory, for instance), checks that specific filename.

#!/usr/bin/env php  
<?php declare(strict_types=1);  
  
// This will need to be named `/phpunit` if you want to be able to use that tool alongside paratest for phpstorm,  
// for instance when you are running all tests in a directory.  
// The best thing I came up with so far is to create a directory `PhpStormForLaravelTest` and create that file inside.  
echo "👋 Laravel Wrapper for Phpstorm PHPUnit 10.5.15-Fake. For the real PhpUnit and Paratest versions see below.\n";  
  
// Artisan test doesn’t have a --filter argument, we need to use phpunit when running only part of the tests.  
if (in_array('--filter', $argv)) {  
    $command = 'php vendor/phpunit/phpunit/phpunit';  
  
    // Add single quote for every argument to avoid escaping issues that prevent  
    $args = array_slice($argv, 1);  
    $args = array_map(fn($a) => "'${a}'", $args);  
  
    $command .= ' ' . implode(' ', $args);  
} else {  
    $command = 'php artisan test --parallel --teamcity';  
}  
  
echo "🔍 Running {$command}\n";  
  
passthru($command);

As you can see, this is clearly a hack. There is no guarantee it will work in every situation. It’s a wrapper on top of a wrapper that uses a library wrapping the testing framework. You have to account for Paratest specifics (naming the file /phpunit) and artisan test specifics (switching to PHPUnit when the --filter option is used because otherwise it doesn’t work).

What this script does is simple: when there are no options, it uses php artisan test --parallel to execute tests as fast as possible, while asking for TeamCity-compatible output that PhpStorm can parse and display in its test view. When you’re trying to run a single test or test class, PhpStorm will use the --filter option, and in that case, we redirect to PHPUnit.

A big hack, but one that serves us well.

Configure PhpStorm

Once installed, you need to configure it in PhpStorm.

In PhpStorm settings, go to PHP > Test Frameworks.

Add a new test runner by clicking the + and selecting PHPUnit by Remote Interpreter.

Select the interpreter that will run the tests. If you already have a test runner configured to use that interpreter, you’ll have to remove it, as PhpStorm only allows one test runner per interpreter.

Configure the interpreter to use a .phar file and point to the wrapper file with the correct mapping. It should look something like /var/www/html/phpstorm-laravel-test-wrapper.php. Click the reload button to check if it works. You should see a fake PHPUnit version displayed.

In your test run settings, ensure you’re using the correct PHP interpreter—in our case, laravel.test.

And there you go! Enjoy running your tests with speed and ease in PhpStorm 🏎️.

Conclusion

Using PhpStorm’s test interface with Laravel offers a much better developer experience, but it requires some clever configuration hacks. By setting up this wrapper, you can enjoy the best of both worlds: the speed of php artisan test --parallel and the flexibility of PHPUnit for individual tests.

Streamlining Development with Custom Shell Functions: Reusing a command with a custom config for each project

2024-10-18T00:00:00Z

As a freelance technical agile coach, working across multiple projects means dealing with a variety of tech stacks. Every client has their own preferred tools, some with Docker, some without, and often there are custom setups or hacks. Over time, managing these different workflows can become mentally exhausting. You don’t want to waste time remembering specific commands for each environment or tool every time you start a project.

To solve this, I’ve created custom shell functions to simplify and streamline my workflow. These helpers sit on top of the usual tools, allowing me to invoke commands without remembering intricate details or worrying about specific configurations for each project.

Here’s how I did it.

Example: Custom Shell Function Command for PHPUnit

Here’s an example of a custom bash function I’ve written for PHPUnit. This function is placed in my shell alias file.

phpunit() {
  if [ -f "./my-conf" ]; then
    cmd=$(grep ^PHPUNIT ./my-conf | awk -F"PHPUNIT=" '{print $2}')
  fi

  if [ ! -z "$cmd" ]; then
    eval $cmd $@
  elif [ -f "my-phpunit.xml" ]; then
    vendor/bin/phpunit -c my-phpunit.xml $@
  else
    vendor/bin/phpunit $@
  fi
}

Breaking it down:

Check for a custom configuration file: The function first looks for a file called my-conf in the current project directory. This file can override the default behavior by specifying custom commands.
Extract custom PHPUnit command: If my-conf contains a line starting with PHPUNIT=, the script grabs that line and uses awk to extract the command that follows. This allows me to define a custom PHPUnit command per project if necessary. For instance, some projects require running PHPUnit inside a Docker container; I just don’t want to have to think about that.
Execute the custom command if found: If a custom PHPUnit command is found, it is executed using eval.
Fallbacks to default behavior:
- If no custom command is found, the function checks for a my-phpunit.xml configuration file and runs PHPUnit with that. I don’t remember why I put that in, as it supposed to be the default PHPUnit behavior. No code is perfect, and at least you get the idea if you want to include some configuration in your command by default.
- If neither is available, it runs the default PHPUnit command from the vendor folder.

This setup allows me to run PHPUnit across multiple projects without having to remember specific configurations for each one.

I’ve created such function for every tool that is reused accross project

The `my-conf` File: Customizing Behavior for Each Project

In each project, I can create a my-conf file that overrides the default behavior for specific tools. This file is included in my global .gitignore, so it doesn’t get committed to version control.

Here’s an example of what a my-conf file might look like for a Laravel project using Sail and Docker:

PHPSTAN=sail php ./vendor/bin/phpstan --memory-limit=-1  
RECTOR=docker compose -f my-docker-compose.yml run --rm -it rector ./bin/rector  
COMPOSER=sail composer  
PHPUNIT=./bin/phpunit

In this example, I have custom commands for various tools like PHPStan, Rector, Composer, and PHPUnit. This file allows me to run these commands seamlessly without having to think about whether I need to use Sail or Docker in a specific project.

Why Use Custom Shell Functions?

Using custom shell functions gives me several key advantages:

Consistency: I can run the same command across different projects without worrying about project-specific details.
Efficiency: With these helpers in place, I don’t have to spend mental energy remembering how each tool is configured in each project.
Flexibility: I can easily adapt to new client setups by modifying the my-conf file without touching the global setup.
Collaboration: Teams can have their own configurations, while I keep my personalized workflow without impacting theirs.

Conclusion

By creating custom shell functions for tools like PHPUnit, Composer, PHPStan, and others, I’ve streamlined my workflow significantly. These helpers give me the flexibility to work across multiple projects with different setups, while maintaining a consistent and efficient workflow.

If you’re working across multiple projects and stacks like me, this method can save you valuable time and mental energy. And if your team isn’t quite ready to adopt these shortcuts, no worries — you can keep them in your local environment without disrupting the broader team workflow.

Displaying Process Behavior Charts on iPhone with Charty and Google Sheets

2024-11-06T00:00:00Z

In this article, we’re building on our previous guide on creating Process Behavior Charts (PBC) in Google Sheets, with (data submission automated through iOS Shortcuts and Google Forms)[/articles/collecting-data-via-google-form-and-apple-shortcuts/]. Now, we’ll close the loop by bringing this data back to your iPhone, visualizing it using the Charty app. With this setup, you’ll have your real-time PBC data right at your fingertips!

Preparing Data for Export

First, let’s set up a new sheet in Google Sheets specifically for export. In your existing Google Sheets file, create a new sheet called "PBC Export". This sheet will serve as the source for data that will be published and accessible via Charty.

Over time, your PBC data may accumulate a substantial number of points. To maintain a manageable display on your phone, you’ll want to control which data points are shown. Use the PBC Export sheet to filter data from your main PBC sheet according to your needs. For instance, if you’d like to include only points after a particular date, add this formula to cell A1:

=FILTER(PBC!A1:G1000, PBC!A1:A1000 > DATE(2024,10,23))

This filter will retrieve only data from the PBC sheet for dates later than October 23, 2024. You can adjust the date as necessary, or even reference another cell to make this date dynamic.

Publishing the Data

Now that you’ve prepared your data, let’s make it publicly accessible so that an iOS Shortcut can fetch it. Follow these steps:

In Google Spreadsheet, navigate to File > Share > Publish to the web.
In the options, select the PBC Export sheet instead of the entire document, and choose the CSV format.
Google Sheets will provide a link to this CSV file – copy this link as we’ll need it in the next step.

Creating a Shortcut for Charty

With your data link ready, we’ll now create a shortcut in the iOS Shortcuts app to feed this data into Charty.

Open the Shortcuts app and create a new shortcut.
Add the following actions:
- Create a chart: Name it something relevant, like “PBC Chart.”, or "Weight" in my case.
- Get contents of URL: Paste the URL from the previous step here.
- Add series: Use the data from the “Contents of URL” action as a line series in your chart (this should be the “Chart Id” you created earlier). Assign:
  - Date for X values.
  - Measurement, Upper Limit, Lower Limit, and Average for Y values.
Add an action to Update all widgets.

To complete the setup, add a Charty widget to your home screen and link it to the chart you just created. You should now see your PBC data right on your iPhone!

Customizing Your Chart

But it’s probably not super nice to read.

To enhance the visual clarity:

For each Y value, add an action to Update Series Style. Here’s a suggested styling:
- Upper and Lower Limits: Red (#FF0000).
- Average: Blue (#0000FF).
- Measurement: Black (#000000), with markers enabled and marker size set to 2.

An example of measurement settings:

Here’s my final result with the PBC displayed on my iPhone. Looks great, right?

Bonus point: Setup an automation to refresh the chart daily

On the shortcut app, create an automation that runs every day and executes the shortcut you’ve just created.
That way you won’t have to manually trigger a refresh of the chart.

Wrapping Up

This is the final piece of that series about collecting data from an iPhone right into a Google sheet to build a PBC, and then displaying it on your phone.
This flow gives you instant access to up-to-date charts, empowering better, data-informed decisions right from your home screen.

This was fun to build.

A clean commit history with Git WipSquash

2025-01-10T00:00:00Z

Maintaining clean and meaningful commit histories in Git is crucial for collaboration and code maintenance.
Yet, the process of ensuring every commit is atomic can feel tedious, especially during active development.
I like to do super small commits, especially when I’m refactoring code or working on something I’m not super sure about, but I’m way too lazy to type a meaningful message each time.

Maybe someday I’ll write a Jetbrains IDE extension that generates commit messages after an automated refactoring, but as that doesn’t exist yet, I sometimes go with a commit containing only wip (for Work In Progress).
It takes no time to create, you can even have an alias for it, but it left you with a commit history that you are not proud of, and clearly not meaningful at all.

I wanted to find a solution and finally took some time to work on it and create a script that solves my pain point.
I’m currently experimenting with it and with that workflow, and I wanted to share it with others to see if some people find that idea useful.

Enters `git wipsquash`

Let’s talk about git wipsquash.

The idea for wipsquash is that you make an effort to say what you are doing only once, at the start.
This goes well with atomic/conventional commits.

Imagine you’re working on a feature or bug fix. You start with an intentional, well-named commit, such as:

git commit -m “Add authentication middleware”

Then, as you continue tweaking, experimenting, or debugging, you add several more commits, hastily labeled wip:

git commit -m “wip”
git commit -m “wip”

Now you’re done. This is a great, good job, but the commit history looks like this:

1. Add authentication middleware
3. wip
2. wip

Will you be brave enough to push that to the shared repo with your colleagues?
What do you feel about that commit history?
I don’t know about you, but I know that I feel a bit ashamed.

This is the time to call wipsquash.

wipsquash allows you to combine your wip commits with the nearest meaningful one, creating clean, atomic commits without disrupting your workflow.

Calling git wipsquash merges the three commits together and keeps them at one, properly named Add authentication middleware.

When git wipsquash is called without any parameter, it will squash up to the first past commit with a meaningful name (defined here a not containing wip or WIP or any variation of that).
When called with a commit hash, the script will run will squash up to that commit, creating as many commits as you have non-wip commits.
For instance, with that commit history

0. ...
1. Add authentication middleware
2. Wip
3. wip
4. Add error message when people are not logged in
5. wIp

git wipsquash 1

will produce

0. ...
1. Add authentication middleware
2. Add error message when people are not logged in

with the right commits squashed in the right place.

A little aside on keeping a history of all the small changes

Sure, we are losing all the small steps that we took during the implementation.
Some teams like to keep them, others don’t.
My opinion about that changed a few times.
At the moment, it is that I like having the small commits while I’m working on the feature (and this is why I would like to automate the naming and not just go with wip), but I’m not sure of the value of keeping them after that.

Getting `wipsquash`

Copy the script below into a file named git-wipsquash in a directory in your PATH, and give it execution right with chmod +x git-wipsquash.

As you can see, the scripts actually runs a git rebase -i’ and makes a lot of fixup` commits.

#!/bin/sh

# git-wipsquash: Rewrite the commit history on the current branch so that
# any commit containing "wip" in its message is squashed into the nearest
# preceding commit that does NOT contain "wip".

START_COMMIT="$1"

# If no start commit is given, find the most recent commit that doesn’t contain "wip" in the message.
if [ -z "$START_COMMIT" ]; then
  # Walk from HEAD backwards, find the first (i.e., newest) commit without "wip".
  for C in $(git rev-list HEAD); do
    SUBJECT=$(git log -1 --pretty=%s "$C")
    if ! echo "$SUBJECT" | grep -iq "wip"; then
      START_COMMIT="$C^"
      break
    fi
  done

  if [ -z "$START_COMMIT" ]; then
    echo "No commit found without 'wip' in its message. Aborting."
    exit 1
  fi
fi

echo $START_COMMIT

# 1) Make sure START_COMMIT is an ancestor of HEAD.
if ! git merge-base --is-ancestor "$START_COMMIT" HEAD; then
  echo "ERROR: $START_COMMIT is not an ancestor of HEAD. Aborting."
  exit 1
fi

# 2) Collect commits (oldest first) strictly after START_COMMIT up to HEAD.
COMMITS=$(git rev-list --reverse "${START_COMMIT}..HEAD")


# 2) Build the interactive rebase instruction list
INSTRUCTIONS=""
for C in $COMMITS; do
  SUBJECT=$(git log -1 --pretty=%s "$C")
  if echo "$SUBJECT" | grep -iq "wip"; then
    # squash into the previous picked commit using fixup to avoid prompting a message
    INSTRUCTIONS="$INSTRUCTIONS
f $C"
  else
    # pick normal commits
    INSTRUCTIONS="$INSTRUCTIONS
p $C"
  fi
done


# 3) Write the instructions to a temp file
TMPFILE=$(mktemp)
# The leading newline in $INSTRUCTIONS can confuse 'git rebase -i', so trim it:
echo "$INSTRUCTIONS" | sed '/^[[:space:]]*$/d' > "$TMPFILE"


# 4) Run interactive rebase:
# Rebase the listed commits on top of START_COMMIT
# so everything after START_COMMIT is subject to rewriting.
GIT_SEQUENCE_EDITOR="cat $TMPFILE >" git rebase -i "$START_COMMIT"


# 5) Clean up
rm -f "$TMPFILE"

You’re ready to go.

Also, you might find it interesting to create an alias for quickly creating git wip commits.
Add the following alias to your .gitconfig:

[alias]
cwip = "! git add . && git commit -m ‘wip’ --no-verify”

You can now type git cwip to add all your changes and create a wip commit.
Because this adds all your changes you want to run it often, to know what you are commit without the need to review it.

Conclusion

git wipsquash seems to solve my issue with super small commits and keep a global shared commit history clean. It’s a simple yet powerful tool to automate tedious commit cleanup, that would otherwise take too long.

How test frameworks work : Cleaning after a failure, the birth of TearDown and SetUp

2025-03-31T00:00:00Z

This is part 3 of the creation of our mental model of the internal of test frameworks.
So far, we’ve seen that test frameworks can be seen as loops over functions, and that the mechanism to know that something bad occurred is to throw an exception. To prevent the exception from stopping the loop, it contains a try/catch block. This is the basics.
In the second part, we’ve looked at how they manage to allow us testing that an exception is thrown by the system under test. This is not as easy as it seems.

In this (long awaited) third part, we will see why the tear down method is far more important than the setup method - even if we actually rarely use it.

Let’s refresh our memory and see what our framework currently looks like:

tests = [
	test1() {},
	test2() {},
	test3() {},
	test4() {
		expectedException = MySuperException

		throws MySuperException
	
	}
];

failedTestExceptions = []

// Test loop

foreach(tests as test) {
	expectedException = null
	try {
		test()
	} catch(exception) {
		if(exception != expectedException) {
			failedTestExceptions[] = exception
		}
		continue;
	}
	// Did we get our expected exception?
	if(expectedException) {
		failedTestExceptions[] = NeverReceivedExpectedException(expectedException)
	}
}

// Error display
if(count(failedTestExceptions) === 0) {
	displayGreenBar()
}
else {
	foreach(failedTestExceptions as exception) {
		displayErrorFor(exception)
	}
}

Why do we need a TearDown method?

Let’s see two examples to understand why a TearDown method is needed.

Example 1: The unclosed connection

Imagine that we have a test that requires a connection to a database.
See how we thought about everything: because we don’t want to keep connections open, we close it at the end of the test.


testWithConnection() {
	connection = openConnection()
	// Do stuff
	assertSomething()
	closeConnection()
}

Now, what happens when the test fails? This can be either because the assertion reports an error or because the code is just plain failing and throws an exception.
Because of the exception, the test doesn’t get to reach the end of the function, and the connection is never closed. If you have multiple tests failing in that way and you can exhaust your database connection pool. Leading to even more tests failing. That’s bad for test independence. The failure of a test can cause the failure of multiple others.

Apart from database connections, another example is opened files. While the issue might not lead to a failure of multiple tests, it can lead to your test suite running slower.

Example 2: The interdependence via a shared resource

In this example, let’s talk about 2 tests.

The first test adds a burrito to the collection, does something with it, and, because it’s a nice citizen, cleans the mess at the end and removes the added burrito from the collection. Here, imagine that the collection is something we share across multiple tests, like a database.

test1() {
	addBurritoToCollection('Swiss Burrito')
	// Do something with the burrito and make an assertion
	removeBurritoFromCollection('Swiss Burrito')
}

The second test also deals with burritos.
Maybe it’s a test about counting the number of burritos.

test2() {
	addBurritoToCollection('French Burrito')
	addBurritoToCollection('Mexican Burrito')
	assertEqual(2, countOfBurritos())
	removeBurritoFromCollection('French Burrito')
	removeBurritoFromCollection('Mexican Burrito')
}

Both tests are cleaning after themselves.

Now, what if test1 starts failing for some reason? The Swiss Burrito stays in the collection because the test never reaches the last line where it was supposed to remove it.
Test2 starts, adds two more burritos, makes its assertion, and fails. Now the collection contains 3 burritos. Swiss, French, and Mexican.

We clearly see how the failure of a test impacts another one. Our tests aren’t completely independent. This might not seem like a big problem, but now imagine that you have more than 2 tests, and that you don’t run them that often.^[1] You are in trouble, you need to find the cause of the issue, you look in the code, possibly starting from a test that would probably continue to be ok if the other test wasn’t failing. You look around, everything seems ok. You might not even touched at that part of the code. WTF?

In both examples we’ve seen that even with good intentions and cleaning in the test function, if a test fails we can run into some problems.

Introducing the tearDown method

The TearDown mechanism is here to answer that problem. The tearDown provides a way to act after the test, even in case of failure.

To do so, we need to register what we want to do after the test.

In our simplified test framework, let’s add a variable that can be overridden by every test. This is where that simplified test framework starts to diverge a lot from the ones you probably are used to. In your usual test framework, tearDown (and setUp) are shared by a group of tests. To add that to the model would require to start creating a sort of collection of tests, and this is not that interesting for the mental model that I want to share.

// Test loop

foreach(tests as test) {
	
+   tearDown = null
	expectedException = null
	try {
		test()
	} catch(exception) {
		if(exception != expectedException) {
			failedTestExceptions[] = exception
		}
		continue;
	}
+   finally {
+       if(tearDown) {
+		    tearDown()
+	    }
+}
    	
    // Did we get our expected exception?
    if(expectedException) {
        failedTestExceptions[] = NeverReceivedExpectedException(expectedException)
    }
}

And we can rewrite our two examples to use the tearDown. For the connection:

testWithConnection() {
	tearDown = () => {
		closeConnection()
	}
	
	connection = openConnection()
	// Do stuff
	assertSomething()
}

And for the burrito collection:

test1() {
	tearDown = () => {
		removeBurritoFromCollection('Swiss Burrito')
	}
	addBurritoToCollection('Swiss Burrito')
	// Do something with the burrito and make an assertion
}
test2() {
	tearDown = () => {
		removeBurritoFromCollection('French Burrito')
		removeBurritoFromCollection('Mexican Burrito')
	}
	
	addBurritoToCollection('French Burrito')
	addBurritoToCollection('Mexican Burrito')
	assertEqual(2, countOfBurritos())
}

Now, in case of failure, the connection is properly closed, and the burritos added to the collection are removed. Thank you, tearDown. Our tests are way more independent than before.

My theory about tearDown and setup methods

This is why the tearDown method is actually more important than the setUp one. The tearDown is needed, while the setUp is just something convenient to factorize some common test setup in one place and avoid repetition. Which is nice too but not as crucial.

My theory is that the tearDown was discovered as something we needed for test independence, to make sure we properly clean after each test and that adding setUp was just a question of symmetry. After all, if I have a tearDown shared by all tests, why not have something that helps me setUp all my tests? Also, if we clean the connection after the test, wouldn’t it make sense to be able to create it before the test^[2]

Since I started that series of articles long ago I started sharing carousels on LinkedIn about tests, and shared one about what this series talks about. Someone commented that Kent Beck demonstrates how to create JUnit in TDD in his book TDD By Example. This is a fun example because it shows how to create a test framework in TDD when you don’t have a test framework yet, and to create something using itself. Anyway, I totally forgot that Kent did that in the book and grabbed my copy to see what he was saying. With the steps he takes, the setUp method is introduced before the tearDown one. This contradicts my theory but this might only be because Kent already knew which form the test framework will have in the end. And to add some credit to my theory, the chapter is called "Cleaning up after" and the introduction is

Sometimes tests need to allocate external resources in setUp(). If we want the tests to remains independent, then a test that allocates external resources needs to release them before it is done, perhaps in a tearDown() method.

I find the formulation strange, because I don’t think that tests need to allocate external resources in setUp(), they could really do it inside the test, but for sure, they do need to release external resources no matter what.

So, here it is, end of part 3. We’ve seen how and why the tearDown method came to be and how important it is. Always remember to clean after your test if you use something that can mess with other tests or that can cause issues in the system.

Hey, let’s have a chat!.

Taking smaller steps would reduce the difficulty of dealing with that problem.
You see that the tests are failing, you revert to the previous good state, and you start again. ↩︎
My examples are probably itching some of you.
How can the closeConnection function have access to the connection variable that seems to be declared in the test? You are right, that code probably doesn’t work, but is good enough for building a mental model.
Sorry, not sorry :) ↩︎

Quickly open Google Calendar Tab in Google Chrome from Alfred

2025-07-04T00:00:00Z

I have pinned tabs for Calendar and Gmail, and I wanted to quickly jump to them from anywhere using Alfred.

Another option could have been to use Alfred’s Web Search feature to open a new tab. That would be a simple version of what I wanted, but I was curious to see if it was possible to go to an existing tab.

Additionally, Alfred offers another option with an Automation Task accessible in the Workflow menu (Automation > Automation Task > Extra Automation Task > Web Browser> Chromium > Switch to Chromium Browser Tab Matching URL). I discovered that solution after implementing mine, but I decided to keep what I had because the Automation Task doesn’t properly handle having multiple Chrome windows opened, and I usually have several of them.

In the end, my solution is an Alfred Workflow I made that runs an AppleScript^[1] when a keyword is entered in the Alfred command palette.

The Keyword trigger configuration is simple: Take the keyword "calendar" with no argument. then make it look nice by adding a title, a subtext and Google Calendar icon.

Add an AppleScript action with the following script:

on alfred_script(q)
 tell application "Google Chrome"
	activate
	
	-- Loop through all windows
	repeat with windowIndex from 1 to count of windows
		set theWindow to window windowIndex
		
		-- Loop through all tabs in the current window
		repeat with tabIndex from 1 to count of tabs of theWindow
			set currentTab to tab tabIndex of theWindow
			set tabURL to URL of currentTab
			
			-- Check if the tab contains Google Calendar
			if tabURL contains "calendar.google.com" then
				-- Make this window active and switch to the calendar tab
				set index of theWindow to 1
				set active tab index of theWindow to tabIndex
				return -- Exit the script once found
			end if
		end repeat
	end repeat
  end tell
end alfred_script

For an extra boost of productivity^[2], make it so the calendar opens on the weekly view for today. For this, you can modify the script to ask Chrome to change the tab URL:

  set active tab index of theWindow to tabIndex
+ set URL of currentTab to "https://calendar.google.com/calendar/u/0/r/week"
  return -- Exit the script once found

I also created a similar script for Gmail by using another Keyword trigger and using Gmail's url.

Now I can access my calendar and emails from anywhere!

Hey, let’s have a chat!.

Actually, Claude made the AppleScript. I still don’t understand how this language works, and didn’t take the time to get it. ↩︎
Lol ↩︎

Testing Rule of Thumb: if it looks too much like the real implementation, doubt it

2025-07-16T00:00:00Z

This can mean a variety of things.

If you have logic in tests or are doing some calculations be suspicious, especially if this is for the expected value of the assertion. Prefer literal values or variables without any manipulation there.

Another good example is complex mock setup. If you are writing a complex matcher that is a mirror of what you need to write in the implementation you are not testing anything, you are painting by number, but you are the one making the template. Don’t be surprised if the implementation doesn’t do what you expect it to. The only solution here is to introduce some tests that really exercise the thing you mock, following the Don’t mock what you don’t own advice. The classical example of this is people mocking ORMs and writing their queries in the tests and in the implementation. The double the work but still no guarantee that your DB will actually do what you want it to do.

One final example is validating an external message using its internal representation. Here think about APIs schema. You shouldn’t use your API response DTO to construct the expected API response. The test is supposed to validate a contract. If that contract moves when you are modifying the code it isn’t of any help. Imagine that you rename a field in your DTO using an automated refactoring tool. Shouldn’t your test fail and tells you that you are breaking the contract established with your API clients? If you use the same DTO the test will continue passing as both the implementation and the test will be modified. If you don’t preserve the behavior, this is not refactoring. Write your tests in a way that they tell you if are changing behavior: do not build your expectation the same way you are building the actual value.

As you see, if your test looks too much like the implementation, there is possibly something shady.

Test Migration: From Mocks to Real Database with Test Switcher

2025-08-25T00:00:00Z

Gradual Database Migration Strategy: From Mocks to Real Database

Last week, I needed to migrate a test suite from Prisma mocks to a real database to improve test reliability. In short, using mocks of your ORM is one of the worst things you can do to your test suite. They give you false confidence and make your tests hard to read and maintain.
So we wanted the mocks out, and I was tasked by my team to do that work.
However, I wanted to avoid the chaos of having all tests fail simultaneously when switching implementations. Instead, I opted for a gradual, test-by-test migration approach.

The Strategy

Here’s the step-by-step approach I used:

1. Create the Interface Layer

Fortunately, we had a repository class encapsulating the calls to the ORM. My first move was to add an interface on top of the existing repository to abstract the implementation details.

2. Build the Test Switcher

I created a new implementation called TestSwitcherXXXRepo that acts as a proxy between the two instances of the original repository.
It takes two parameters: one for the repository instance using the mocked ORM and one instance using the real database.

The proxy contains an instance property that defaults to the mocked implementation. By default, we will keep using the mocked version, which is the one in use for the tests in the current state.

The proxy also offers a switchToDatabase() method to toggle to the real database implementation.

Here is an example of a test switcher:

/*This class is not meant to stay. This is something temporary to allow for switching the implementation test by test */
export class TestSwitcherPonyCareRepository implements PonyCareRepository {
  private instance: PonyCareRepository

  constructor(
    private mockPrismaPonyCareRepository: PrismaPonyCareRepository,
    private realPrismaPonyCareRepository: PrismaPonyCareRepository,
  ) {
    this.instance = mockPrismaPonyCareRepository
  }

  switchToRealDatabase() {
    this.instance = this.realPrismaPonyCareRepository
  }

  async fetchCareSchedules(
    stableId: StableId,
    ponyId: PonyId,
  ): Promise<PonyCareSchedule[]> {
    return this.instance.fetchCareSchedules(stableId, ponyId)
  }

  async saveCareUpdates(
    careScheduleToSave: PonyCareScheduleToSave,
  ): Promise<void> {
    return this.instance.saveCareUpdates(careScheduleToSave)
  }
}

Yes, I’m lucky and working on a pony club management application.
(Unfortunately, that’s not true, and the original repository was about a less glamorous topic.)

Creating a proxy is an easy task using the refactoring tools that good IDEs have to offer.

3. Set Up the Test Infrastructure

Now it’s time to start modifying the tests.
The first thing to do is instantiate the TestSwitcherXXXRepo with both the mock and real database repositories.

The next thing is to replace the usage of the original repository with the switcher implementation.

This setup took only a few minutes. The trickiest part was remembering how to select specific interface implementations in NestJS.

4. Migrate Tests Incrementally

We now have the infrastructure in place to switch each test one by one.

For each test, start by adding testSwitcher.switchToRealDatabase() as the first line. This also serves as a clear marker of migrated tests.

Then, set up the necessary database fixtures and update the assertions to work with real data instead of mock expectations.

This step was the most time-consuming. I had to reverse-engineer each test to understand what data it needed to demonstrate the expected behavior.

This becomes significantly more challenging with complex database queries—if you have intricate queries, consider testing them at the repository level instead of mixing them with testing business behavior as well.

5. Clean Up

Once all tests are migrated, you can inject the real database repository directly, remove all switchToDatabase() calls, and delete the TestSwitcherXXXRepo implementation.

Benefits

This approach has several benefits:

We can fix tests one at a time, with only a single test failing at any given moment
We are able to push changes after each test migration, keeping CI green and enabling continuous merging
We maintained a working test suite throughout the migration process
We easily tracked migration progress through the switchToDatabase() markers

The gradual migration strategy proved to be efficient for our migration. At no point were we lost with a lot of tests failing, fearing that we would face a massive merge conflict while other team members could be working on that same test file. As often happens, the key to migrating code is to create the tools to make it a less stressful experience.

Solving Obsidian + Readwise Merge Conflicts with a Custom Git Driver

2026-01-12T00:00:00Z

If you're using Obsidian with a Git-based backup system and the Readwise plugin, you've probably encountered this frustrating scenario: you sync your vault on two different machines, both have new
highlights from Readwise, and suddenly you're staring at merge conflicts in a JSON file you never intended to edit manually.

I got tired of resolving these conflicts by hand. So I asked Claude Code to help me build a solution.

The Problem

My Obsidian vault is synced across multiple devices using Git. The Readwise plugin stores its state in .obsidian/plugins/readwise-official/data.json, which contains:

A lastSavedStatusID field tracking the sync progress
A booksIDsMap object mapping file paths to Readwise book IDs

When I read articles on my phone and sync Readwise on my laptop, then do the same on my desktop, Git sees two divergent versions of this file. The result? Merge conflicts with those dreaded <<<<<<<
markers in a JSON file.

<<<<<<< HEAD
  "lastSavedStatusID": 23934436,
=======
  "lastSavedStatusID": 23924421,
>>>>>>> other-branch

The fix is obvious to a human: keep the highest ID (it represents the most recent sync) and merge the book mappings. But doing this manually every time is just plain annoying.

The Solution: Custom Git Merge Drivers

Git has a lesser-known feature called custom merge drivers. Instead of showing conflict markers, Git can call a custom script to resolve conflicts automatically.

I described my problem to Claude Code: "I want a merge rule for readwise-official/data.json. In case of conflict, always keep the highest ID."

Claude Code immediately understood the requirement and created a solution using jq for proper JSON manipulation. When I later realized the booksIDsMap also needed merging, Claude updated the script
accordingly.

Testing First with Bats

I didn't want to blindly trust a script that modifies my data during merges. I asked Claude Code to write tests before we finalized the implementation.

Claude chose Bats (Bash Automated Testing System), a testing framework for Bash scripts.
This was a good use case to finally get started with bats.

Here's what the test structure looks like:

setup() {
    TEST_DIR="$(mktemp -d)"
    SCRIPT_DIR="$BATS_TEST_DIRNAME/.."

    BASE_JSON='{
  "token": "test-token",
  "lastSavedStatusID": %s,
  "booksIDsMap": {%s}
}'
}

teardown() {
    rm -rf "$TEST_DIR"
}

create_json() {
    local status_id="$1"
    local books_map="$2"
    printf "$BASE_JSON" "$status_id" "$books_map"
}

I was surprised, in a good way, to see that Claude thought about using fixture builder like the create_json function.

It helped creating some nice tests. Here are a few examples:

@test "lastSavedStatusID: takes other when other > current" {
    create_json "100" "" > "$TEST_DIR/ancestor.json"
    create_json "150" "" > "$TEST_DIR/current.json"
    create_json "200" "" > "$TEST_DIR/other.json"

    run "$SCRIPT_DIR/readwise-merge.sh" \
        "$TEST_DIR/ancestor.json" \
        "$TEST_DIR/current.json" \
        "$TEST_DIR/other.json"

    [ "$status" -eq 0 ]

    result=$(grep -o '"lastSavedStatusID": [0-9]*' "$TEST_DIR/current.json" | grep -o '[0-9]*')
    [ "$result" -eq 200 ]
}

@test "booksIDsMap: no data loss - all unique entries from both sides are preserved" {
    create_json "100" '' > "$TEST_DIR/ancestor.json"
    create_json "100" '"current1.md": "1", "current2.md": "2", "shared.md": "100"' > "$TEST_DIR/current.json"
    create_json "100" '"other1.md": "3", "other2.md": "4", "shared.md": "999"' > "$TEST_DIR/other.json"

    run "$SCRIPT_DIR/readwise-merge.sh" \
        "$TEST_DIR/ancestor.json" \
        "$TEST_DIR/current.json" \
        "$TEST_DIR/other.json"

    [ "$status" -eq 0 ]

    # Should have exactly 5 unique entries
    count=$(jq '.booksIDsMap | length' "$TEST_DIR/current.json")
    [ "$count" -eq 5 ]

    # Shared key should have current's value, not other's
    shared_value=$(jq -r '.booksIDsMap["shared.md"]' "$TEST_DIR/current.json")
    [ "$shared_value" = "100" ]
}

Running bats tests/readwise-merge.bats gives us confidence that the driver behaves correctly:

 ok 1 lastSavedStatusID: keeps current when current > other
 ok 2 lastSavedStatusID: takes other when other > current
 ok 3 lastSavedStatusID: keeps current when equal
 ok 4 booksIDsMap: keeps current entries when other has none
 ok 5 booksIDsMap: adds entries from other that current doesn't have
 ok 6 booksIDsMap: merges entries from both current and other
 ok 7 booksIDsMap: no duplicates when same entry in both
 ok 8 booksIDsMap: current wins when same key has different value
 ok 9 booksIDsMap: no data loss - all unique entries from both sides are preserved
 ok 10 combined: handles both lastSavedStatusID and booksIDsMap
 ok 11 output is valid JSON

The Merge Driver

Here's the final script:

#!/bin/bash
# Custom merge driver for readwise-official/data.json
# - Keeps the highest lastSavedStatusID
# - Merges booksIDsMap (current wins on conflicts)

# Arguments: %O %A %B (ancestor, current, other)
ANCESTOR="$1"
CURRENT="$2"
OTHER="$3"

# Extract values using jq
CURRENT_ID=$(jq -r '.lastSavedStatusID // 0' "$CURRENT")
OTHER_ID=$(jq -r '.lastSavedStatusID // 0' "$OTHER")

# Determine highest ID
if [ "$OTHER_ID" -gt "$CURRENT_ID" ]; then
    HIGHEST_ID="$OTHER_ID"
else
    HIGHEST_ID="$CURRENT_ID"
fi

# Merge: start with other's booksIDsMap, overlay current's (current wins on conflicts)
# Then update lastSavedStatusID with highest value
jq -s '
   .[0] as $current |
   .[1] as $other |
   $current |
   .lastSavedStatusID = '"$HIGHEST_ID"' |
   .booksIDsMap = ($other.booksIDsMap // {}) + ($current.booksIDsMap // {})
' "$CURRENT" "$OTHER" > "$CURRENT.tmp" && mv "$CURRENT.tmp" "$CURRENT"
exit 0

Installation

1. Create the driver script

mkdir -p .git-merge-drivers
# Save the script above as .git-merge-drivers/readwise-merge.sh
chmod +x .git-merge-drivers/readwise-merge.sh

2. Create `.gitattributes`

.obsidian/plugins/readwise-official/data.json merge=readwise-highest-id

3. Configure Git

git config merge.readwise-highest-id.name "Readwise merge - keep highest ID and merge books"
git config merge.readwise-highest-id.driver "$(pwd)/.git-merge-drivers/readwise-merge.sh %O %A %B"

4. If you already have a conflict

git checkout -m .obsidian/plugins/readwise-official/data.json
git add .obsidian/plugins/readwise-official/data.json
git rebase --continue  # or git merge --continue

Conclusion: Beyond Readwise

This experience got me thinking about other merge pain points.

Lock files. Anyone who has resolved conflicts in package-lock.json, yarn.lock, or composer.lock knows the frustration. These files are generated, not authored. The correct resolution is almost
always "regenerate the file"—run npm install, yarn install, or composer install after merging package.json. A custom merge driver could automate this.

LLM-backed merging. For simple, rule-based conflicts, we could describe the merge logic in a prompt and let an LLM resolve them. "Always keep the higher version number." "Merge arrays by taking the
union." "For this config file, prefer values from the current branch." This could work for cases where the rules are clear but the file format is complex.

Deterministic scripts over manual resolution. The real win here is moving from ad-hoc manual fixes to deterministic solutions. Given the same inputs, the merge driver always produces the same
output. No human judgment required, no mistakes from rushing through conflicts at 6 PM on a Friday.

The next time you find yourself resolving the same type of conflict repeatedly, consider: can you describe the resolution rules? If yes, you can probably automate it.

Easily use multiple git identities on one laptop

2026-02-01T00:00:00Z

When you're a consultant with multiple clients, or fetching personal Git repositories on your work laptop, you might need to switch between multiple identities. You might use your personal handle and email address to commit on your personal projects, and use a work-related email and full name for work repositories.

And if you're in that situation, chances are high that you've used the wrong identity in a repo at least once.

This happened to me again the other day. I have some config files that reuse one multiple laptops, my work one included, and that config comes with a global git config that is set for my personal projects. Long story short, I committed with my personal info on a work repo. That's not a huge issue, but I like to keep these two things separate when possible. So, I wonder, how could I avoid that? And as often lately, I turned to Claude to find a solution.

I ended up creating a zsh hook that warns when entering a git repository without local identity configuration, plus a helper function to quickly apply the right identity.

The hook

The following code hooks on chpwd which allows to detect a change of directory, either with cd or jumper utilities like z/zoxide.

When entering a directory, it first looks to see if that is a git repository, if not, it exits.
Otherwise it looks if an identity is configured in the local git config. If that is the first time we enter the repository for that session it will display a message:

[git-identity] No local identity configured
Run: git-identity-apply <module>

We will talk about git-identity-apply shortly after.
It keeps track of which repositories were seen during a session to avoid showing the message each time we cd inside a repo.

Here is the hook script:

# Git identity check on directory change
# Warns when entering a git repo without local identity configuration

# Track warned repos to avoid repeated warnings per session
typeset -gA _git_identity_warned_repos

# Find the dotfiles directory
: ${DOTFILES:="$HOME/.dotfiles"}

# Check git identity and warn if not configured
_git_identity_check() {
  # Only proceed if we're inside a git worktree
  [[ "$(git rev-parse --is-inside-work-tree 2>/dev/null)" == "true" ]] || return

  local git_root
  git_root=$(git rev-parse --show-toplevel 2>/dev/null) || return

  # Skip if we've already warned about this repo this session
  [[ -n "${_git_identity_warned_repos[$git_root]}" ]] && return

  # Check if local identity is configured
  local local_email local_name
  local_email=$(git config --local user.email 2>/dev/null)
  local_name=$(git config --local user.name 2>/dev/null)

  # Warn if either is missing
  if [[ -z "$local_email" || -z "$local_name" ]]; then
    _git_identity_warned_repos[$git_root]=1
    echo "[git-identity] No local identity configured"
    echo "  Run: git-identity-apply <module>"
  fi
}



# Register the chpwd hook
autoload -Uz add-zsh-hook
add-zsh-hook chpwd _git_identity_check

# Also run on shell startup in case we start in a git repo
_git_identity_check

Now we have an easy way to know that we didn't set a local identity, but we still need to add one. This means two git config commands to run, but I never remember what they are and I mostly want to use the same identities—either my personal one or a client-related one.

My configuration system comes with modules. Module can be anything. I have a module for PHP that install things PHP related, one named "Perso" that installs things that I want on every laptop. I also have one, you guessed it, by company I work for. Using symlinks I can decide on each laptop which module I want to install, run a script, and I keep everything up to date using brew.

So, let's go back to our git identity thing.

In each module where I want a separate identity, I add a file called git-identity that looks like:

git config --local user.email "jane.doe@acme-corp.com"
git config --local user.name "Jane Doe"

I made a script, the git-identity-apply you've been waiting to meet, that takes a module name as an argument and configures the repo with the matching identity. To make things even more convenient, you can run the command without parameter, the command lists all modules with an associated identity and lets you select one using fzf. Select one, press enter, and boom, you're ready to go.

# Apply git identity from a module configuration
git-identity-apply() {
  local module="$1"
  local modules_dir="$DOTFILES/Modules"

  # Interactive selection with fzf if no module specified
  if [[ -z "$module" ]]; then
    local modules=()
    for identity_file in "$modules_dir"/*/git-identity(N); do
      modules+=("${identity_file:h:t}")
    done

    if [[ ${#modules[@]} -eq 0 ]]; then
      echo "[git-identity] No modules with git-identity found"
      return 1
    fi

    module=$(printf '%s\n' "${modules[@]}" | fzf \
      --header="Select git identity" \
      --preview="cat $modules_dir/{}/git-identity" \
      --preview-window=up:3)

    [[ -z "$module" ]] && return 0
  fi

  # Find the module's git-identity file (case-insensitive match)
  local identity_file
  for dir in "$modules_dir"/*(N); do
    local dir_name="${dir:t}"
    if [[ "${(L)dir_name}" == "${(L)module}" ]]; then
      identity_file="$dir/git-identity"
      break
    fi
  done

  if [[ ! -f "$identity_file" ]]; then
    echo "[git-identity] Module '$module' not found or has no git-identity file"
    return 1
  fi

  # Source the identity file to apply the config
  source "$identity_file"

  # Display the applied identity
  local applied_email applied_name
  applied_email=$(git config --local user.email 2>/dev/null)
  applied_name=$(git config --local user.name 2>/dev/null)

  echo "[git-identity] Applied identity from $module"
  echo "  user.email = $applied_email"
  echo "  user.name  = $applied_name"

  # Clear the warned flag for this repo since it's now configured
  local git_root
  git_root=$(git rev-parse --show-toplevel 2>/dev/null)
  if [[ -n "$git_root" ]]; then
    unset "_git_identity_warned_repos[$git_root]"
  fi
}

Using agents to create small tools to make my life easier as a developer is something that I enjoy. I like that I can explain my problem, brainstorm a solution with an agent and have a tool that I wouldn't have created otherwise.