With Test Data Randomization Comes Great Responsibility


In this week’s testing from trenches, I present my bug in a unit test. The bug was caused by the random unit test data, and the check was not robust enough.

In the above image, you can see the Elixir Espec testing framework in action. Framework (in test setup code) creates testing HTTP connection (this is conn structure). Check is expect function, where we check, does conn resp_body (HTML page) contains project structure name. The project is created using insert_project() the helper, where project.name is set using the Elixir Faker library. We use Faker’s name function that returns a random name.

That test randomly failed, and I know that problem was with the project name attribute.

Root Cause

Faker name function could return name with special characters, for example, O'Brian. Due to security reasons, `conn resp_body is HTML escaped, which replaces ' with ' and check failed. It did not fail when Faker.name did not contain special characters.


Be careful what you check!

Originally published at https://blog.tentamen.eu on June 22, 2020.

Founder of Tentamen, software testing agency.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store