With Test Data Randomization Comes Great Responsibility

TL;DR

In this week’s testing from trenches, I present my bug in a unit test. The bug was caused by the random unit test data, and the check was not robust enough.

In the above image, you can see the Elixir Espec testing framework in action. Framework (in test setup code) creates testing HTTP connection (this is conn structure). Check is expect function, where we check, does conn resp_body (HTML page) contains project structure name. The project is created using insert_project() the helper, where project.name is set using the Elixir Faker library. We use Faker’s name function that returns a random name.

That test randomly failed, and I know that problem was with the project name attribute.

Root Cause

Faker name function could return name with special characters, for example, O'Brian. Due to security reasons, `conn resp_body is HTML escaped, which replaces ' with ' and check failed. It did not fail when Faker.name did not contain special characters.

Conclusion

Be careful what you check!

Originally published at https://blog.tentamen.eu on June 22, 2020.