TestRail review

Short official summary:
• Official website: http://www.gurock.com/testrail/
• Purpose: Maintaining testing documentation
• License: Paid product, from €159
• Status: release 1.0
• Platform: LAMP

For all the projects we process testing documentation is obligatory, from short checklist to branchy test suite with cross-links and hundreds of test cases. The idea to store documentation in local files was rejected right away, for it's inefficient and simply unprofessional. We tried a “universal” solution, wiki (DokuWiki), but it wouldn't take on. It's hard to tell why, perhaps with the lapse of time we could have got used to it, but the TestRail appeared, so wiki-sort stuff was forgotten immediately.

Short terminology disclaimer:
• Test case — a test-plans’ atomic element, consists of following component.
• Case run environment description (it can be moved to a higher level – to the test-plan)
• Steps to reproduce
• Expected result
• Test suite is a list of cases, combined according to any common factor (whole product, specific feature, etc.)
• Test plan is a list of test suites, chosen for the test
• Test run – is a run of the test plan

Installation and System Requirements

Two variants: typical LAMP or "IIS+SQLServer+FastCGI/PHP". The ionCube and cURL is also needed, but it’s not uncommon. It works rather fast, and after enabling gzip compression using mod_deflate it started to run at railway speed.

Basic features

Multiuser multiproject system for testing documentation maintaining and tests results accounting. The system provides jobsite for everyone: for test managers, who are interested in “when it’s to be done” and “how many bugs are found”; for ordinary testers, who find easy way to report completed cases; for analysts, who're trying to crack the system behind app issues and understand what's causing the troubles.

Workflow

1. Create a project, point out "Defect View Url" and "Defect Add Url". The integration with bug-tracker is simple and yet brilliant: http://your-bug-tracker.com/view/%id%, and TestRail automatically substitutes this bug %id%. It fits the majority of bug-trackers.
2. Create Milestone: actually, it is a version of a product tested and a deadline.
3. Create a Test Suite:a set of test cases, e.g. “UI testing”.
4. Within a Test Suite cases and sections of cases are created: for example, in “UI testing” there are two sections: “Client UI” and “Admin dashboard UI”. These sections support unlimited nesting.
5. Each specific test case consists of Preconditions, Steps and Expected Result. I presume, everything is obvious here. You can attach pictures to them and cross-link them.
6. After describing the testing system with the means of test cases, it is time to start testing. You need to create a Test Run, which includes necessary Test Suite or specific test cases. An executing person (from a list of active system users) is assigned to it, and it's the final point for a test designer.
7. A tester starts executing a test-run as he receives a notification about it. After tests completion each test case receives a status according to test results: Passed, Failed or Retest. Comments and found bugs’ IDs are also posted there.
   After the completion of all tests, the test run is closed, results are sent to the Task Owner, and all data about tests is kept in the system.
8. Optionally, all the results can be exported to different formats, printed or even dumped to xml for some special use. TestRail creates some graphs for quick situation and application quality analysis.

Competitors comparison

I’ve exeperience working with two alternative systems: Rational Quality Manager and TestLink. I beg open-source upholders' pardon, but never I want to deal with the TestLink again. It is free, and that’s a telling argument, but that’s the only merit. Interface is old and inefficient; the system of cases and suites linking is poorly thought-out; the structure of suites… you know, in my (and entire teams') opinion – “absolutely ineligible”.

The entire Rational Suite is great, but it’s too massive for small projects, and the propagation of its own methodology is overdosed. Let alone the price, which is comparable to the price of a batch of magic carpets.

Also I'd like to mention Bugzilla add-on named Testopia, but I haven’t seen it on stream myself and never heard of someone using it in production. If you use Bugzilla in your project, I think you need to pay attention to Testopia, because the high level of integration is promised. Also, there are some similar solutions, inbuilt to integrated development systems (i.e. Mercury) or desktop standalone products (TestLog, QaTraq). They don’t fit us because of lack of normal web-access.

Personal impression

The system proved to be almost perfect for us, both for its’ features and everyday usability. We’re involved in outsourced testing and run lots of small projects, and because of that the vendors' license policy really fits us: the limit is applied to active system users only, so we deactivate client accounts at the end of the project.

Only two shortcomings are to be mentioned: current version of TestRail doesn’t support Unicode, which means, that it’s impossible to maintain documentation in Russian/Ukrainian without any hacks. The respective feature request is already announced. It is not an essential feature for our team, as all projects deserving to be maintained in TestRail are in English. The second problem is the lack of any WYSIWYG-editor in all text edit fields. It makes engineers happy, but regularly makes our customers feel confused.

Summary

If you maintain testing documentation of a big project or get lost in tons of doc- and xls- files with cases, you need to pay attention to TestRail. This product is not something everyone needs, but if you’re professional in testing, remember the name, and maybe TestRail will come in useful to you when processing some further projects.