Testing Man-Hours

Defining the task

Normally they consider testing to make up 40% of the entire time spent on the project, with no proper definition for what testing actually means there. Does it mean the quantity of man-hours tester and coders spend at their workplace? In this case 40% is too much. Does it mean various types of activities engineers are involved in? And does testing involve debugging? And what about test scripts?

Let’s consider only those software development groups which imply clear role differentiation:

• Analysts
• Program architects
• Technical writers
• GUI designers
• Testers
• Other

Also, let’s associate testing costs with all the work types performed by a testing department and programming costs with those performed by a programming department.

General theory

Let’s consider programmers-testers ratio first. There are two basic parameters to determine this ratio:

1. system criticality (the severity of loss in the event of a bug)
2. system complexity

System criticality can be calculated according to Cockburn:

1. loss of comfort
2. loss of discretionary money
3. loss of irreplaceable money
4. loss of life

Loss of money should include distribution costs and reputation damages.

* Thus bugs in box products damage worse than bugs in the software distributed within its network. That’s why with its product being comparable to Nival’s product in terms of scale and complexity, Yandex can do with a smaller testing department.

* Reputation damage depends on market development level. At the dawn of the search engines development [the company we-all-know] could indulge itself in producing some openly slapdash workmanship (see also Ashmanov’s “Living in a bubble”). But the present day market doesn’t forgive such doubtful tricks.

Let’s try to scale. For non-critical (level of criticality 4), simple, and smallest systems (below 1 man-month) the normal ratio should be of 100:0. This is the reason why small business card websites (normally) do not require any testing.

Placed at the other end of the scale are NASA products (level of criticality 1). Some rumors say they have 5 to 10 times more testers than coders. And with its documenting level NASA projects are likely to involve only 5% of coders.
Again, according to some rumors Microsoft’s coders-testers ration is 1:2.

For a middle-sized automation box system (20-50 man-years) the appropriate ratio should be from 4:1 to 3:2 man-months.

Various details

But there are certain small yet crucial details. E.g what king of software we are working on? Or what is the role of a person who creates unit tests for command line compiler? There’s no other testing there, but how should we call him? If a tester – why does he know C++ still better than the majority of other companies’ leading programmers? And who is to perform the roles of an analyst or a technical writer, and are these roles envisaged? When formally a tester, I created and/or verified coding and UI design standards, ran project documenting (the concept, SRS, the glossary). What kinds of job are these?

Wrong interpretation

Let’s say that testers of the SuperSMS project spent 400 hours working in October 2009, while programmers spent 600 hours. It is said sometimes that in such a case testing man-hours on the project make up 40%. Such a conclusion is profoundly invalid. Let this project last from July till October. Project man-hours distributed every month (July/August/September/October) as follows:

• Management – 200/200/200/200
• Infrastructure development – 100/20/20/10
• Process implementation (QA) – 100/0/0/50
• Analysis – 400/200/100/0
• Design and front end implementation (design department) – 300/100/50/50
• Programming – 600/600/600/600
• Testing – 50/50/400/400
• Creating user documentation – 0/0/200/200

Thus after we’ve taken into account the work performed by all of the departments we can see that the number of testing man-hours has appeared to decrease. And we should also keep in mind that there’s a great variety of work performed before actual project starting, and it makes up a considerable part. But even without taking it into account we have the following:

• Testing: 900/(800+150+150+700+500+2400+900+400) * 100% = 15%
• Programming: 2400/6000 * 100% = 40%

And this result is rational and in line with observations.

What else is worth knowing when planning man-hours

Let’s say the calculations above match the real needs. If we lived in the imaginary world of RUP or PMBOK, it would have been reasonable to plan such a budget. However, “Welcome to the real world”, like Morpheus said. Real world is where some deviations can take place – someone may fall ill or a task solved may turn out less complicated than planned.

In real world they suggest limiting the most costly area with providing other areas with extra resources when needed (buffers can be also built into a plan, but in such a case or project may last 2 or 3 years instead of 4 months). In our case the one to be limited is the programming department, with other departments getting a bigger budget.

Precise figures depend on the deviations level, cost losses in the event of delay and the relative value of each department. The cheapest areas require the biggest reserve. Let’s say that normal reserve is 30% for all the departments but programming. Thus we get the following:

Testing: 1170/(1040+ 195+195+910+650+2400+1170+520) ? 100% = 16,5%
Programming: 33,9%

P.S.

An interesting fact is that during some business management games people tend to make the cheapest department a bottleneck. Naturally, it results in huge money losses. And I’ve not yet come up with any explanation for this phenomenon.

One thing more. If the SuperSMS project involved only developers (don’t mix them up with programmers) instead of analysts, tech. writers, and programmers, the project would have been completed much faster and the man-hours would have decreased considerably (approximately two or three times). Still it is a small project, which demands cross-functional specialists rather than narrow ones. Unfortunately, developers are not in demand on our market and are rather rare. But still they are.