User Acceptance Testing/UAT Workshop Findings. Is it Proper Testing?

There have been a few occasions now when I have had the pleasure of hosting a round table discussion on User Acceptance Testing (UAT).  I find it interesting because to many in the world of Software Quality and Testing, it is somehow not “proper testing” or not really part of the important task of delivering software changes.   You don’t see pseudo academic papers on it with interesting formulae to calculate metrics such as the mean time between failure or percentage of code coverage. The whole subject is just not meaty enough for the hardened professionals.  And perhaps there is some truth in that.  The last time I was involved in such a discussion one of the standout tips for success was “cookies” or “biscuits” as we say in the UK.  I can’t take any credit for this insight, but I was happy enough to borrow the idea for the most recent event.

The round table last week was held at a user conference for Infor M3, LN and Baan users.  If you think about that situation, most of the “meaty” testing will have already been done further down-stream in development, system testing, integration testing and perhaps even regression testing by Infor, or whichever vendor’s products we are talking about (other ERPs are available…).  They have probably used continuous build, continuous integration and continuous testing as well as other automation to maximize the chance of defect detection.

So, could you argue that any more testing is a waste of time?  Maybe.  But that might be valid if the vendor was going to underwrite or compensate you for any errors that did get through and had a negative business impact.  But they don’t of course and the buck stops with the users of the application, so we understand the desire, the need to make sure it is right.

Then if you consider,let’s say, a significant change such as an upgrade, how many users might be involved across the business in testing that, probably over a number of weeks and cycles?  20? 50?  More?  And that is per company, so if there are 1000 companies using that particular solution does it turn out that most of the testing taking place in the world is UAT?  Is it the biggest form of software testing there is?

In that context, these were some of the points that were raised in the discussion on the subject.

UAT Challenges

• UAT is painful and a lot of work.
• Consistency of approach, coverage, documentation are all common issues.
• Standardisation was desired, but not easy to impose or get buy into to deliver.
• Managing the people and the tasks, especially where tests spread across a number of people or groups was not easy.
• Where there were local differences from one environment to another, it made the task harder.
• Scripts were generally prepared by the users, but they needed guidance and input on what to test and how to document it.
• It was felt the burden of repetition in UAT did not help, asking users to repeat something several times was not popular.

Strategies discussed

• Training users about testing to bring awareness of the importance and the methods.
• Using tools to organise and manage the tasks gives visibility to IT and business sponsors of what needs to be done when, by whom.
• Using test capture tools so that users don’t have to worry about documenting what they have done, as the evidence is gathered automatically.
• Running automated regression testing prior to UAT so the main issues are weeded out before involving users.  Leaving users to focus on the changes, rather than anything and everything.
• Oh yes, and of course “cookies”.  Make sure you have a plentiful supply of cookies at those Conference Room Pilot or workshop sessions and you are more likely to get the users to attend, maybe multiple times.

Conclusions

UAT is a substantial amount of effort and cost, but you still need to do it.  Users don’t like to do it; IT teams can’t do it like the users and don’t want to do it either.  So where is the compromise?

The ideal position is to have an easily created and easily maintainable automated regression test suite that is run by IT on behalf of the users.  It finds all the differences, wherever they are, even where you didn’t know to look.  This tells you what you need to look at, what in some cases the users will need to look at and determine whether there is an issue or not.  It might mean just a revision to a standard operating procedure or training content.  When the users are testing, they are empowered with tools that make this task easier for them.

If you are looking for a simple, easy to implement solution to your UAT challenges you might like to watch this two minute explainer video or request a free trial. Would automated regression testing make a difference in your organisation? Find out more here.