Introduction
User Acceptance Testing (UAT) is inevitably near the end of any software change cycle, and it performs a number of key objectives in the overall software quality process. It serves to ensure that the software is usable, safe, understood, and meets the objectives laid out in the first instance.
User Acceptance Testing (UAT) is inevitably near the end of any software change cycle, and it performs a number of key objectives in the overall software quality process. It serves to ensure that the software is usable, safe, understood, and meets the objectives laid out in the first instance. The word ‘acceptance’ has become less important in most current forms of UAT as it is not typically used to ‘accept’ the software or system against a set of acceptance criteria, more it is an additional set of checks and at the same time provides familiarization and training so that the implementation is smooth and successful.
Organizing, managing, and executing any type of testing or testing project can be a substantial and daunting task, and UAT is no exception. For every project that UAT is implemented in, there are unique challenges and factors to consider. However, the effort is worth it, and in many cases, essential for the success of the project and saves time and money in the long run.
However, these results are entirely dependent on the appropriate implementation of testing and good input from users. Without this, the software’s residual errors will not be discovered negatively impacting the change of systems or cause costly disruption and discontent internally and externally.
Any UAT phase is likely to be the most costly type of testing undertaken due to the number of people, timescales, and repeated cycles. Hence, it is well worth optimizing this activity to get the best outcome and do it in the most efficient and cost-effective way.
This handbook looks at the potential activities that might apply in a UAT project. It provides the knowledge for putting in place frameworks that give a project the best chance of success and maximize positive outcomes from each activity and the overall project.
As with any testing process, not all phases need to be implemented within every project, but each should be considered to achieve the best possible outcome.
Scope
Initially, you need to determine the scope of your testing through three stages.
Determine timescales
Whilst implemented at the end of projects; it is advisable to start early and engage with all teams involved to ensure UAT is on the radar and considered in their planning. Often, departments consider the challenges and defects that will arise but do not realise that UAT identifies defects for fixing and overcoming.
When determining timescales, you also need to consider where resources will need to be implemented and the overall strategy which the UAT is being implemented to achieve. Part of the planning process will involve determining why the project is being carried out and the benefit which the organization is hoping to gain from the change.
Once the benefit is realized, it must be effectively communicated to all UAT participants so they understand what they need to be looking out for whilst testing to achieve success.
Things to consider
What does the whole project look like?
What are the business’s goals?
What previous phases will the project go through?
Who is involved in the whole project?
The methodology being implemented and where UAT will be used?
The strategy and content used in previous test phases and what can be reused?
The resources set aside for fixing issues and bugs
The planned date to go live and is it flexible to allow changes?
The extent of the changes and how will they impact the business?
Are there any known challenges and issues the change is bringing?
Analyse the change
It’s necessary to determine the scope of the project to understand the changes that the business is hoping the software will bring and the resources and time needed to test it sufficiently.
When analyzing the change, if it is a unique software a business is developing, you can communicate expectations from previous teams and phases, including development and quality assurance, to understand the software’s expectations.
If it is a cloud-based software or package being implemented, obtain data from other users via forums and other channels to gauge the scale of the testing and resources needed.
Things to consider
Significance of change
The scale of the change
Systems which will be impacted
Business teams affected and number of people who will be using the software
The existing assets affected
Input from the provider and reviews from clients
Determine impact of changes and risks
Whilst implemented at the end of projects; it is advisable to start early and engage with all teams involved to ensure UAT is on the radar and considered in their planning. Often, departments consider the challenges and defects that will arise but do not realize that UAT identifies defects for fixing and overcoming.
By conversing with those involved with the system, that being the system architects if developing software internally or with the software provider, determine the impact of changes on the business.
It is always important to remember that whilst the scale of a project and impacts can be closely related; it is not for certain that small changes only result in small impacts but rather result in disproportionately large impacts to the business.
Therefore, it is important to assess each change accordingly and determine its impact to efficiently plan User Acceptance Testing
Things to consider
Impact to current systems
Impact on the overall business and how affected systems can impact further impact the business
Impact to IT and other infrastructure
The risk to the business from undetected errors
Risk of timescales not being met
The scope of the testing required
Strategy
Once a plan is put in motion, creating a comprehensive and holistic strategy is important for successful UAT implementation.
Refer to test policies
Different organizations have different cultures and business practices; some may have developed from historic need and others from dynamic approaches and may have pre-existing test policies.
Having these, however, do not mean they will be appropriate for every testing, and reviewing them to determine if they can be implemented needs to be determined early on. However, if you’re going to adapt these policies for the test, this also needs to be outlined early in the process.
If your organization does not have test policies, take the opportunity to outline a policy for future guidance and best practice but be mindful of how testing is dynamic and policies may not fit every scenario.
Things to consider
Does your already existing policy for UAT work for this project?
Does the policy need altering?
Does a policy need to be written?
If you’re not following the policy, outline why not and get secondary feedback.
How is the project risk impacted by policies?
Determine the strategy
Once the plan and initial preparations have been completed, determine a strategy that accounts for everything discovered from the preparations.
Things to consider
Do you need a fully test case driven, documented approach?
How much exploratory testing is appropriate?
What resources will be used and in what way?
Who will write the test cases?
Level of detail needed vs SME knowledge
Are detailed test scripts required?
Who will carry out the tests?
How will they do them?
How many rounds of testing will be required?
Will it all be manual?
Is there an exploratory testing phase?
Can you build in automation as you go?
How will you determine readiness?
What metrics will guide you in this and how will you gather these along the way?
Determine the tools to be used
As with every test, using specific technology can make life considerably easier.There are 4 key areas to consider
4 KEY AREAS TO CONSIDER
1) You may need a platform to manage the projects’ assets combined with communication, resources assignment and scheduling. Fundamentally, it will act as a test management solution to support the users and the project’s needs.
The working and strategy undergoing may have different facets depending on the approach being taken. For example, you may have ‘issues’ that do not fall under ‘defects’ or ‘feedback’ segmented by positive, negative and neutral results. Regardless of your approach, your test management platform will need to be designed to support the style of UAT being planned and the users’ needs.
Technology capturing user actions unobtrusively and accurately with little to no effort, especially if you plan on implementing exploratory testing. Historically, the capturing of results is one of the most frustrating aspects of UAT, with users getting tired of documenting via print screen and pasting to Word when errors occur and manually detailing what happened.
This historic method is unreliable as it depends on users being honest and meticulous in documenting the process. There are inexpensive solutions for capturing the process, which can be easily implemented and take the pressure off users.
UAT needs to be repeated to ensure the results are as accurate as possible; however, after several testing cycles, automation can be implemented to seamlessly run the software, checking for issues that manual testing has overlooked.
The testing process needs a test environment and ideally consistent data to go with it. Test data management tools help handle the data collected, ensuring it is easy to validate when integrated with automation and capturing user actions technology.
Factors to consider for technology
Before implementing technology for the project, there are factors that need to be considered:
Factors to consider
How will you organise test assets?
Requirements
Test cases
Test scripts
Test results
Issue Reports
Defects
Surveys
How will you ensure traceability for test coverage?
How will tasks be allocated?
How will users know what to do and when to do it?
How will progress and KPI’s be measured?
How will test results be captured and analyzed?
Can aspects be automated?
What automation solution fits team, skills, timescales, approach and business objectives?
Are there existing tools or will you need to acquire them?
How will you create consistent test environments?
How will you communicate with product teams, business users, stakeholders and buy-ins?
Planning
Once the scope and strategy have been determined, map it out by a time unit that best suits the UAT manager, users (participants) and business objectives. It will need to be flexible to change as factors will inevitably impact it.
Outline the test plan
Outline the strategy into weekly segments
How realistic is this approach?
What are the risks?
Where are the compromises?
Get feedback from product teams, business users and stakeholders for the time process.
Determine resources needed
Assess the resources needed
Where do they need to be implemented?
Skills associated with the resources
How available and accessible are the resources
How available and accessible are the resources
Can they be supplemented?
Assess the role and impact of interfaces and other systems
Any capable and meaningful application and software undergoing UAT will connect to other applications, internally and potentially externally. This can complicate the change, which extends the coverage and nature of test cases and resources you may need to assess the impact of changes as many businesses operate over several integrated solutions.
Things to consider
Will the impact on other software mean more teams need to be involved?
Do you have existing linked test environments to support this end-to-end testing?
The impact of changes on test cases and data
Preparation and Initiation
There are many things to consider in developing a plan; this covers the majority of considerations but may vary from project to project.
Understand and capture business processes
Know how the business operates to be able to test a change in an area. Is this information to be gained via documentation or key users?
The business processes may change, and as the UAT is implemented in a separate environment, you will need to know the new business process changes, including inputs, outputs, and outcomes.
You may want to document this understanding and associated data capture using a classic structure. However, the downside of this is it does not lend itself to multiple scenarios. In this case, you will need to support any process definition with the data variations that are required, both for input and output.
If the project is largely revisions and additions to an already existing system, you may want to capture the existing process from production if the knowledge is not already available. You can use this to create text style instruction cases or even how-to videos, which users can refer to alongside data scenarios.
The UAT management tool should be used here to provide a) a repository for these assets and b) the ability to organize and hence reuse them.
Create or adapt test cases
It is important to look at Test Caser coverage, identifying if you have covered all the processes and scenarios that need to be tested. This analysis is critical, and it may require considerable input from business users to create them.
Some organizations opt to delegate this task entirely to the users, which is often the best route; however, if you take this approach, you will need inspection and reconciliation to ensure that you have achieved the necessary coverage level.
Check coverage
Two simple things you’ll need to consider for the coverage:
- Do the test cases provide full coverage of the test scope?
- Maintain traceability between changes/requirements and test cases
Detail in test cases may vary
It is possible to perform effective UAT with very lightweight cases, but this depends on the system knowledge level in the users performing tests. If you don’t require extensive detail, there is no reason to put it in the test case.
Historically, detailed test cases and scripts also helped define what the tester was doing so any issues could be described in the context of that series of events. Modern UAT, however, uses capturing technology showing precisely what the user was doing and documenting their concerns and feedback.
Regardless of the approach, your UAT will take, test cases have become valuable assets that we want to re-use, and hence they need to be stored and organised in a way that supports that.
Things to consider
Organize test case repository and classification
Depending on who is preparing test cases, what help and guidance do they need?
Amend existing test cases, if available, to match the changes/requirements
Create test cases where they are needed
Consider the sequence and dependencies
Create test scripts
The same principles as outlined for test cases apply to any scripts which take instructions to the next level. The coverage and level of detail in relation to the skills and knowledge of the persons executing the tests. A purpose–designed repository that makes scripts accessible, maintainable, referenceable, and reusable is fundamental for increasing organisational efficiency and avoiding frustration.
Things to consider
Create a standard script template if there is not an existing one.
Consider if detailed scripts for each test case are required.
Base the scripts on captured existing processes, if available.
The UAT management platform should support a script repository and linked to test cases.
Consider what format to hold the scripts in.
Are training aids required to assist in script execution?
Determine test data needs and strategy
There are two aspects of data that need to be initially considered:
Data aspects to consider
What data is needed to be processed
What supporting data is required
You’ll need to consider where the required data will come from, and this is inevitably based on what data is already available. This data needs to be treated as an asset; you should aim not to rebuild it or recreate it every time you go through a test cycle. Automation is sometimes used in this scenario to automatically create the data needed for subsequent testing and to carry it out when it is difficult to manage the condition of supporting data. However, the use of automation can be avoided in these scenarios by using a solution to manage data, this will allow for automation to be implemented in other areas which would benefit from it more.
You can avoid this by utilizing available technology and virtualized environments, which will save considerable time and effort.
Things to consider
Plan where and how to provide test environments(s).
Can virtual environments be used?
Consider the backup and reset strategy for the data.
What supporting technology is required?
You will probably need to coordinate with IT, infrastructure teams.
Test the test scripts
If you have created test scripts to be given to the users, then you will need to check, for at least some of them, that they are workable and understandable. If the users have created them you probably won’t need to.
Things to consider
Check both the content and format
Liaise with users to check their needs and the script or the scripts they have developed
Risk and priority assessment
There are times when a project doesn’t go to plan. To successfully achieve a competent and effective UAT, it is important to understand the critical testing areas and those that can be deemed as lower-risk. If the situation arose, it could be cut from the test. Through understanding these risks, you will need to prioritise important tests first.
Things to consider
Which tests are most important?
Are any tests able to be skipped?
Detailled Plan
The nature of the strategy you are taking will determine the level of detail required. If you are allocating test tasks at an individual or perhaps department level, you will need a granular plan combining the task, test case, user and timescale.
If you’re adopting a looser ‘crowd’ style strategy, the allocation of individual tasks becomes less of an issue in the first instance, but the monitoring of who has done what (or has not been done) becomes more important.
The crowd approach broadly involves making test cases available to anyone who has the time and skills to carry them out and then monitoring the results with a pre-determined execution and pass level. The advantages of this are that it is more dynamic and resource availability determines the progress with a greater sense of visibility of the whole project creating greater accountability.
Collaborate with User department heads
Collaboration will allow for faster access to resources as and when needed through early planning of who will do what, when and what is needed.
Things to consider
Publish goals, approach and timescales company-wide
Determine skills, names and availability of seconded resources
Establish communication methods and frequency
Use a central platform to enhance visibility and remove the reporting burden.
Plan the sequence and dependencies of tests.
Allocate time, resources, and areas for exploratory testing.
Consider timescales.
Determine resources.
Plan communication and access to the central platform.
Exploratory testing
Exploratory testing is essential for ensuring high quality and successful UAT. The UAT’s value as a whole may depend on many factors, not least the influence you have over the final users. The less influence, the more exploratory testing you should conduct to understand the results and their accuracy better.
For example, suppose you have a small group of in-house users who are dedicated and will be committed to following the UAT process rigorously. In that case, exploratory testing will not be a major factor or relevant to your test. However, suppose the application is to be rolled out on a mass scale publicly and contains sensitive or valuable information. In that case, internal exploratory testing may be critical for ensuring your testing process is functional.
Exploratory testing may have been included in earlier quality assurance phases if the software is being developed internally. Still, it is also good practice for the end-user to test the software too before distribution.
Communicate the plan to managers and individuals
Hold briefing session as a group
Inform individual business units
Consider how they will be notified of changes and progress
Provide access to key stakeholders to central dashboards and metrics
Execute the plan
Depending on the team’s level of skills and knowledge, it may be beneficial to have workshops prior to executing the plan to get the team ready, understand the process, and mitigate their issues and concerns. They will need to understand how the process works and their expectations and set the standards for producing and gathering feedback.
The feedback will need to be specified to the UAT’s objective; if you are interested in understanding issues, you will need to communicate this efficiently to all participating stakeholders. Using a capture tool is essential to make the gathering of this data accurate and support the users’ experience.
Competent capturing tools should capture the inputs and screenshots against the target applications with the ability for users to add comments, feedback and issue information. UAT best practices have managers seeking feedback beyond the initial issue reports; encouraging this helps improve engagement and ownership whilst also reducing the possibility for conflict with development and configuration teams.
Surveys are another way for increasing user engagement providing the opportunity for a broader perspective on acceptance and readiness. It also helps highlight the differences between broadly accepted concerns against individual ones which do not have broad support. Ideally, the UAT management platform will support the concept of user surveys and sharing of the results.
Train users
It is important to train all users to ensure the success of the UAT.
Things to consider
Teach the testing process you want
How to access the central system
How to access the work allocation
How to execute task and capture test steps
Feedback process and standards
Execute the plan and monitor
Once the plan has been executed, ongoing monitoring is essential to ensure the UAT works efficiently and plan.
Things to consider
Ensure the central system is updated.
Assess progress and rate of progress
Deal with issues and concerns
Maintain team motivation.
Ensure required results achieved
Encourage good feedback
Communicate status with business users and stakeholders.
Monitor completion rates
Monitor forecast completion.
Monitor level of issues reported.
Provide a feedback loop to users of issue status.
Triage test results
Whilst a full-time quality assurance team may correctly identify and raise defects, it is often preferred with users testing to distinguish between their issues raised and defects of the system, combined with a triage process to determine which issues should be promoted to defects.
Triaging results can avoid swamping development teams with incorrect or duplicated defects reports, making it easy for users to report what they find. To be effective, good evidence and audit trails of the test result is needed, and this requires complying and easy to use capture technology.
Things to consider
Review feedback and issue reports.
Examine the collected evidence
Update central system with new issue status
Document and communicate defects to the product team.
Ensure the central system is updated.
Provide a feedback loop to users of issue status.
Review and manage
In an ideal testing process, the tests will have been developed to provide 100% coverage of the areas and scenarios needing testing. These will all have been executed correctly and passed, even if the process has undergone several cycles.
At the same time, users will not have significant questions or concerns and will have provided positive feedback about the software’s quality and readiness. Upon completing this, all shareholders will have agreed that the changes will positively impact the organisation and achieve project goals.
Reaching this position is largely dependent on the factors previously considered and implemented before this point; however, you will also need clarity of the project metrics to determine when this project status is approaching or arrives.
This data will need to have been collected as the project has progressed and cannot be realistically constructed in retrospect, so it needs to be reinforced into the management platform and methods from the start.
Assess overall status and readiness
Once the test has concluded, we recommend following these 6 steps:
6 steps
What are the test passes against failures? Any outstanding tests?
Are there any outstanding issues?
Are there any outstanding defects?
Has there been progress made against key metrics?
Assess the state of the surveys and content and review accordingly
Communicate the status of the UAT to stakeholders
Automation build strategy
Automation of tests can be extremely useful for UAT. It provides the most value in areas that have already ‘passed’ user inspection but will need retesting or re-execution either as part of a subsequent cycle in this project or a future release cycle where it forms the basis of regression testing. Automation can also provide value in data loading to support testing activities.
Whilst manual testing is essential for successful UAT, having users repeatedly test the same software will eventually lead to less accurate results from users becoming tired of repeating the same hypothetical task over and over again. However, if automated, UAT managers can still validate and accept the results without users repeating tests and still gain valuable test assets for future use.
To be successful, automation will need to be relatively easy to create, maintain and understand. By delegating some repetitive aspects of the testing scope to automation time, users time is freed up substantially. Through planning for automation at the start of the testing process, this freed up time can be redirected to exploratory testing.
Things to consider
Can past manual tests be converted to automation?
Where is the greatest gain?
Is the process knowledge available, captured?
What future requirement for regression testing can be addressed now?
Consider intensive, repetitive manual tests or tasks (such as data loading) for early automation.
Ensure you have selected an automation solution to match team skills and rate of progress.
Are there configuration and environment tasks that will need to be repeated when moved to production, captured in automation for consistency?
Repeat
If your UAT, Conference Room Pilot (CRP), or Model Office Testing (MOT) project has completed successfully with only one cycle, you are very lucky, or very good, or both! It is most likely that a number of cycles will be needed, perhaps with reducing effort and coverage as the testing progresses and confidence in some areas solidifies.
But this is also a time to be careful as spotlighting only the areas of latest concern may allow errors to sneak past in previously cleared tests that have not been repeated. This is where automation of these areas can pay dividends, providing confidence without the hard work
However, during this time, is it important to still be careful in reviewing the whole software and not only areas that have previously been concerned. This may result in undetected errors not being flagged up; using automation in these scenarios can provide confidence without committing more resources. Having got successfully through the first cycle of testing, the remaining tests will be easier and the plan considerably shorter.
Replan for testing
Determine the approach you wish to take
Ensure that you are testing this holistically and not just for repeated failures
What regression testing will be taking place – Whole, part, manual, automated?
Provide a data environment matching the tests needs
Apply automation to areas not needing manual testing
Re-baseline scripts to include changes
Review the results with users
Once all preparation for repeating the tests has been conducted, you will need to repeat the execute and review and manage sections of the handbook.
Retrospective
There will always be things that could have gone better, and during this phase, it’s important to reflect on these for improving future projects. Hopefully, the project will have been a triumph, and with the successful implementation, you will need to positively feedback the project’s success to all stakeholders to encourage future UAT participation within the organisation.
Analyze and critique the project
What was the level of issues found and was it acceptable?
What was the level of defects found and was it acceptable?
Where was time lost or gained?
How effective was the communication process?
How was the quality of the delivered results perceived?
Consider the level of questions and concerns after/during the UAT
Praise users and stakeholders where appropriate
Organize assets from the plan and results for future use
Collate assets and organize for future use
Test cases
Test scripts
Automated scripts
Creation of automated regression
Data and results
Adapt test policies if necessary
Once the reflection has been completed, adapt your test policy for future UATs
What strategic changes would have created a better framework?
Conclusions
The nature of User Acceptance Testing is bound to vary according to the project’s needs, the business, the users, and the overall risk. In some cases, it is the only form of testing an organization will experience, but even if it is not, UAT is the most expensive type of testing that can be performed.
Many Quality Assurance theorists and academics will be unhappy that cost is even mentioned in the context of testing. They will consider it irrelevant compared to the project’s goals and will argue this by making the comparison to the cost of not testing—a fair point. However, the cost is a real factor and a real constraint for many projects, and given that UAT is the most costly type of testing that can be performed, it is worth considering how the cost can be controlled and reduced. Thankfully, a cost reduction is a by-product of other more inspiring objectives and is achieved by efficiency, re-use, improving communication, and better testing approaches that drive out more issues in less time, with less effort. In the end, delivering an issue-free system, which the users know how to use and gain benefit from, will be the most valuable factor in the project’s success.
This guide has been created to help identify many of the topics and considerations which should be examined when looking to achieve the above goals. The use of technology is a recurring theme because it plays a valuable, indeed essential, part in making the improvements (compared to a manual process) to enable these goals to be achieved. In some organizations which are too IT and development-focused, testing is deemed to be only worthy of attention within the IT team, and once in the hands of the business users, it can be forgotten. Perhaps because by then, it is someone else’s budget, but the reality is that it impacts the business as a whole by increasing cost and reducing project success if not done well. The business case for investment in improving this process is usually easy when considering the amount of time that can be saved and reducing issues that hit production systems.
No organization wants to be the next ‘Software testing failure’ headline, and UAT is a crucial component to maximize the quality of the delivered solution and stay out of the news for the wrong reasons.
About Original Software
The Ultimate UAT Guide has been brought to you by Original Software. Our mission is to deliver quality software, that is easy to buy, has an intuitive UI, and is loved by our clients. Original Software enables organizations to meet their testing objectives rapidly, by delivering innovative solutions to support application quality.
Our software provides the fastest way to capture and share business processes, validate application functionality, and manage projects in real-time.
If you’re only just starting out with UAT or have been conducting it for years, get in touch with us today to discover how our Qualify UAT approach can dramatically improve the efficiency of your process.
Our clients have reported time and effort savings of 60%
