User Acceptance Testing (UAT) is inevitably near the end of any software change cycle, and it performs a number of key objectives in the overall software quality process. It serves to ensure that the software is usable, safe, understood, and meets the objectives laid out in the first instance.
The word ‘acceptance’ has become less important in most current forms of UAT as it is not typically used to ‘accept’ the software or system against a set of acceptance criteria, more it is an additional set of checks and at the same time provides familiarization and training so that the implementation is smooth and successful.
Organizing, managing, and executing any type of testing or testing project can be a substantial and daunting task, and UAT is no exception. For every project that UAT is implemented in, there are unique challenges and factors to consider. However, the effort is worth it, and in many cases, essential for the success of the project and saves time and money in the long run.
However, these results are entirely dependent on the appropriate implementation of testing and good input from users. Without this, the software’s residual errors will not be discovered negatively impacting the change of systems or causing costly disruption and discontent internally and externally.
Any UAT phase is likely to be the most costly type of testing undertaken due to the number of people, timescales, and repeated cycles. Hence, it is well worth optimizing this activity to get the best outcome and do it in the most efficient and cost-effective way.
This handbook looks at the potential activities that might apply to a UAT project. It provides the knowledge for putting in place frameworks that give a project the best chance of success and maximize positive outcomes from each activity and the overall project.
As with any testing process, not all phases need to be implemented within every project, but each should be considered to achieve the best possible outcome.
Initially, you need to determine the scope of your UAT testing through three stages.
Whilst implemented at the end of projects; it is advisable to start early and engage with all teams involved to ensure UAT is on the radar and considered in their planning. Often, departments consider the challenges and defects that will arise but do not realize that UAT identifies defects for fixing and overcoming.
When determining timescales, you also need to consider where resources will need to be implemented and the overall strategy which the UAT is being implemented to achieve. Part of the planning process will involve determining why the project is being carried out and the benefit which the organization is hoping to gain from the change.
Once the benefit is realized, it must be effectively communicated to all UAT participants so they understand what they need to be looking out for whilst testing to achieve success.
It’s necessary to determine the scope of the project to understand the changes that the business is hoping the software will bring and the resources and time needed to test it sufficiently.
When analyzing the change, if it is a unique software a business is developing, you can communicate expectations from previous teams and phases, including development and quality assurance, to understand the software’s expectations.
If it is a cloud-based software or package being implemented, obtain data from other users via forums and other channels to gauge the scale of the testing and resources needed.
By conversing with those involved with the system, that being the system architects if developing software internally or with the software provider, determine the impact of changes on the business.
It is always important to remember that whilst the scale of a project and impacts can be closely related; it is not for certain that small changes only result in small impacts but rather result in disproportionately large impacts to the business.
Therefore, it is important to assess each change accordingly and determine its impact to efficiently plan User Acceptance Testing
Once a plan is put in motion, creating a comprehensive and holistic strategy is important for successful UAT implementation.
Different organizations have different cultures and business practices; some may have developed from historic need and others from dynamic approaches and may have pre-existing test policies.
Having these, however, do not mean they will be appropriate for every testing, and reviewing them to determine if they can be implemented needs to be determined early on. However, if you’re going to adapt these policies for the test, this also needs to be outlined early in the process.
If your organization does not have test policies, take the opportunity to outline a policy for future guidance and best practice but be mindful of how testing is dynamic and policies may not fit every scenario.
Once the plan and initial preparations have been completed, determine a strategy that accounts for everything discovered from the preparations.
As with every test, using specific technology can make life considerably easier. There are 4 key areas to consider
1) You may need a platform to manage the projects’ assets combined with communication, resources assignment and scheduling. Fundamentally, it will act as a test management solution to support the users and the project’s needs.
The working and strategy undergoing may have different facets depending on the approach being taken. For example, you may have ‘issues’ that do not fall under ‘defects’ or ‘feedback’ segmented by positive, negative and neutral results. Regardless of your approach, your test management platform will need to be designed to support the style of UAT being planned and the users’ needs.
Technology capturing user actions unobtrusively and accurately with little to no effort, especially if you plan on implementing exploratory testing. Historically, the capturing of results is one of the most frustrating aspects of UAT, with users getting tired of documenting via print screen and pasting to Word when errors occur and manually detailing what happened.
This historic method is unreliable as it depends on users being honest and meticulous in documenting the process. There are inexpensive solutions for capturing the process, which can be easily implemented and take the pressure off users.
UAT needs to be repeated to ensure the results are as accurate as possible; however, after several testing cycles, automation can be implemented to seamlessly run the software, checking for issues that manual testing has overlooked.
The testing process needs a test environment and ideally consistent data to go with it. Test data management tools help handle the data collected, ensuring it is easy to validate when integrated with automation and capturing user actions technology.
Before implementing technology for the project, there are factors that need to be considered:
Once the scope and strategy have been determined, map it out by a time unit that best suits the UAT manager, users (participants) and business objectives. It will need to be flexible to change as factors will inevitably impact it.
Assess the resources needed
Any capable and meaningful application and software undergoing UAT will connect to other applications, internally and potentially externally. This can complicate the change, which extends the coverage and nature of test cases and resources you may need to assess the impact of changes as many businesses operate over several integrated solutions.
There are many things to consider in developing a plan; this covers the majority of considerations but may vary from project to project.
Know how the business operates to be able to test a change in an area. Is this information to be gained via documentation or key users?
The business processes may change, and as the UAT is implemented in a separate environment, you will need to know the new business process changes, including inputs, outputs, and outcomes.
You may want to document this understanding and associated data capture using a classic structure. However, the downside of this is it does not lend itself to multiple scenarios. In this case, you will need to support any process definition with the data variations that are required, both for input and output.
If the project is largely revisions and additions to an already existing system, you may want to capture the existing process from production if the knowledge is not already available. You can use this to create text style instruction cases or even how-to videos, which users can refer to alongside data scenarios.
The UAT management tool should be used here to provide a) a repository for these assets and b) the ability to organize and hence reuse them.
It is important to look at Test Case coverage, identifying if you have covered all the processes and scenarios that need to be tested. This analysis is critical, and it may require considerable input from business users to create.
Some organizations opt to delegate this task entirely to the users, which is often the best route; however, if you take this approach, you will need inspection and reconciliation to ensure that you have achieved the necessary coverage level.
Two simple things you’ll need to consider for the coverage:
It is possible to perform effective UAT with very lightweight cases, but this depends on the system knowledge level of the users performing tests. If you don’t require extensive detail, there is no reason to put it in the test case.
Historically, detailed test cases and scripts also helped define what the tester was doing so any issues could be described in the context of that series of events. Modern UAT, however, uses capturing technology to show precisely what the user was doing and document their concerns and feedback.
Regardless of the approach, your UAT will take, test cases have become valuable assets that we want to re-use, and hence they need to be stored and organized in a way that supports that.
The same principles as outlined for test cases apply to any scripts which take instructions to the next level. The coverage and level of detail in relation to the skills and knowledge of the persons executing the tests. A purpose-designed organizational repository that makes scripts accessible, maintainable, referenceable, and reusable is fundamental for increasing organizational efficiency and avoiding frustration.
There are two aspects of data that need to be initially considered:
You’ll need to consider where the required data will come from, and this is inevitably based on what data is already available. This data needs to be treated as an asset; you should aim not to rebuild it or recreate it every time you go through a test cycle. Automation is sometimes used in this scenario to automatically create the data needed for subsequent testing and to carry it out when it is difficult to manage the condition of supporting data. However, the use of automation can be avoided in these scenarios by using a solution to manage data, this will allow for automation to be implemented in other areas which would benefit from it more.
You can avoid this by utilizing available technology and virtualized environments, which will save considerable time and effort.
If you have created test scripts to be given to the users, then you will need to check, for at least some of them, that they are workable and understandable. If the users have created them you probably won’t need to.
There are times when a project doesn’t go to plan. To successfully achieve a competent and effective UAT, it is important to understand the critical testing areas and those that can be deemed as lower-risk. If the situation arose, it could be cut from the test. Through understanding these risks, you will need to prioritize important tests first.
The nature of the strategy you are taking will determine the level of detail required. If you are allocating test tasks at an individual or perhaps department level, you will need a granular plan combining the task, test case, user, and timescale.
If you’re adopting a looser ‘crowd’ style strategy, the allocation of individual tasks becomes less of an issue in the first instance, but the monitoring of who has done what (or has not been done) becomes more important.
The crowd approach broadly involves making test cases available to anyone who has the time and skills to carry them out and then monitoring the results with a pre-determined execution and pass level. The advantages of this are that it is more dynamic and resource availability determines the progress with a greater sense of visibility of the whole project creating greater accountability.
Collaboration will allow for faster access to resources as and when needed through early planning of who will do what, when and what is needed.
Exploratory testing is essential for ensuring high quality and successful UAT. The UAT’s value as a whole may depend on many factors, not least the influence you have over the final users. The less influence, the more exploratory testing you should conduct to understand the results and their accuracy better.
For example, suppose you have a small group of in-house users who are dedicated and will be committed to following the UAT process rigorously. In that case, exploratory testing will not be a major factor or relevant to your test. However, suppose the application is to be rolled out on a mass scale publicly and contains sensitive or valuable information. In that case, internal exploratory testing may be critical for ensuring your testing process is functional.
Exploratory testing may have been included in earlier quality assurance phases if the software is being developed internally. Still, it is also good practice for the end-user to test the software too before distribution.
Depending on the team’s level of skills and knowledge, it may be beneficial to have workshops before executing the plan to get the team ready, understand the process, and mitigate their issues and concerns. They will need to understand how the process works and their expectations and set the standards for producing and gathering feedback.
The feedback will need to be specified to the UAT’s objective; if you are interested in understanding issues, you will need to communicate this efficiently to all participating stakeholders. Using a capture tool is essential to make the gathering of this data accurate and support the users’ experience.
Competent capturing tools should capture the inputs and screenshots against the target applications with the ability for users to add comments, feedback, and issue information. UAT best practices have managers seeking feedback beyond the initial issue reports; encouraging this helps improve engagement and ownership whilst also reducing the possibility of conflict with development and configuration teams.
Surveys are another way for increasing user engagement by providing the opportunity for a broader perspective on acceptance and readiness. It also helps highlight the differences between broadly accepted concerns against individual ones which do not have broad support. Ideally, the UAT management platform will support the concept of user surveys and sharing of the results.
It is important to train all users to ensure the success of the UAT.
Once the plan has been executed, ongoing monitoring is essential to ensure the UAT works efficiently and plan.
Whilst a full-time quality assurance team may correctly identify and raise defects, it is often preferred with users testing to distinguish between their issues raised and defects of the system, combined with a triage process to determine which issues should be promoted to defects.
Triaging results can avoid swamping development teams with incorrect or duplicated defects reports, making it easy for users to report what they find. To be effective, good evidence and audit trails of the test result are needed, and this requires complying and easy-to-use capture technology.
In an ideal testing process, the tests will have been developed to provide 100% coverage of the areas and scenarios needing testing. These will all have been executed correctly and passed, even if the process has undergone several cycles.
At the same time, users will not have significant questions or concerns and will have provided positive feedback about the software’s quality and readiness. Upon completing this, all shareholders will have agreed that the changes will positively impact the organization and achieve project goals.
Reaching this position is largely dependent on the factors previously considered and implemented before this point; however, you will also need clarity of the project metrics to determine when this project status is approaching or arriving.
This data will need to have been collected as the project has progressed and cannot be realistically constructed in retrospect, so it needs to be reinforced into the management platform and methods from the start.
Once the test has concluded, we recommend following these 6 steps:
Automation of tests can be extremely useful for UAT. It provides the most value in areas that have already ‘passed’ user inspection but will need retesting or re-execution either as part of a subsequent cycle in this project or a future release cycle where it forms the basis of regression testing. Automation can also provide value in data loading to support testing activities.
Whilst manual testing is essential for successful UAT, having users repeatedly test the same software will eventually lead to less accurate results. Users will become tired of repeating the same hypothetical task over and over again. However, if automated, UAT managers can still validate and accept the results without users repeating tests and still gain valuable test assets for future use.
To be successful, automation will need to be relatively easy to create, maintain and understand. By delegating some repetitive aspects of the testing scope to automation time, users’ time is freed up substantially. Through planning for automation at the start of the testing process, this freed-up time can be redirected to exploratory testing.
If your UAT, Conference Room Pilot (CRP), or Model Office Testing (MOT) project has completed successfully with only one cycle, you are very lucky, or very good, or both! It is most likely that a number of cycles will be needed, perhaps with reducing effort and coverage as the testing progresses and confidence in some areas solidifies.
But this is also a time to be careful as spotlighting only the areas of latest concern may allow errors to sneak past in previously cleared tests that have not been repeated. This is where automation of these areas can pay dividends, providing confidence without the hard work
However, during this time, is it important to still be careful in reviewing the whole software and not only areas that have previously been concerned. This may result in undetected errors not being flagged up; using automation in these scenarios can provide confidence without committing more resources. Having got successfully through the first cycle of testing, the remaining tests will be easier and the plan considerably shorter.
Determine the approach you wish to take
Ensure that you are testing this holistically and not just for repeated failures
What regression testing will be taking place – Whole, part, manual, automated?
Provide a data environment matching the tests needs
Re-baseline scripts to include changes
Review the results with users
Once all preparation for repeating the tests has been conducted, you will need to repeat the execute and review and manage sections of the handbook.
There will always be things that could have gone better, and during this phase, it’s important to reflect on these for improving future projects. Hopefully, the project will have been a triumph, and with the successful implementation, you will need to positively feedback the project’s success to all stakeholders to encourage future UAT participation within the organisation.
What was the level of issues found and was it acceptable?
What was the level of defects found and was it acceptable?
Where was time lost or gained?
How effective was the communication process?
How was the quality of the delivered results perceived?
Consider the level of questions and concerns after/during the UAT
Praise users and stakeholders where appropriate
Collate assets and organize for future use
Test cases
Test scripts
Automated scripts
Creation of automated regression
Data and results
Once the reflection has been completed, adapt your test policy for future UATs
What strategic changes would have created a better framework?
The nature of User Acceptance Testing is bound to vary according to the project’s needs, the business, the users, and the overall risk. In some cases, it is the only form of testing an organization will experience, but even if it is not, UAT is the most expensive type of testing that can be performed.
Many Quality Assurance theorists and academics will be unhappy that cost is even mentioned in the context of testing. They will consider it irrelevant compared to the project’s goals and will argue this by making the comparison to the cost of not testing—a fair point. However, the cost is a real factor and a real constraint for many projects, and given that UAT is the most costly type of testing that can be performed, it is worth considering how the cost can be controlled and reduced. Thankfully, a cost reduction is a by-product of other more inspiring objectives and is achieved by efficiency, re-use, improving communication, and better testing approaches that drive out more issues in less time, with less effort. In the end, delivering an issue-free system, which the users know how to use and gain benefit from, will be the most valuable factor in the project’s success.
This guide has been created to help identify many of the topics and considerations which should be examined when looking to achieve the above goals. The use of technology is a recurring theme because it plays a valuable, indeed essential, part in making the improvements (compared to a manual process) to enable these goals to be achieved. In some organizations which are too IT and development-focused, testing is deemed to be only worthy of attention within the IT team, and once in the hands of the business users, it can be forgotten. Perhaps because by then, it is someone else’s budget, but the reality is that it impacts the business as a whole by increasing cost and reducing project success if not done well. The business case for investment in improving this process is usually easy when considering the amount of time that can be saved and reducing issues that hit production systems.
The Ultimate UAT Guide has been brought to you by Original Software. Our mission is to deliver quality software, that is easy to buy, has an intuitive UI, and is loved by our clients. Original Software enables organizations to meet their testing objectives rapidly, by delivering innovative solutions to support application quality.
Our software provides the fastest way to capture and share business processes, validate application functionality, and manage projects in real-time.
If you’re only just starting out with UAT or have been conducting it for years, get in touch with us today to discover how our approach can dramatically improve the efficiency of your process.
© 2022 Original Software. All rights reserved
We are sorry to tell you that using Internet Explorer as your browser won’t give you the best experience of this website.
To get the best value visit us via Chrome, Edge or Firefox
