Main image of article How to Manage User Acceptance Testing

User acceptance testing (UAT) has been an ongoing topic of influence for me in my career. It’s particularly interesting because I’ve typically been on the IT side of things. But business organizations/partners often don’t have the knowledge or aren’t willing to do take part in testing for themselves. So, I’ve started to just volunteer to lead and manage UAT to ensure the stress of managing a foreign process is removed from the team. It also goes a long way toward building relationships with business partners. Here’s the process I use to manage an effective UAT session. The goal is to keep the process organized, contain the time frame and lead to a success. It may not be perfect, but it does provide a good general starting place for PMs without a lot of UAT experience.

1. Determine Correct User Roles

One of the most important items to define first is who from the team will be involved in the UAT process. There are a couple roles to consider -- the UAT owner, project sponsor/business owner and the UAT team. The UAT owner (or PM) will manage the process and own any final decisions with the team. This does not necessarily mean that this person can push decisions, but he or she owns the responsibility to determine the best next steps and execute them. The UAT owner is responsible for updating the business owner or project sponsor on the status of the tests, engaging them in decisions and managing the work for the actual testers. The project sponsor or business owner is responsible for the project’s requirements and for guiding the UAT owner in testing for them. This person is consulted in decisions for defects or changes reported and should be included in any Change Control discussion. The sponsor should own the responsibility for the Change Control items that are agreed to and manage them through the proper channels to get additional funding and approvals when necessary. The UAT team will need to be filled with resources who are willing and have the time to test the product from end-to-end. This is strictly user-based testing with support from the development teams. These resources will test according to the defined schedule and test plan and provide feedback according to that plan as well.

2. Choose the Type of Testing

Either in-person or self-paced, the type of testing is defined by the location of your team members and whether or not you need multiple shifts of tests. If you have a co-located team or teams within the same time zone, it’s very easy to book a room and manage a united UAT testing format. On the other hand, if resources are available in shifts or time zones are not conducive for a single testing time frame, self-paced testing can be a better choice. Also, Keep in mind that it’s possible to manage both types of testing. You can have a larger group of resources that can work in-person and a few that need a more flexible approach. You should be flexible to the needs of the team (within reason).

In-Person Testing

In-person testing is the method I prefer, for several reasons. It contains the UAT testing to a defined time frame and tends to be very controlled. It requires the team to be centrally located (either in person or by teleconference) and it lessens the chances of the UAT participants to get distracted. It also reduces the noise of "what to test” during UAT. When in-person testing is used, test cases are typically managed more tightly and documentation is more immediate.

In-person testing should take between two to four hours and should be a relaxed time for users to interact. The development team should be available to answer any questions and provide immediate feedback. It also gives the team a chance to interact in a more social environment and helps continue to build those relationships.

The drawback of in-person testing is that the team must dedicate significant time away from their daily roles to participate. While it’s only a perception that this method takes more time, sometimes that perception is all that is necessary to complicate this approach. It does require more planning to ensure the proper venue is available with appropriate testing machines.

Self-Paced Testing

While I prefer in-person testing more, self-paced testing also has some advantages. It still requires planning, but the testing is more relaxed and can occur anytime throughout the day. Some users prefer it simply because their role will not allow for two to four hours away from their desk. This method allows them to complete the testing during off hours. Also, many companies have users across the globe and will encourage the use of these resources.

Some challenges of self-paced testing include keeping up with testers and defining check points for the team, ensuring that proper test cases are being defined and used throughout testing, controlling what to test (basically the scope of the project) and enabling the team to test scenarios that are not in scope.

3. Determine Time frames

When defining the project plan at the beginning of a project, you should always include a placeholder for UAT for the standard time frame your company expects. Most places accept two weeks, but it should be defined for each project. There are several items to consider when defining time frames. They include:

  • Environment Standards: These typically relate to the standard project plan defined by the group you work with. As stated above, most places allow two weeks -- ten business days -- to complete UAT. If the scope and requirements were defined properly and the right care was taken during the project to review progress along the way, two weeks should be more than enough for the UAT team to go through and test the system to ensure it functions as expected.
  • Scope of Work: There are some rare occasions that UAT should take longer. This may be if there is an extensive amount of UI testing or user-driven changes that should be accounted for. Cases where UAT needs to be longer should be recognized and estimated early enough in the project to ensure the timeline doesn’t slide.
  • Availability of Team: While the availability of the team should have some influence, it can’t completely dictate when UAT happens. It's important to consider environmental constraints that could affect the team’s availability for UAT before the time frame hits. There may be a particularly busy time of year/quarter that resources just can’t be available. This should be discussed up front when you are building your initial timelines.
  • Venue Availability: If an in-person testing method is used, the time frame will depend on the availability of a proper venue with machines for testing.

4. Determine Documentation Standard

Documentation is essential for UAT. It's the responsibility of the IT PM to ensure that documentation is created and maintained throughout the project. UAT is not typically the responsibility of the IT PM, so I’ve included a list of standard documentation that is important to create and maintain if you lead this effort for your business partner. Testing Strategy and Plan: The test strategy and plan should outline the following:

  • Owners and participants.
  • Scope of work and how to test new functionality as well as Out of Scope scenarios for testing.
  • Venue and outline for testing expectations.
  • General agreements and standards for pass/fail outcomes.
  • Test Cases/Use Cases.

The inclusion of test cases and use cases is essential for any UAT to be successful. By successful I mean for it to be organized, structured and easy to manage. The use cases test how the user will interact with the new tool and help define requirements to build the new software or application. The test cases are based on use cases and requirements to test the new software or application. Test cases should be documented in a way that they can be tracked and commented on throughout the process. Test Case Outcomes: I typically create templates in Excel for tracking the outcomes of the test cases and their impact because they provide several options to filter and sort that are very helpful in testing. This document is defined typically by the UAT team to help guide the tester in their testing time frame. They should use this document along with some ad-hoc testing to manage UAT. (You can see a sample of the spreadsheet here.) Below is my suggestion for columns this document:

  • Business Requirement # (BR#)
  • Name of Tester
  • Date Executed
  • Test Case Unique ID (or number)
  • Test Case Name
  • Test Case Steps Defined
  • Business Impact - High/Medium/Low
  • Acceptance Criteria
  • Expected Outcomes
  • Pass/Fail
  • Comments

Testers should use the document throughout testing as follows:

  • For Tracking Tests: Testers need to record the results of the tests, which should be reviewed on a daily or agreed-upon basis. The testers should document all comments and statuses to enable the team to track progress and issues during UAT.
  • For Reporting Issues: As issues are found, they need to be managed and answered. There are two types -- defects and new items. The terminology may be different, but basically it is either functioning as the requirements say it should or it isn’t. If something wrong is found, but it’s still “as expected,” then the users have a choice to create a Change Control to work through new items.

Requirements for Sign-Off: Most QA teams have requirements that must be met to move from the DIT environment to the SIT environment and from the the SIT to the UAT environment. The users should then have a set of requirements to move from UAT to Production. I suggest that the users define (and document) a set of "show stoppers" and a means of measuring high, medium and low impact test cases. Any issues with test cases that are classified as anything but "low" should be solved before moving into production. Solving these issues may mean code fixes or some other solution, but these are not typically deferred. The only things that may be deferred are low priority items.

5. Determine the Change Control Process

Changes may come up during UAT. You need to have a standard process that you can follow for making these changes when they are needed. In one of my last projects, we had several items of scope that were not determined until UAT. This caused some significant issues with the production date. We were able to leverage the existing change control process and analyzed the impacts to the cost, timeline and scope to determine how to manage going forward. We ended up not changing our scope and production dates due to the perceived costs of the change, but created a supplemental production date to input the changes properly. All of this is a lot to consume and manage, especially if you're not used to doing it. Remember to speak with your manager to make them aware of the additional effort and cost associated with managing a UAT. Depending on the time frame, it can be a costly process. Regardless, be upfront with all parties involved and make sure that communication stays open throughout the process. That is one way to ensure that you have a successful UAT. Do remember, also, that while it's fine to lead your business partner through the UAT process, you need signoff to move forward to production. Signoff can be simply an email or other formal way to document that the UAT lead person and business partner agree the software or application fills the requirements and is ready to go into production.