Sunday, August 12, 2018

These 6 hidden reasons explain why your outsourced software testing isn't working

Far too often we receive mayday calls from prospects who have invested a lot of money, but more importantly time, into selecting and working with a outsourced test automation services provider, only to be rewarded with a never-ending tug of war that inflicts much damage to their professional reputations. At the root of these frustrating and poor outcomes is a test automation service provider who was either out of their depth, unwilling to own up to failure and probably did not fully understand the customer's desired future state.

In the case of outsourced test automation services, most customers are trying to prevent losses of the magnitude reported by Computer Weekly magazine from the UK:

One analyst, who follows the outsourced software testing space closely, believes:
Cost savings are a major driver and also the need to do testing in a more professional manner.
Dominique Raviart, analyst at Nelson Hall

While he is definitely on the mark, it is obvious to us that the thirst for cost savings often drives a disproportionate share of decision-making when selecting outsourced software testing providers. This only serves to diminish the importance of and attention to actually conducting "testing in a more professional manner." In fact, as far back as 2007, Gartner predicted:

Although the statistic above related to the BPO industry, our experience tells us that this would also hold true for outsourced software testing engagements. Gartner research director Alexa Bona put that finding down to high staff attrition rates suffered by many outsourced service providers. If you think about it, there are no free lunches, right? If you pay peanuts, you get...

While some software testing outsourcing engagements fail because of technical reasons, it is our analysis and that of other storied industry veterans, that most fail because of misaligned priorities, inadequate discovery and the IT services industry's propensity to try and fit square blocks in round holes.

At Audacix, we ask a lot of questions. Especially at the start of an engagement, but also throughout each project. This process of understanding each another is critical to building confidence and setting up the relationship for mutual wins. Contrary to popular belief, we have evidence that this process must start before the agreements are signed. Starting it later down the line leaves too much room for miscommunication and misalignment.

We have seen that the following 6 questions are often not asked or only broached very superficially during the "courtship" phase between a customer and their software testing services provider. It's almost never as simple as answering these questions alone, but they are a great starting point to understand your software testing service provider's capability to help you achieve your goals and deliver on their promises.

The 6 hidden reasons below address critical knowledge gaps that are the primary causes of failure in outsourced software testing engagements. These 6 reasons are, in fact, a set of 6 questions that you should ask yourself before your outsource your software testing. Answering these questions for yourself even if the outsourcing service provider does not ask you these questions will help give any engagement a much higher chance of success.

1. What is the big picture?

As an outsourced software testing service provider, this helps us quickly prioritise wins for potential engagement. This prioritisation is not based on our goals, but those of our customer. Clarifying the priority of outcomes helps us understand if the customer is looking for fast and effective test automation services that really kick goals, or just an attempt to get some cost transferred elsewhere.

By no means is attempting to achieve cost savings an unworthy goal. However, if cost saving is the primary reason you want to outsource your software testing then you will do well to find other goals and benefits you want to realise from such an engagement. Financial and non-financial reasons together will help you form a lasting and mutually fruitful relationship with your software testing partner.

If your software testing outsourcing partner has not asked the overarching goals for engaging, then you should explain these goals and link any performance evaluation to the achievement of these goals. Mutual and detailed understanding of the desired future state is the bedrock on which your software quality will flourish.

The future state should define quantifiable and measurable metrics covering software quality, internal and external resourcing, timeliness and investment targets.

Want to see how our testing tools & services can cut your testing time from weeks to hours?

2. What do you expect from us and this project?

The desired future state you described above is only going to be achieved if the combined labour of you and your outsourcing service provider produces the desired results. Transferring all project responsibility and accountability to your software testing service provider is the fastest way to guarantee a failed engagement.

We ask this question to understand 2 things:
  • How involved do you want to be throughout the project in terms of time and thought leadership; and
  • How do you view our roles in the engagement changing over time?
The answers to this question also help to clear many misconceptions that are very common in outsourcing engagements. Such differences often lead to increasing distance and friction between both parties. You can minimise, if not eliminate, the "crossed wires" and "we thought you were taking care of that" moments by clearly outlining responsibilities before and consistently throughout the engagement.

This clarity can be a big boon in outsourcing projects where you want your outsourcing partner to develop a test practice for you, then leave you to run it. Constant, structured and unstructured, communication is just as important with an external party as it is between internal teams and colleagues.

3. What will achieving these goals do for you?

This answer really tells a story about your priorities. Are your non-financial priorities at least on par with your cost saving goals? Are you really serious about finding bugs before they get into production and shifting left with your software quality or are you more concerned about shifting risk?

Prioritising these goals before you begin the selection process will help you make a more informed decision about the outsourced testing provider who best suits your needs. Some outsourcing partners will be happy to simply help you achieve your cost targets, whereas others may decide that they would rather concentrate on projects where cost reduction is one element of a good outcome.

How do you ACTUALLY know if your software testing is working? You must have a dashboard to easily track key testing metrics.

4. How will we communicate?

Phone, email and Slack are all relevant and useful modes of communication. The trick is to be clear about which communication medium to use for different needs. Many stalwarts of this industry also produce very fancy looking PDF or Excel reports prior to every progress meeting. But, how do you really assess project progress?

To remove this information asymmetry we developed a real-time dashboard as part our Qsome automated software testing platform. This dashboard is not only the central source of truth for project progress, because our test engineers see exactly what our customers see, but it also helps our customers make highly objective decisions about if and when they release to production. We find that removing the information gap between both parties is a critical component of the trust-building exercise.

By eliminating any doubts about the veracity of the information being discussed, our customers really appreciate being empowered to concentrate on conducting more frequent and more productive discussions about how we can all achieve their desired future state even faster.

Ask yourself: how could objective and accurate real-time data exchange between you and your outsourced software testing partners improve the results that are delivered by your team for the organisation? Think about it.

5. Have we understood you properly?

The clincher! All the thinking and explaining and strategising you have done to date are of no value if you and your outsourced software testing partner are not on the same page.

A simple technique that we ask our team to practice during sales meetings and progress review meetings is to explain what they think they have understood in their own words back to the customer. Communication experts will tell you that nothing promotes clarity and consistency like repetition.
Any idea, plan, or purpose may be placed in the mind through repetition of thought.
Napoleon Hill
This conversation quickly brings unrealistic expectations to the fore. By repeating your goals and expectations aloud back to you, we are able to quickly isolate communication gaps and address them.

If you picked up a strong whiff of communication being the central theme behind these questions, then you are on point. Good communication is an abstract and ambiguous concept to those who miss the forest for the trees. In fact, you know communication is completely absent if someone believes they have communicated well with their outsourced test automation partner because the "legal agreement is watertight!"

The brutal truth is that if any commercial relationship devolves into both parties quoting from the contract, you may as well close the door on achieving any goals. While this is an unfortunate fact of life, you can minimise the chances of suffering this fate if you and your software testing outsourcing partner understands both your financial and non-financial goals.

6. What happens when things fail?

Failure isn't fatal, but failure to change might be.
John Wooden
And therein lies the crux of the most common problems with most outsourced software testing service providers. The ability and willingness to acknowledge, analyse and learn from mistakes, both your own and those of others, is a very noble and productive human characteristic, but sadly and uncommon one.

We have a very clear and concise three-pronged system for communicating and addressing failure within our projects. The surprising thing is how few prospective customers actually ask us about this during the sales process. There are 2 reasons you must ask this question of all your external partners:
  • If the external partner gets uncomfortable and provides glib answers, you have to really wonder whether it is worthwhile continuing discussions; and
  • This answer allows you to assess whether the outsourcing partner's values align with those of your organisation.
I've found that the alignment of values is the most underrated and overlooked aspect of assessing potential outsourced software testing partners. Think about it, how many times has your assessment list for potential partners included something around "aligned valued"?

Every question we ask during the sales process for our engagements is aimed at taking our communication to the next level, but also at ensuring that our values (play with heart, stand out and lead a revolution and become a world champion) will complement and not grate with those of our prospective customers. How do your values align with those of your outsourced software testing partners?

If you need help in cutting testing time and finding more bugs before your customers find them, speak to us understand how we might be able to help you.
Or call us on +61 3 7001 1430 or +44 20 3411 4974 if you're in the UK, or email solutions[at]

SaaS Brief