Analyst firms love to produce reports that compare technology solutions, businesses love to use those reports to support software selection and software suppliers in general love to hate them. Well, maybe that's too harsh, suppliers actually hate the process and only love the reports when it makes them look better than competitors. As a solution buyer, what do these comparison reports really tell you?

Finding the best fit solution requires more than the highest ranking solution based on some analyst firm criteria. It doesn't mean that the reports can't be useful in the evaluation, but by themselves they are not likely to lead you to the best solution for your business situation. That's a bold statement but fairly easy to support, in fact the simple answer is that context matters.

Context Matters

Software in the same category usually develop a fairly similar set of capabilities over time. How those capabilities are delivered though, can vary quite a bit application to application and supplier to supplier. The effectiveness of a feature and function also varies greatly from application to application. In addition to the variable nature of features and functions, software companies approach user experience (UX), user interface (UI), platform, integration and a plethora of other things differently. All of these factors impact the viability of a solution.

Variability of the solutions is only half the issue though. Each business has a unique culture, unique set of issues, unique competitive differentiators, unique processes and a unique set of issues. Which solution would work best in your business depends on the match between what the solution offers and what the specific business needs. In other words best fit solution is relative to the details of your business and business needs, not necessarily to the ranking in a report.

User Stories Establish Context

Using an agile solution selection methodology is the most effective way to get a match between your needs and the possible solutions. If you want to learn more about an agile selection methodology check out this post. The use of user stories is the best way to understand your needs and to facilitate finding the solution that most closely meets them. The user story format of "I (role) want to (capability) so that (benefit)" provides the most transparent view into all facets of the business need and into the perspective solution's ability to solve them. Suppliers responding to user stories find it much more difficult to manipulate the evaluation process, giving you a much better representation of how the solution will function in your context.

Timeliness

SaaS / Cloud software is usually updated much more frequently than traditional on premises solutions. What that means is that suppliers provide newer versions as frequently as daily (or even continuously) to 2-3 times a year. This means that a solution may change significantly in a short period of time. Large analyst firm reports are usually on a annual to 18+ month refresh cycle. In 18 to 24 months the current solution most likely has changed so much that the relevancy and accuracy of the report is questionable. The only way around this issue from a report perspective is to find reports that are generated much more frequently so that the data is valid and represents the current version of the solution. One example of these alternative reports that are refreshed quarterly can be found at G2 (I'm the former G2 chief research officer and still engaged as a strategic advisor). The other benefit of reports from peer review sites is that they are based on a large sample of users and how they used the solution. Because of the broader data set, the reports are often segmented by company size and can address very specific views of the data for things like usability, results, ease of implementation, relationship with the provider, feature - function comparisons and even regional segments.

Considerations

At this point, you’re probably thinking that this is complicated, and you’re not wrong. To find the “best” solution you need to look at several things. In the selection process you need to look at the supplier, the solution, your business and your business needs (user stories).

Vendor and Solution:

• Relationship and support

• Financial stability

• Platform technology and architecture

• Deployment models

• Usability

• Ease of implementation and methodology

• Regional presence and localization of solution

• Price to value

• Long term solution strategy

• Roadmap

Business and User Stories:

• User roles - persona

• Industry vertical

• Company size and growth plans

• Geographies

• Functional fit

• IT Maturity

• Cost to benefit

How to Use the "Reports"

The analyst reports can be a useful part of the solution evaluation. Each report and evaluation criteria is designed to inform in a specific way so it’s important to understand the underlying methodology, data sources and visualizations. Objective criteria and data sources are the highest priority inputs to your decision process. This is particularly important for representations of user sentiment where the data sample needs to be fairly large to accurately represent the “voice of the user”. Smaller sample sizes as well as samples provided by suppliers can lead to inaccurate representations. Subjective analysis, in other words “opinions” of experts, might be useful, but you need to approach them with some level of skepticism. To determine how to use subjective analysis it’s important to understand the source of the opinions, level of expertise, experience and understanding of your industry.

Putting it Together

Selecting the “right” solution for your business and business challenges involves the use of multiple data sources and applying that to the specific user stories and needs of the company. The best fit for one business may or may not be the best fit for another. Evaluating the prospective solutions in the context of your specific situation is essential to driving the highest return on investment (ROI) and the best business outcomes. Getting outside help to guide the evaluation process and help find the best fit can accelerate the selection and streamline the process.