ABSTRACT: This article outlines Eckerson Group’s methodology for evaluating and selecting a data & analytics product, whether that’s a data platform, a data catalog, analytics platform or any other tool.

Eckerson Group has been helping organizations select a variety of data & analytics articles for more than a decade. This article outlines our methodology for evaluating and selecting a data & analytics product, whether that’s a data platform, a data catalog, analytics platform or any other tool. In fact, this methodology works for just about any software or hardware product. 

This article doesn’t address the “soft” decisions involved in evaluating and selecting products. These include whether you should conduct a formal evaluation process or not, who you should include on an evaluation team, how many products you should evaluate, what pitfalls you can expect, and whether you should recruit a project manager or hire an outside firm to guide the process, among other things. For insights on these topics, please watch our webinar or read a summary of the article at ???

The Eckerson Group methodology is divided into five segments and runs for about 12 weeks assuming the evaluation team reviews three products and tests one in a proof of concept.

1 - Define Scope - 1 week

1. Define the problem that the product is supposed to address

2. Identify type of technology you want to evaluate

3. Identify target users (i.e., personas) who will use the product

4. Set project duration based on internal constraints

5. Define max # of products to evaluate

6. Define the methodology

7. Identify project manager

8. Identify evaluation team

2. Gather Requirements – 3-4 weeks

1. Conduct research on the technology and products

a) Compile superset of criteria and products 

2. Gather user requirements 

a) Identify high-level capabilities needed

b) Identify key use cases

c) Prioritize by “must have” and “nice to have” 

3. Gather company requirements 

a) Architecture, price, preferred vendors, support needs

4. Filter technology criteria by requirements

a) Create a prioritized list of criteria 

3. Conduct Vendor Demos – 5 to 6 weeks

1. Create a short list of products (3-4 is ideal)

a) Eliminate products missing a “must have”

2. Create a request for information (RFI) for vendors 

a) Send RFI and schedule a demo meeting. 

b) Optionally replace RFI with a call to each vendor 

c) Optionally, schedule vendor meetings with a technical team to determine whether the product meets technical requirements. If it does, send the RFI or schedule a second meeting with business members of the evaluation team.

3. Create a scoring form for individuals on the eval team 

a) Explain criteria & process to vendors & eval team in separate meetings.

4. Schedule and conduct 3-hour demo sessions.

a) Book multiple 3-hour blocks for the eval team a month in advance to minimizing scheduling difficulties with vendors. 

b) Leave 30 minutes at the end of each demo session so the eval team can debrief and complete the individual scoring form. 

3. Compile and present results.

a) Eval team selects one product for a POC

4. Conduct Proof of Concept – 4-6 weeks

1. Identify proof of concept testers 

2. Write the proof of concept use case 

3. Limit to two use cases max

4. Create the proof-of-concept evaluation and scoring form

5. Coordinate the proof of concept with the vendor(s). 

6. Conduct the proof of concept and compile results

7. Convene eval team to discuss the proof of concept results and vote “Yes” or “No” on the product.

5. Purchase the Product 

1. Alert the purchasing and legal departments early in the process

2. Communicate purchasing and legal requirements to the vendor early on

3. Leverage the purchasing team to help negotiate a good price