AI Without the Hype · Part 9 of 10
The wrong starting question is easy to ask: “Which AI tool is best?”
Best for what? Drafting customer emails, finding approved procedures, working through a spreadsheet, and taking actions across business systems are different jobs. A tool can be excellent at one and unsuitable for another.
Before comparing subscriptions, write down the task, the people using it, the information involved, and the result you need. That brief will do more for your buying decision than a feature-count comparison.
Decide which requirements can rule a tool out
Some requirements are preferences. Others determine whether you can use the product at all.
If a task involves information that must stay within an approved environment, the tool has to meet that requirement. If staff need a shared business account with access removed when someone leaves, verify how that works on the actual plan. If the output has to be exported, inspect an export before committing.
Treat these as checks to pass before scoring writing quality or speed. A strong demonstration does not resolve an unsuitable data arrangement or a missing business requirement.
Read the current documentation and terms for the exact service. Capabilities, prices, and controls change. A comparison written last year may be discussing a different product from the one you would buy today.
Test the same job in each candidate
Choose a small set of representative examples. Use fictional or approved data and the same instructions across tools, allowing reasonable adjustments for how each works.
Suppose you want draft customer updates from job notes. Include a normal completed job, missing details, unresolved work, conflicting dates, and a note with jargon. Write down the expected facts and the commitments the output must avoid.
Have the person who normally handles the work assess the results. Count the time spent preparing inputs, correcting drafts, and approving them. Include retries and abandoned attempts.
You are comparing the effort needed to get acceptable work, not choosing the most impressive first response.
Use a short decision record
For each tool, record whether it passes your required data and account checks, handles the representative tasks, and fits the existing workflow. Then note material errors, review effort, total expected cost, and who would maintain it.
Keep those observations specific. “Looks professional” is less useful than “preserved all five job facts but twice invented a completion date.”
You do not need a complicated scoring model. If a simple table helps you compare candidates consistently, use one. Do not let a high overall score hide a failure on a requirement you cannot compromise.
NIST’s work on AI measurement and evaluation reinforces the importance of evaluating systems. The task set here is a local buying exercise, not a formal benchmark or a promise of future accuracy.
Ask what the connection actually does
“Integrates with your customer relationship management (CRM) system” can describe several arrangements. Does the tool read records, create them, change them, or just let someone copy text into a field? Does that functionality require another paid service or software tier?
Ask the provider to demonstrate the step you need with an example. Ask what happens when access expires, the source is missing, or the connected system is unavailable.
If the tool can act on behalf of the business, ask how approvals, permissions, and activity records work. A drafting tool and a system authorized to send customer messages need different evaluation.
Check whether you already have enough
Your existing software may include a feature that meets the task. It deserves the same test as a new purchase.
A new specialist tool may justify itself through better results or less effort. It may also add another account, subscription, and place to maintain information. Compare that added work honestly.
If your process is unclear before the tool enters it, revisit the five questions to ask before buying business software. Tool selection cannot settle who owns a task or what counts as complete.
Make it possible to change your mind
Find out how cancellation works, which data can be exported, what happens to shared templates, and how staff can continue during an outage or transition. Ask what support is included and who handles problems after setup.
Use a trial or limited purchase where it makes sense. Keep the scope small enough that you can stop without reorganizing the whole business around an unproven result.
You might end up choosing an existing feature, a new subscription, a configured workflow, or no AI tool for this task. A good evaluation leaves all four outcomes available until the evidence points somewhere.
Source
NIST: AI measurement and evaluation. The comparison procedure is a practical buying method, not an endorsement of a particular provider.