Every B2B platform you regret signing looked great in the demo. The demo is the vendor's most polished surface; it's designed to skip the questions where the answers would slow you down. The post-mortem on every bad procurement decision lands in the same places: a missing integration that nobody asked about, a security review that surfaced six months in, a renewal price that doubled because exit costs were prohibitive.

The fix isn't to mistrust vendors. It's to run a consistent evaluation process that surfaces the questions the demo doesn't answer. This is the checklist we use with operators before signing — broken into four categories, each with the questions that actually matter.

Category 1 — Surface (What Marketing Tells You)

The first pass is the cheap one. Before scheduling any demo, you should be able to answer:

  • What does the product actually do? Stated plainly, in one sentence, without buzzwords. If you can't write this from the marketing site, the product is poorly positioned or you don't yet understand it.
  • Who are the case study companies? Names, sizes, sectors. If the only case studies are anonymous "Fortune 500 customers," the company is either too early or not selling into your segment.
  • Pricing transparency. Public pricing is a positive signal. "Contact sales for pricing" means you're in a negotiation, not a comparison.
  • Free trial or sandbox? Yes is a positive signal. No is a yellow flag.

If you can't get past these four without a sales call, the company has friction-loaded the early evaluation and you should be wary.

Category 2 — Depth (What the Demo Doesn't Show)

This is the layer where most procurement decisions get made or broken. Schedule a 60-minute technical session (not a sales demo) and walk through:

Integrations

  • Does it integrate natively with your existing CRM, your data warehouse, your auth provider?
  • Is the integration bi-directional or one-way?
  • What happens when the integration breaks? Webhook retries, dead-letter queue, manual reconciliation?
  • Is the integration on the roadmap or shipped today? "Coming soon" means six months from now at best.

API and Data Access

  • Is there a real API? Read and write?
  • Rate limits — can your usage fit inside them?
  • Can you export your data? In what format? How often? At what cost?
  • Is there a webhook system for events that matter?

Security and Compliance

  • SOC 2 Type 2 report — available without an NDA? Recent (within 12 months)?
  • Where is data stored? Which region? Can data residency be configured?
  • SSO support — on the standard plan, or paywalled to enterprise?
  • Role-based access control — granular enough for your team structure?
  • Data deletion — what happens when you offboard? How fast? Verifiable?

If the vendor can't answer any of these confidently, the product is too immature to bet on. SSO paywalled to enterprise is the single most reliable signal of a vendor that will nickel-and-dime you on the renewal.

Category 3 — Commercial Fit

This is where deals go sideways months after signing. The questions to ask before, not after:

  • What's the actual price for your usage at year two? Vendor pricing pages assume year-one volume. Model out year two with realistic growth and ask the vendor to commit to that price in writing.
  • What's the renewal cap? "Annual renewals subject to standard increases" can mean 5% or 40%. Negotiate a cap up front (8–12% is normal).
  • What's the minimum contract length? Annual is fine; "you must commit to three years for this price" is a vendor manufacturing lock-in.
  • What's the cancellation policy? 30 days written notice before renewal is standard; auto-renewal with a 90-day cancellation window is a trap.
  • Is there a usage tier or seat tier that you'll cross in year one? If yes, negotiate the next-tier price now.

Category 4 — Exit Cost (The Most Overlooked)

Every platform you adopt is also a platform you might leave. The exit cost is what determines whether you have real leverage at renewal time:

  • Data portability. Can you export everything — not just the headline objects but the metadata, the audit logs, the custom fields — in a standard format?
  • Integration migration. If you've built three integrations against this vendor's API, how hard is it to repoint them at a competitor?
  • Team retraining. If the team has learned a vendor-specific workflow, what's the cost of moving them to a new tool?
  • Embedded data. Is your vendor storing primary records that you can't easily reconstruct elsewhere? (CRMs and analytics tools are the classic offenders.)

The platforms with the highest exit cost are the ones to scrutinize hardest before signing — that's where vendor leverage compounds most aggressively at renewal. We've written about moving off tools we'd already adopted; the cost varies massively by category, and you can predict it from the exit-cost evaluation done at signing time.

The Two Reference-Check Questions Worth Asking

Reference calls are the most under-used part of B2B evaluation. The two questions that actually surface useful information:

  1. "What did the vendor sell you that turned out to be different in practice?" Every reference has at least one answer to this. The shape of the answer tells you where the vendor's gap between marketing and reality lives.
  2. "If you could redo the procurement, what would you ask them that you didn't?" This surfaces the questions the reference wishes they'd known to ask — usually the most valuable input you'll get in the whole evaluation.

Don't accept references the vendor selected without also asking your network for an independent reference. The vendor's chosen references are pre-screened; an independent reference often has the more honest story.

What to Avoid

Letting the Demo Set the Agenda

Vendor demos walk you through the product the vendor wants you to see. Bring your own scenarios — the three workflows you actually need to run on this tool — and insist on walking through those. If the vendor can't handle your workflows in the demo, they probably can't handle them in production.

Skipping the Free Trial

If a free trial or sandbox exists, use it. Hands-on time with the product surfaces friction that no demo will. If the vendor refuses to give you a trial because "it requires implementation," that's a signal about the product's complexity, not about the vendor's flexibility.

Buying for Future Needs

"We'll need feature X in eighteen months" is the most expensive procurement justification. Buy for what you need now. The probability that the vendor still has feature X on the roadmap in eighteen months, that you still need feature X, and that the vendor still meets your other needs is much lower than vendors will let you believe.

Trusting the Roadmap

Vendor roadmaps are aspirations, not commitments. If feature X is on the roadmap and the product is unusable without feature X, the product is unusable. Buy what's shipped, not what's promised.

The Decision Framework

Once the four categories are evaluated, the decision should be straightforward:

  • Strong yes: Surface clear, depth meets all critical requirements, commercial fit is reasonable, exit cost is manageable. Sign.
  • Conditional yes: One or two depth gaps that the vendor commits to addressing in writing within a defined timeframe. Sign with an out clause if commitments aren't met.
  • No: Any combination of high exit cost + aggressive commercial terms + depth gaps. Walk.

The bias should be toward "no." A platform you don't adopt costs you nothing; a platform you adopt and regret costs you a year of internal change-management.

Frequently Asked Questions

How long should a vendor evaluation take?

For a mid-stakes platform (CRM, marketing automation, AI tooling): two to four weeks of evaluation, including a free trial and at least two reference calls. For mission-critical platforms (billing, data warehouse): six to eight weeks. Faster than that and you're skipping the depth layer.

Should we always do an RFP?

No. RFPs make sense for high-stakes, multi-stakeholder procurements where formal comparison is required. For most operator-scale decisions, a structured evaluation against this checklist is faster and produces better outcomes than a formal RFP.

What if the vendor won't answer security questions without an NDA?

That's normal for SOC 2 reports. NDAs are fine; what's not fine is a vendor that delays the SOC 2 review until after contract signing. The security evaluation is part of the procurement decision, not a post-signing formality.

How do we evaluate AI-specific vendors?

Add three categories on top of the standard checklist: model transparency (which provider, can it be changed?), evaluation infrastructure (how do they test their AI?), and data-use terms (is your data used for training?). The fundamentals of vendor evaluation don't change for AI; the surface area does.

What's the most underrated category?

Exit cost. The category that gets the least attention at signing time consistently determines whether you have leverage at renewal — the moment when vendor pricing power is highest.

If you're about to sign a platform that you'll be living with for years, the cost of running this evaluation is a few hours; the cost of skipping it shows up at the renewal. Talk to us if you want a pressure-test on a specific vendor decision.