Most vendor evaluations test the wrong thing. They test whether the company can present well for an hour, which is a skill some agencies have invested in far more heavily than delivery. The questions below are the ones that are awkward to answer if the underlying practice is not real.
Use them in a technical conversation rather than an RFP document. Written answers get polished by whoever writes proposals. Spoken answers, from the person who would actually run your team, are considerably more revealing.
About the people
- Can I meet the engineers who would be on this team, before signing? If the answer involves meeting them after contract signature, you are buying capacity from a pool, not a team. Ask why.
- Are they your employees or subcontractors? Subcontracting is not automatically wrong, but it changes who controls quality, who holds the knowledge, and who disappears when a cheaper contract appears elsewhere.
- What happens if someone leaves mid-engagement? Listen for a mechanism — overlap, documentation, a named replacement process — rather than reassurance.
- Where do your engineers come from? A vendor that only hires on the open market is bidding against everyone else in the same city every year. One that grows its own has a structural reason for people staying.
- Will anyone be swapped for a cheaper profile later? Ask it directly. The answer should be a flat no, and it should be in the contract.
About the work
- Show me a pull request you rejected. Not a portfolio piece — a review thread. It tells you whether review is a gate or a formality.
- What is your definition of done? If documentation, tests and handover notes are not in it, they are optional, which means they will be dropped the first time a deadline tightens.
- How long until the first production merge? Weeks of onboarding before anything ships is a warning about how the team actually operates.
- Who signs off machine-written code? Assistants are in every workflow now. The question is whether a named engineer still owns every merge, or whether the review bar quietly dropped for generated code.
- What would you refuse to build? A vendor with no answer will agree to anything, including the thing that damages your system.
About leaving
These are the questions buyers skip because asking about the exit at the start feels pessimistic. They are the questions that decide how expensive the relationship is to end, which is the thing that hurt last time.
- Whose cloud accounts and repositories are these? They should be yours from day one, not transferred at the end. A vendor holding infrastructure in its own accounts holds the exit.
- Have you ever run a handover drill? Not “would you provide documentation” — have you rehearsed the exit while the engagement was healthy.
- What is on the access list? Ask what a complete inventory of accounts and credentials would look like. A vendor that has one is organised. A vendor that would have to reconstruct it is telling you the truth about their operations.
- What does the last month look like? A good answer describes work. A bad answer describes a notice period.
About the commercials
- Is this a blended rate? A blended rate hides the mix. You are told the team average and discover the seniority distribution later.
- What is included in a person-month? Ask what happens to holidays, sick leave and public holidays, and whether they come out of your hours.
- What triggers an extra invoice? Overtime should be agreed before it happens, never discovered afterwards.
- What are the notice terms, in both directions? Asymmetric notice periods tell you how confident the vendor is that you will want to stay.
Reading the answers
Very few vendors fail every question. What separates them is the shape of the failures. Vagueness clustered around people usually means a resourcing pool. Vagueness clustered around exits usually means lock-in is part of the commercial model. Vagueness about review means the work will be as good as whoever happens to be assigned.
You will also learn something from who answers. If the person fielding technical questions is not technical, that is the relationship you are buying. Our own position on this is on how we work, and the fastest way to test any of it is a technical call rather than a sales call.