How to evaluate a software development agency: 8 questions that filter out the bad ones

Most founders who have hired a software agency once will tell you, off the record, that it took two or three attempts before they found a team that actually shipped. The first agency disappeared after invoice four. The second built something that worked until anyone tried to extend it. The third, finally, was the one.

This guide is the shortcut. Eight questions you can ask on the first call that filter out the agencies that produce the first two stories and leave you with candidates who might be the third.

The questions look simple. The answers tell you almost everything.

1. “Who will actually write the code on this project?”

The right answer: a small number of named engineers, ideally introduced to you on the first call or the second.

The wrong answer: vague references to “our team” or “we have a pool of engineers we draw from.” This usually means the work will be done by people you’ve never spoken to, often in a different time zone from the salesperson you’ve been talking to, often more junior than the agency’s pitch suggested.

If the answer is “we’ll figure that out after we sign,” the agency is selling you availability, not a team. The match between the engineers who scope a project and the engineers who build it predicts quality more reliably than almost any other signal.

2. “Can I see something complex you’ve shipped and talk to the client who paid for it?”

The right answer: a case study with detail, paired with a recent client willing to take a 15-minute reference call.

The wrong answer: a portfolio of polished home pages that all look great and reveal nothing about engineering quality. Or — worse — testimonials from “the CEO of Acme Corp” that don’t link to a real person.

Senior agencies are happy to introduce you to recent clients because their recent clients say good things. Agencies that won’t are protecting something. The thing they’re protecting is sometimes the truth.

3. “What does your maintenance contract look like, and do you actually keep one with most clients?”

The right answer: a defined retainer model, often disclosed up front, with a meaningful percentage of past clients still on it.

The wrong answer: “Maintenance? We can figure that out after launch.” Or any version that suggests the agency builds and ships and then disappears.

Maintenance is where bad agencies lose money, which is why they avoid it. An agency that runs production software for its clients has learned things that the build-and-flee model never teaches them. Those lessons end up in your codebase.

4. “Walk me through a project where something went wrong. What did you do?”

The right answer: a specific story with details, an honest admission of what they got wrong, and what they changed because of it.

The wrong answer: “Honestly, things mostly go well.” This is the single biggest red flag in agency evaluation. Every team that has shipped enough software has had projects go sideways. The ones who deny it are either lying or haven’t shipped enough.

Listen for whether they describe the problem as the client’s fault. Some of that is fair (clients do cause problems). All of it is a tell that you’ll be blamed if your project goes sideways too.

5. “How do you scope a project before you’ve written code?”

The right answer: a structured discovery phase with a written deliverable, ideally including some kind of architecture sketch or data model before development starts.

The wrong answer: a quote produced after a thirty-minute call. Or worse, a quote produced before any call. Software estimates that don’t involve discovery are guesses dressed up to look like commitments.

The agencies that scope properly will catch the things that would have blown the budget if they hadn’t. The agencies that skip scoping will discover those things three months in and ask for a change order.

6. “What’s your team’s hourly or weekly rate, and what’s it based on?”

The right answer: a clear number, in your currency, with a brief explanation of what’s included.

The wrong answer: a runaround. “It depends” — on what? “We package projects” — that’s a different question. “Let’s discuss after we understand your needs better” — translated: we adjust rates based on how much money we think you have.

You are entitled to know what the meter reads. Agencies that hide it are typically variable-pricing you based on perceived budget. The good ones price transparently and let value, not opacity, justify their rate.

7. “Who owns the code, the data, and the deployments when the project is done?”

The right answer: you do. The code is in your repository. The infrastructure is in your cloud account. The credentials are documented and handed over.

The wrong answer: anything involving the agency continuing to host, control, or gatekeep your production system. Some of this is necessary during development. None of it is necessary at handover.

Vendor lock-in disguised as “we manage everything for you” is one of the most common patterns in the small-agency market. Read the contract carefully. The right answer is in writing, not just in conversation.

8. “What would you tell us not to build?”

The right answer: a thoughtful response that identifies one or two parts of your scope that probably aren’t worth the cost, with reasoning.

The wrong answer: enthusiastic agreement with everything you described. Agencies that don’t push back on scope are optimizing for closing the deal, not for shipping a useful product.

The best engineering partners save you from your own scope. They’ve seen which features get built and never used. They tell you so before the budget is committed, not after.

Bonus: the timing pattern

This isn’t a question to ask, but a thing to watch.

Bad agencies are extremely responsive during the sales process. Replies in minutes, pre-built proposals, the works. Then communication slows the moment you sign. Good agencies treat the sales process and the engagement with similar responsiveness — sometimes a little slower up front because they take their time on the scoping, and the same pace after.

If your potential partner is dramatically more attentive before the contract than they’ll be after, you’ll find out in month two and it won’t be pleasant.

Filters that don’t matter as much as people think

A few things people obsess over that turn out to be less important than the eight questions above.

Where the team is located. Time zone matters for collaboration patterns, but country-of-origin is a weaker predictor of quality than team composition and process.

The tech stack on the marketing page. The right stack for your project is the one your engineers know. A team that’s deep in Ruby will build a better Rails app than a team that’s “open to whatever” but actually weak in everything.

The size of the agency. Larger isn’t safer. The team building your project is, at most, a handful of people. Whether they sit inside a 5-person agency or a 50-person one matters less than whether they’re the right handful.

If you’re in this conversation now

We’re a small senior team that takes on a small number of projects at a time. We answer the questions above honestly on the first call, and we’d rather not work with you than work with you badly. If you want to scope a project — or get a sanity check on a proposal from someone else — see how we work or email [email protected].