AI can make a software demonstration look finished before the hard engineering has started. The screens work. The happy path is convincing. The code arrived quickly.
The gaps appear when the software meets real orders, messy permissions, supplier timeouts, corrections, and staff handover. A prototype may contain plenty of code while offering little evidence that the business can rely on it.
That is the risk behind AI slop: fast output without the engineering needed to run, repair, change, and own the software in production.
Your business carries the operational consequences. Ask the supplier to show how important requirements are tested, how a change reaches production, how the team sees what the live system is doing, and what happens during an outage.
How do you know the software does what you expect?
A demonstration proves that one selected path worked once. Production software must keep the operation under control when data is late, a service is unavailable, two people act at once, or somebody needs to correct a mistake.
Write down what the business expects the system to do. A new order workflow may need to:
- Accept orders from several channels.
- Protect customer-specific prices.
- Prevent duplicate submissions.
- Keep warehouse and finance records in step.
- Continue when one supplier system is unavailable.
- Show staff which orders need attention.
Ask the supplier to prove each promise. Send the same order twice and show that only one is created. Disconnect a supplier service and show staff where the order waits. Change a customer price and show which system wins. The test should match the failure the business would actually have to manage.
AI can help engineers explore options and implement the chosen design. The buyer should still receive a clear account of the assumptions and limits. A fast build has little value when the team cannot explain which record is authoritative or what happens after a partial failure.
The 2025 DORA research describes AI as an amplifier of the system around software delivery. Strong feedback, testing, user focus, and delivery controls help teams use it well. Weak practices produce more change without better control.
Look for operational decisions in the design
Professional engineering turns messy operational detail into deliberate software behaviour.
Ask the supplier to walk through a difficult case before discussing the happy path. For example, two people edit the same order, a customer has two account codes, or an external service accepts a payment but times out before sending confirmation.
The design should make these decisions visible:
- Which system owns each fact.
- Which events can arrive twice or out of order.
- Which changes need an audit trail.
- Which actions require approval.
- Where work waits after a failure.
- How staff repair an incomplete transaction.
These are business decisions expressed in software. An attractive interface cannot compensate for the wrong source of truth or a missing recovery path.
Look for restraint as well. A scheduled job may suit a daily reconciliation better than a complex real-time integration. A fixed rule may be safer than a model. A visible manual exception may cost less than automating a rare case.
The supplier should be able to explain why the design fits the operation and which trade-offs the buyer is accepting.
Inspect the route from change to production
Software needs continual work after launch. Users find exceptions, suppliers alter interfaces, security fixes become available, and the business changes policy.
Ask to see how a change moves from an idea to the live service. Look for:
- A recorded requirement or defect in a ticket tracking system.
- The source-code change in a controlled repository.
- Review by someone other than the author where the risk warrants it.
- Tests that protect important behaviour.
- A secure build and deployment process.
- A record of what reached each environment.
- A rollback or repair plan.
- Monitoring after deployment.
The NCSC secure development principles cover maintainable code, protected repositories, secure build pipelines, continuing testing, and planning for flaws. Those controls apply whether an engineer typed the first draft or generated it with AI.
AI-generated tests are useful only when somebody checks that they test the right behaviour. A passing test can preserve the wrong assumption. Ask who chooses the cases, reviews failures, and decides that a release is safe.
Monitoring should also cover the actual operation. A healthy server does not prove that orders are reaching the warehouse. Useful alerts cover missing business events, growing queues, failed synchronisation, and records stuck between systems.
Can you run it without the supplier?
Ownership is real when your business can operate the software, recover from a failure, and hand it to another capable team. A copy of the source code is only one part of that.
Before signing, make the contract and delivery plan state who controls:
- The source repository and deployment settings.
- Cloud accounts, domains, data stores, and backups.
- Model providers, third-party services, and licence renewals.
- Secrets, administrator access, and support accounts.
- Operating documentation and decision records.
- Data export and supplier exit.
Ask the supplier to rehearse an incident before launch. Pick a realistic failure and follow it through detection, containment, customer impact, data repair, and communication. This reveals whether recovery exists beyond a sentence in the proposal.
Handover should also be tested. Another capable engineer should be able to deploy the software, investigate an alert, and make a small change using the documentation and access the client receives.
The NCSC Software Security Code of Practice guidance treats secure deployment, maintenance, and customer communication as supplier responsibilities. These are part of the product the buyer is purchasing.
Compare what each supplier can prove
Two proposals can describe the same features while pricing very different products. One may include monitoring, recovery, security testing, documentation, and handover. Another may leave that work until after launch. Ask every supplier to prove the same things against the same difficult workflow.
Ask each supplier the same questions:
- Which operational outcome will the first release improve?
- Which difficult cases shaped the design?
- How will important behaviour be tested?
- What evidence will each deployment produce?
- How will the team detect and repair partial failure?
- Which accounts, code, data, and documents will the client control?
- Who remains accountable when AI contributes code?
A credible supplier can answer in plain language and show examples from the proposed delivery process. They should also identify areas that need discovery instead of hiding uncertainty inside a fixed feature list.
If you are comparing an AI-assisted software proposal, ask ORBN. Bring us the highest-risk workflow and the proposal you are assessing. We can trace what must happen, what can fail, and the evidence a production-ready system should leave. That gives you a practical basis for deciding whether the proposal is a sound foundation or fast output dressed as finished software.