"We need it in real time" sounds like a clear integration requirement. It often hides several unanswered questions.
Does real time mean an immediate response while a user waits? Is a delay of two minutes acceptable? Would an hourly update change any operational decision? What should happen if the warehouse system is available but the finance system is undergoing maintenance?
Connecting systems more quickly can be valuable. It can also create tighter dependencies, higher support demands and more ways for a temporary fault to interrupt work. The right design starts with the business decision that needs fresh information, then chooses the least complex exchange that can meet it reliably.
Start with the decision and its deadline
Begin with the action, not the technology.
An ecommerce checkout may need a prompt answer about whether an order has been accepted. A warehouse team may need stock availability that is only a few minutes old. Finance might be perfectly well served by a complete, reconciled file after the trading day. Sending each of these flows through the same pattern would add complexity without improving every outcome.
For each integration, ask:
- Who acts on the data?
- How old can it be before their decision becomes worse?
- What is the cost of waiting, and what is the cost of acting on an incomplete update?
- Which system is authoritative when the two sides disagree?
- What should the user see while a dependency is unavailable?
Turn the answers into a measurable service level. "Orders available to the warehouse within five minutes for 99 per cent of successful checkouts" is useful. "Real time" is difficult to test and leaves the support team to interpret every delay.
Volume and shape matter too. Ten small status updates per hour create a different problem from a catalogue containing 200,000 price records. The deadline may be similar, but the safest transport will not be.
Choose the simplest exchange that meets it
Three broad patterns cover many operational integrations.
- Request and response — Useful when: A person or system needs an immediate answer before continuing Main trade-off: The caller depends on the other system being available and fast enough
- Asynchronous message or event — Useful when: Work can continue while the update is processed and bursts need absorbing Main trade-off: Users and operators need visibility of queued, delayed and failed work
- Scheduled batch — Useful when: A complete set can move at a known interval and some delay is acceptable Main trade-off: Corrections and late records need clear cut-off and reconciliation rules
Microsoft's messaging guidance describes how queues can separate a producer from a consumer in time. That is useful when a carrier API slows down or an ERP is briefly unavailable. The source system can record the work and the consumer can catch up, provided the queue and its age are monitored.
A batch is often the calmer choice for large reference data or a daily financial close. It can be validated as a complete set before it replaces the previous version. Faster transport would not correct a poor source record or an unclear business rule.
Design recovery before speed
Every network call can fail after the receiving system has completed the work but before the sending system receives the response. A simple retry might then create a duplicate order, shipment or payment.
Idempotency is the ability to repeat a request without repeating its business effect. The AWS Builders' Library guidance on idempotent APIs describes the use of a caller-provided request identifier so a service can recognise a repeated operation. The exact implementation varies, but the business identifier and expected outcome should be agreed before retries are added.
Recovery design should also cover:
- bounded retries with backoff and jitter, so a struggling dependency is not flooded;
- a quarantine or dead-letter route for records that cannot be processed;
- replay after a fault has been corrected;
- correlation identifiers that allow one transaction to be traced across systems;
- reconciliation that compares what each side believes happened.
Many messaging systems deliver a message at least once. Consumers should therefore cope with duplicates. A dead-letter queue also needs an owner and an alert. Moving a bad record out of the main flow is useful only when someone can diagnose and reprocess it.
Prove the service level in operation
A successful demonstration proves the happy path. An operational integration needs evidence from the awkward paths.
Test a slow dependency, a complete outage and a response that arrives after the caller has timed out. Test records in the wrong order and two copies of the same message. For batch work, test a partial file and a file that is delivered twice. These cases reveal whether the design recovers or quietly loses trust.
Monitoring should describe the business flow as well as the infrastructure. Useful signals include the age of the oldest order waiting to be processed, the number of records in quarantine and the gap between source and destination totals. CPU and memory graphs alone will not tell an operations manager which orders need attention.
Ownership needs to be explicit. Name the team that responds to alerts, the person who can correct a source record and the process for replaying work. Agree planned maintenance and escalation with the owners of the connected systems.
ORBN's system integration work starts with the workflow, source of truth and recovery requirements. We can help decide where an immediate API is justified, where messaging will reduce coupling and where a well-controlled batch is the more dependable option. For warehouse operations, our WMS integration service also covers cutover, reconciliation and the evidence needed before a flow becomes business critical.
Photo by Adrian Sulyok on Unsplash. Photographer's site. Photo ID: sczNLg6rrhQ.