How we validate and back up the data behind a live automation
The validation and recovery layer around an automated purchase-order system. Every morning, before the working day starts, it takes a verified backup of the two files everything depends on and keeps a rolling thirty days of restore points, then runs an independent audit of the automation's own output to confirm the numbers still add up. Clean mornings pass silently; anything that does not reconcile is reported, graded by severity, the same morning. No AI: the whole layer is deterministic arithmetic and reconciliation.
- 6 min
- 32 workflow steps
- 9 processing stages
- Live system
Client details withheld for confidentiality. System details and recorded figures are real.
How do you validate the output of an automated system and keep it recoverable?
Every morning, before the working day starts, the system takes a verified backup of the two files everything depends on (the master order ledger and the work-order number counter) and keeps a rolling thirty days of restore points. Then a separate, independent check reads the automation's own output and makes sure it still adds up: that the numbering has no duplicates or gaps, that nothing is stuck half-processed, and that the ledger's totals actually equal the sum of its lines. If everything reconciles, the morning passes silently; if anything is off, a single itemised report, graded by severity, goes to the monitored inbox.
When a system produces work automatically, the danger is not usually a dramatic crash. It is the quiet error: a total that silently stops adding up the newest rows, a number reused by accident, a file that never finished processing, a figure that drifts a little out of line and is not noticed until it reaches a customer or an invoice.
And underneath all of it sits the fear every business has about an automated file: what if it gets corrupted, or something overwrites it, and yesterday's good version is gone. The job here was to make the system's output continuously checkable and recoverable from common file-level failures, so a small mistake is caught the same morning and a recent clean version is available to restore.
What the finished system does
The system now takes a verified backup of the two critical files every morning, keeps thirty days of restore points, and then runs an independent audit of the automation's output. It reconciles the work-order numbering against the actual files, checks nothing is stuck or backing up, and re-computes the ledger totals from the line items. Clean mornings pass silently; anything that does not reconcile is reported, graded by severity, to the monitored inbox the same morning.
We design managed automation systems like this around the tools a team already uses.
How the system works, stage by stage
A verified copy is taken first, every morning
Before the working day, the two files everything depends on (the master order ledger and the work-order number counter) are copied to a dated backup.
The backup is verified, not assumed
The system checks that both backup files actually exist afterwards, because a copy request being accepted by the storage service is not the same as a file being saved. If a copy did not complete, it alerts.
Thirty days of restore points are kept
Older dated backups are pruned automatically, leaving a rolling thirty-day recovery window.
Then an independent audit runs
A separate check reads the automation's own output and looks for anything that does not add up.
The numbering is reconciled
The work-order counter is cross-checked against the actual generated files: duplicates, numbers above the counter, and gaps in the sequence are flagged.
The queue is checked for stalls
Files stuck mid-processing beyond an hour, or a backlog that is not draining, are surfaced.
The ledger's arithmetic is re-computed
The order-value total is worked out again from the individual lines and compared to the total in the sheet; a mismatch, or a total that has stopped including the newest rows, is flagged as an error.
Duplicates and stray rows are caught
Repeated work-order numbers, re-processed purchase orders (same order, part and date), and blank rows inside the data are all reported.
Only exceptions are raised
If everything reconciles, the morning is silent. If anything is off, one itemised report, graded red, orange or green, goes to the monitored inbox.
Under the hood: the 9 internal stages
The public stages above explain the system by business purpose. Internally, the two connected workflows divide the work into nine technical stages that group the 32 workflow steps. The Daily Backup (stages 01 to 05) runs at 06:00 and takes the verified restore point; the Data Integrity Watchdog (stages 06 to 09) runs at 06:30 and audits the automation's output. The backup deliberately runs first, so the morning's verified restore point is normally in place before the checks begin. The two never call each other; they run on schedule against the same files.
- 01Wake early and load settingsInternal stage: backup.schedule_config
A scheduled early-morning trigger starts the backup, then the configuration is loaded.
- 02Build the backup planInternal stage: backup.plan
Works out the dated file names and the target folder for today's backup.
- 03Copy the two critical filesInternal stage: backup.copy
Ensures the backups folder exists, then copies the master ledger and the work-order counter into it as dated files, idempotent so a re-run on the same day does not duplicate them.
- 04Verify each backup, or alertInternal stage: backup.verify
Waits for the asynchronous copies to finish, then independently fetches each backup file back to confirm it is really there, retrying if needed; on any copy or verify failure it alerts the monitored inbox.
- 05Prune beyond thirty daysInternal stage: backup.prune
Lists the backups folder, builds the list of files older than the retention window, and deletes them, leaving a rolling thirty-day recovery window.
- 06Wake just after the backup and load settingsInternal stage: watchdog.schedule_config
A scheduled trigger fires just after the backup completes, then loads its configuration.
- 07Read the three recordsInternal stage: watchdog.read_records
Reads the work-order counter, lists the generated, review, error, processing and ready folders, and reads the master ledger's used range.
- 08Run the checks in one passInternal stage: watchdog.analyse
Analyses everything in one deterministic pass: numbering continuity, stuck files, backlog, and the ledger's arithmetic and duplicates, across ten distinct checks.
- 09Report exceptions onlyInternal stage: watchdog.report
If anomalies are found, sends one graded, itemised report to the monitored inbox; if everything reconciles, finishes silently.
Why this is more than a scheduled file copy
Copying a file on a schedule is easy. Knowing the copy actually saved, and knowing the data inside it is still correct, is the hard part. Most of the work here is refusing to trust the obvious: that a backup request succeeded, and that the numbers a system wrote for itself are right.
The safeguards that make it dependable
Verified backups
After copying, the system fetches each backup back and confirms it exists, with retries, before recording it as good.
Both critical records backed up
The order ledger and work-order counter are backed up together, so both critical records have matching daily restore points.
Rolling thirty-day retention
Old backups are pruned automatically.
Backup scheduled before the audit
The backup starts before the integrity audit, giving the system time to create and verify the morning's restore point before the checks begin. Any backup failure raises an alert.
Independent audit workflow
The checker runs separately from the workflow that produces the data and re-derives the results using its own rules.
Cross-record reconciliation
The counter, the files, and the ledger are checked against each other, not in isolation.
Re-computed arithmetic
The ledger total is worked out again from the lines, not trusted as written.
Severity grading, exceptions only
Clean mornings are silent; only problems are reported, graded by severity.
Deterministic checks
No AI in the safety layer; the results are repeatable.
The obvious way to back a file up is to send a copy request, see it succeed, and move on. The catch is that a cloud storage service reports success the moment it accepts the request, which is before the copy has actually finished, so a 'successful' backup can quietly be a file that never appeared. So the system does not trust the request: after every copy it fetches the dated backup file back and confirms it is really there, retrying if needed, and only then records the backup as good. The same refusal to trust runs through the audit: it does not take the automation's own totals at face value, but re-computes them from the individual lines and rebuilds the numbering from the actual files, because checking a system's work with the same assumptions that produced it would just repeat any mistake.
Controlled validation before go-live is where the arithmetic and reconciliation rules were tuned against real historical order data, so that on day one a total that had drifted, or a number used twice, would be caught rather than trusted.
Two honest bounds are worth stating: the backups sit in the same cloud storage as the originals, so this protects against corruption, a bad automated edit, or an accidental deletion, recoverable within the thirty-day window, but not against the loss of the whole storage account, which would need an off-platform copy. And the audit runs on the same platform as the automation, so it reports anomalies in the data, not a full outage of that platform.
What is now running
The checks were tuned against real historical order data before go-live, so a drifted total or a reused number would be caught rather than trusted.
The system is now live and managed by Noventra, validating and backing up the data produced by a live automated purchase-order workflow.
Questions businesses ask
What does this actually protect against?
It provides restore points for corruption, bad automated edits, and accidental deletion, while detecting numbering, queue, duplicate, and ledger inconsistencies before they pass unnoticed. It keeps thirty days of verified restore points and re-checks the numbers every morning.
Is this the same as checking whether the automation is running?
No. That is a separate system. This one checks that the output is correct and recoverable, not just that the machine is up.
Does it use AI?
No. The checks are deterministic arithmetic and reconciliation rules, so they give the same answer every time.
What happens if a backup does not save?
The system finds out, because it verifies each backup file exists after copying, and alerts the same morning.
How far back can you recover?
Thirty days of daily restore points, for the two files the automation depends on most.
Does a normal day create noise?
No. If everything reconciles, nothing is sent; you only hear about exceptions.
Want a system like this in your business?
Tell us what eats your week. We will tell you what can be automated, what should stay human, and what a first version would involve.
No build cost · Clear terms upfront · Managed by Noventra