From Unicycles to Ferraris, Part 3: How to Tell If Your Automation Is Actually Working (Or Just Looks Cool)

There’s a quiet moment that happens about three weeks after you build your first automation.

The novelty has worn off. The Slack channel still goes “Cha-ching!” but nobody cheers anymore. And one morning, somewhere between your second coffee and your tenth Teams notification, a small voice in your head asks: Is this thing actually helping - or did I just give myself a more colourful way to do the same job?

This article is for that voice.

Because here’s an uncomfortable truth about automation: most people never actually check whether it worked. They build the bot. They briefly feel like a wizard. They tell three colleagues at lunch. And six months later, when their boss asks for the ROI, they panic and invent a number that sounds vaguely defensible.

Don’t be that person.

Step Zero: Measuring Without Drowning

Before we get into the how of measurement, let’s deal with the part that scares everyone off.

Measurement is not a finance exam. You don’t need a 47-tab spreadsheet, a data scientist on retainer, or a dashboard that looks like a NASA control room. You need three numbers and one honest conversation. That’s the whole article. You could stop reading here. (But please don’t - we worked hard on the jokes.)

Without measurement, “we automated our reports!” is the corporate equivalent of buying a fancy gym membership and bragging about the keychain. Looks great. Means nothing. So let’s give you the actual numbers - and the actual question - that prove the thing works.

Step 1: The “Life Before the Bot” Baseline

Here’s something nobody warns you about: the moment you flip the switch on your automation, the “before” disappears forever.

You’ll never again remember exactly how long the old way took. You’ll never accurately recall how many errors used to creep in. You’ll never reconstruct how loudly your team groaned every Monday morning at 9:14 AM. Memory is generous like that - it quietly smooths the edges off the past until you can’t remember why you automated the thing in the first place.

Capture the painful truth before you flip the switch.

And we mean actual numbers. “It felt like forever” is not a baseline. Neither is “ages.” Neither is “longer than my Monday-morning will to live.” Those are feelings, not measurements. We love feelings - but they don’t survive in a boardroom.

Time the task. Count the errors. Ask the team how much they hate it on a scale of one to ten. Write it all down somewhere you won’t lose it. Future-you will be very, very grateful.

Step 2: The Metrics That Actually Matter

Now for the part where most people go off the rails.

The temptation, once you start measuring, is to measure everything. You discover Power BI. You discover Notion dashboards. You discover that you can chart literally anything against literally anything else, and before long you’ve built a 47-chart monstrosity that nobody, including you, will ever look at again.

A dashboard with 47 charts is not a metric. It’s a screensaver.

Track three things. That’s it.

1. Time saved - hours per week reclaimed by the team. The single most defensible number you’ll ever produce in a meeting.
2. Errors prevented - count them. Mistakes the bot doesn’t make, deadlines it doesn’t miss, customers it doesn’t accidentally email twice.
3. Team sanity - yes, just ask them. “On a scale of one to ten, how much does this task still ruin your week?” Then ask again in a month.

Three numbers. That’s the whole instrument panel. Nobody cares that “throughput volumetric efficiency” is up 12%. They care whether Karen from accounting got her Wednesday afternoons back.

Here’s the test we use: if a metric doesn’t change a decision you’d make, it’s decoration. Cut it. If knowing the number wouldn’t make you do anything differently, it’s noise dressed up as insight. Spend your time on the numbers that actually move you to act.

Step 3: The “Would You Go Back?” Test

Here’s the most honest metric we’ve ever found, and it costs nothing.

Thirty days after the automation goes live, walk over to your team and ask one question:

“Want to go back to doing it the old way?”

Then watch their faces.

If you get a horrified gasp and a quiet “please no” - congratulations. You automated something that genuinely mattered. The numbers will back it up, but you already know.

If you get a shrug, or worse, an indifferent “sure, whatever” - you’ve discovered something important. You automated something nobody actually cared about. That happens more than people admit. It’s not a failure; it’s a data point. Pick a different soul-crushing task and try again.

The “gasp test” works because humans are extremely bad at lying about relief. Spreadsheets can be massaged. Dashboards can be cherry-picked. But the involuntary “oh god no” when you suggest going back to the old way? That’s the realest metric in the building.

Trust the gasp.

The No-Stress Recap

Because we know by now you’re skimming (still respect):

1. The Baseline - Capture life before the bot. Hours, errors, eye-rolls. Write them down.
2. The Right Metrics - Track three things: time saved, errors prevented, team sanity. Skip the rest.
3. The Gut Check - After 30 days, ask the team if they’d go back. Trust their faces more than your charts.
4. The Decision Rule - If a number doesn’t change what you do next, it’s decoration. Cut it.

That’s it. That’s measurement. No PhD required.

Still Reading?

If you’ve made it this far, you’re either genuinely committed to measuring this properly, or you’re avoiding something even worse than spreadsheets. Either way, welcome.

You’ve now got everything you need to measure an automation honestly: a baseline, three real metrics, and the gasp test. Go forth and count.

And if you ever fancy some company on the rest of the ride - Timepress does the whole automation journey with you. Discovery, analysis, the actual building of the software, hosting it somewhere sensible, keeping it cheerful in the long run, and yes, the measurement we just spent 1,200 words on. Full menu, or just the bits you’d rather not solo. We’re easy like that.

Book a call. We’ll bring the software, the snacks, and a suspicious amount of enthusiasm for Karen’s Wednesday afternoons.