The marketing automation reports that mislead you
The open rate is the most quoted number in email marketing and it now largely measures which mail clients your audience uses. Privacy features in common mail apps fetch tracking images before a human sees the message, so a rising open rate can mean nothing more than a shift in device mix. Several other standard reports have similar problems, and knowing which ones to distrust is more useful than adding another dashboard.
Why is the open rate no longer a real metric?
Because an open is recorded when a tracking image loads, and mail privacy features in widely used clients load that image on a proxy server as soon as the message arrives, whether or not anyone reads it. For an audience using those clients the open rate approaches the delivery rate and stops varying with anything you control.
That does not make it entirely useless, but it changes what it can be used for. As a trend within a single stable audience it can still show a large break, such as mail suddenly not arriving. As a comparison between two subject lines it is now noise, because the proxy opens are unaffected by the subject.
The replacement is click behaviour and reply behaviour, with the caveat that some security scanners follow every link in a message and generate clicks nobody made. Filtering clicks that arrive within a second of delivery, or from a single address hitting every link at once, removes most of that.
Which denominators are wrong by default?
Several, and the difference is often large enough to change a decision. Click rate is reported against delivered in some tools and against sent in others, which makes cross-tool comparisons meaningless. Click-to-open rate inherits every problem the open rate has and should be retired rather than corrected.
The most consequential error is the funnel conversion rate computed across mismatched cohorts. Dividing deals closed this month by leads created this month compares two different groups of people, because the deals closing now came from leads created across the previous months. When lead volume is growing, that ratio understates conversion; when volume falls, it flatters it.
The fix is to follow cohorts. Take the leads created in one month and report what happened to that specific group over the following periods. The number arrives later and is worth waiting for, because it is the only version that can be compared with the same figure from last year.
| Metric | What people think it measures | What it measures | Read instead |
|---|---|---|---|
| Open rate | Interest in the subject line | Which mail clients your audience uses | Clicks and replies, with bot clicks filtered |
| Delivery rate | Mail reaching the inbox | Mail the receiving server accepted, including into spam | Seed account placement across providers |
| Click-to-open rate | Content quality among readers | The same proxy behaviour, divided by itself | Click rate against delivered, consistently |
| Monthly conversion rate | How well leads convert | This month's deals over an unrelated month's leads | Cohort conversion, tracked forward from creation |
| Campaign influenced pipeline | Revenue the campaign produced | Every deal that touched the campaign, counted in full | One arbiter model, with the double counting removed |
| Qualified lead count | Sales-ready demand | Where the threshold currently sits | Qualified leads that sales accepted and worked |
Why does influenced pipeline always look enormous?
Because influence is not exclusive. A deal that touched six campaigns appears in the influenced total of all six, at full value, so the sum across campaigns can exceed total revenue several times over. Nothing is being calculated incorrectly; the number simply cannot be added up, and it is almost always presented in a table that invites adding it up.
The number does have a legitimate use, which is to see whether a campaign appears in the histories of deals that closed at all. A campaign absent from every won deal is a reasonable candidate for review.
For anything involving budget, use a single fractional model with one arbiter system so the totals reconcile to real revenue. Then state plainly, in the report itself, that the split is a convention rather than a measurement, because the next person to read it will otherwise treat those decimals as facts.
How do A/B tests go wrong in these platforms?
Most often by being stopped when the result looks good. Platform test features frequently display a running winner, and watching that indicator until it says what you hoped is not a test, it is a way of guaranteeing a result. The decision to stop must be fixed in advance, on a sample size chosen before the send.
The second problem is testing things that cannot move enough to be detectable at your volume. Small differences require large samples, and a send to a few thousand people cannot reliably distinguish a small improvement from noise. That does not mean the improvement is absent, it means the test cannot see it and running it anyway generates false confidence.
The third is testing the subject line while measuring the open rate, which returns us to the proxy problem. If you test subject lines, measure clicks or replies as the outcome, since those at least require a human.
What should a monthly report actually contain?
Few numbers, each with a decision attached. New contacts by source, cohort conversion for the cohorts old enough to have converted, sales acceptance rate of qualified leads, complaint and unsubscribe rates, and the count of contacts receiving mail from more than one live sequence. That is close to sufficient for most programmes.
Every number should have a named owner and a stated action if it moves in either direction. A metric nobody would act on is decoration, and removing it makes the ones that remain easier to notice.
Include one qualitative item: a handful of actual replies, quoted. They are the only part of the report that tells you what people think, and they routinely explain a movement in the numbers that no dashboard could.
How do you check whether your reports are honest?
Pick your headline number and trace it to the underlying records. Export the rows that make it up and count them by hand. Discrepancies are common and usually come from a filter someone added a year ago for a reason nobody remembers, or a date field that is not the one you assumed.
Then ask which system each number came from, and whether two of them describe the same thing with different values. If the CRM and the platform report different lead counts for the same period, resolve which is correct before anyone presents either, because that gap destroys credibility more thoroughly than a bad result.
Finally, look for numbers that only ever move upwards. Cumulative counts, database size and lifetime totals grow regardless of performance, which is what makes them attractive on a slide. Any metric that cannot get worse is not measuring anything.
Common questions
- Why is email open rate unreliable?
- An open is recorded when a tracking image loads, and privacy features in widely used mail clients load that image on a proxy as soon as the message arrives, before any human sees it. For audiences on those clients the open rate approaches the delivery rate and stops responding to anything you change. It can still reveal a large break, such as mail not arriving, but it cannot compare two subject lines.
- Why do funnel conversion rates calculated monthly mislead?
- Because they divide deals closed this month by leads created this month, which are two different groups of people. The deals closing now came from leads created over previous months. When lead volume is growing the ratio understates conversion, and when volume falls it flatters it. Follow cohorts instead: take one month's leads and report what happened to that specific group over time.
- Why does campaign influenced pipeline exceed total revenue?
- Because influence is not exclusive. A deal that touched six campaigns is counted at full value in all six, so the column sums to several times actual revenue. Nothing is calculated wrongly, but the number cannot be added up and is usually shown in a table that invites exactly that. For budget decisions, use a single fractional model in one arbiter system so totals reconcile to real revenue.
- How should you run an A/B test in a marketing platform?
- Fix the sample size and the stopping point before sending, and ignore the running winner indicator, because watching until it shows the result you wanted guarantees that result. Measure clicks or replies rather than opens, since privacy proxies corrupt open data. Accept that small differences are undetectable at modest volumes, and that running the test anyway produces confidence rather than information.
- What belongs in a monthly marketing automation report?
- New contacts by source, cohort conversion for cohorts old enough to have converted, the share of qualified leads sales actually accepted, complaint and unsubscribe rates, and the number of contacts enrolled in more than one live sequence. Each number needs an owner and a stated action if it moves. Include a few real replies quoted directly, since they routinely explain movements no dashboard can.
