Some background:
The biggest internal stakeholder of our analytics team is the sales team. The CGO / CSO is hands on, joins all of our analytics-sales team meetings, and outranks the head of our analytics team, who is also present during all of our meetings.
Our company sells a B2B product. The bulk of our analytics projects involve running analyses to show that this product works in the particular customer context (sorry for the vagueness, trying to be anonymous here). The sales team presents these analyses to these customers to convince them to buy it.
We belong in an industry where it seems that customers infrequently conduct their own analyses or experiments to verify that products (like ours) works. So, weirdly to me, customers tend to just trust the analytics we present.
That’s the set-up. Now the troubling parts:
The sales team and analytics teams have tried running A/B tests in the past to demonstrate product value, but the results showed the product was of no benefit on the single, key metric of interest. Since then, the sales team has discouraged the use of A/B tests because “we don’t understand the results.” The head of the analytics team goes along with that.
We run observational, causal analyses to demonstrate an effect and the raw results vary from weak negative effect, to no effect, to weak positive effect. It’s a noisy analysis.
The few times one of our external customers has tried to A/B test the product in a trial run before fully buying it, the results have always been null.
To my eyes, all of the above averages out to *the product does not work*.
Whenever we find a negative or null effect in an analysis, the sales team pressures us into re-running the analysis looking at different sub-groups until we find a positive effect / “good story for the client”. They do this in sometimes subtle ways (e.g. “I think our data isn’t capturing a subtlety in this customer’s industry”) to more obvious pressure (e.g. “look, we have a client meeting on Friday. We need a good story. Try looking at it X way or Y way in the data). I have ample email evidence of this.
I’ve tried explaining to both teams that it is bad statistical practice (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4840791/) to do multiple testing / fishing / p-hacking to find good stories, but no one seems to understand this or perhaps care. So we keep doing this practice, and the sales team finds a way to spin it into a narrative for the customer.
I’ve spoken with the head of the analytics team, the HR team, and even the CEO about the situation. I’m diplomatic, so in all of these conversations I’ve made sure to not heap blame on individuals but calmly insist that the system is at fault and provide suggestions on how to fix it. In all of these cases, people give me platitudes about the need for integrity, thank me for speaking up, but literally nothing happens.
I’m still at the company but currently actively looking around for new jobs. What is an employee supposed to do in this situation? It’s a private company so there’s no whistleblowing to be done, I imagine. Would it be right to secretly inform our customers of the situation? Am I being overly dramatic?
If you are unwilling to sell something useless, then your best bet unfortunately is to quit.
You should just find another job. I wouldn't inform customers.
I've been on the other side of this as a customer and I think you might be assuming some standard of ethics/quality at the customer side that may not exist. I'll provide an example.
A few years ago I managed a Data Science team at a large public company. The division I was in had no real marketing strategy for social media. A new CMO (for the division, not the whole company) was hired who was very fixated on marketing on Facebook. So they built a social media marketing team.
Over the next year or so they spent >$1.5 million in various proof-of-concepts and bake-offs with various companies to find a vendor who could "move the needle" on Facebook. IMHO the whole thing was a pile of nonsense and all but 1 vendor (who got eliminated fairly early) was full of crap. The vendor who won was the one who had a bunch of PowerPoints showing how their strategy on Facebook had great results - I think it was all BS.
The social media team loved it took the charts to the CMO. The CMO took it higher up and used it to talk about how they had built the right team, how they had the right vision. Everybody was patting their own backs and probably polishing their resumes so they could say how they transformed the marketing strategy at the company. Nobody wanted the truth, they just wanted to show their bosses how great they were.
4-5 years later the whole division got a massive haircut and the marketing functions got moved back the parent company. But in those 4-5 years, if you said "This is all BS" in public you were likely to get slaughtered.
Just move on. I left pretty quick after realizing nobody cared what I thought unless I agreed with all of it.
Good luck!