The 7 Circles of Implementation Hell
Key learnings on implementing and fixing data implementations
With over a decade of helping companies get more out of their data and helping companies with their data architecture I’d like to share with you the main issues I’ve seen.
I start each client conversation with “I have two goals. Making sure the data is trustworthy and usable.”
What is usable? Making sure you are able to answer all your questions.
After seeing so many implementations at many stages of companies and wanted to share the most common issues and the impact they have.
But before we dive into good or bad. Let’s understand why we should care:
Enable better conversion rate
Reach dormant users and retain them
Have better customer experience
Segment data for enrichment and personalization
Reduce costs
And so much more
So let’s start with what is a good implementation:
Main KPIs are easily reportable
A new user who is familiar with the product should be able to easily find data and create the basic funnel
Data is standardized - a consistent naming convention exists across events; events/properties are parallel across platforms
Data governance structure in place
Incoming data is close to source of truth
Now let’s dig into what are the biggest issues sorted by fix effort
Level 1: Missing/Incomplete Data: Fix Effort 2/10
Why it matters
In short. You can’t answer your key product or business questions
Your company isn’t getting the full value out of the data
How to Fix
Determine which KPIs aren’t reportable
Add requisite events/properties to capture data that will allow all KPIs to be reported on
Send the missing data
Level 2: Inconsistent Naming Conventions: Fix Effort 3/10
Why it matters
Can be a good indicator of future problems
Likely no data governance in place
Different teams use different events and if you bring someone in they won’t easily be able to build the basic funnel.
How to fix
Merge similar events (via CDP or other methods)
Rename outlier events/properties
Create a standardization document and add descriptions to each event and property
Level 3: Colliding/Illogically Separated Data: Fix Effort 5/10
Why it matters:
Likely to drive incorrect conclusions from data -- reports aren’t telling users what they think it’s telling them.
How to fix:
Work to find a way to logically separate the data. For example event A coming from the web means that it should be defined as X and the same event coming form mobile means a different
Level 4: ID Issues: Fix Effort 6/10
Why it matters:
Current data set likely borderline unusable - unique reporting, funnels, retention, and flows all incorrect
Users counted multiple times, Pre auth / Post auth are not connected and more.
How to fix:
Reimplement identity management and think to start on a new project
Level 5: Wrong/Unknown Data: Fix Effort 7/10
Why it matters:
Data to measure KPIs either don’t exist, or not known which events would be used.
How to fix:
Ideal time to start a new project - Will likely need to add new tracking/fix existing tracking.
Level 6: Tracking at Wrong Depth: Fix Effort 8/10
Why it matters:
Building reports in your Analytics/BI tool is difficult and time-consuming
How to fix:
Need to fully overhaul the implementation - design a new spec and recode everything
Level 7: Not knowing what to track: Fix Effort 10/10
Why it matters:
This is a different article.
How to fix:
Define what you want to measure and build a tracking plan
As you can tell there’s a common denominator to all of the levels. Design quality and data governance. It’s key to understand that your data should be treated as an additional product and not as a side just like your technology needs and stack evolve so does your data needs and stack.
Keep it clean and usable
Alon