When you think about product analysts, what comes to your mind?

I bet it is people analysing metrics and a Mixpanel dashboard.

Those dashboards represent data coming from oddly sourced test groups. The test group is often made up of friends, acquaintances, or a Discord server, which is hardly representative of a broader target audience.

And if you are performing data science, cohort analysis, funnel work or retention analysis on those users, it is frankly equivalent to garbage because the data was never random or representative in the first place.

If I was running product at an early-stage company that had just got its first users, my analytics dashboard would literally just be a table of users with these four columns:

  1. Name
  2. Registration Date
  3. Last Active Time
  4. Number of Sessions

Then I would do these four things:

  1. Sort the list by Last Active Time.
  2. Look up the most active users on Instagram or LinkedIn.
  3. Interpret their behaviour on my app through the lens of their online identity: what would this person think of this?
  4. If I have a messaging channel, I would send them a generic message asking for feedback and maybe offer a small gift card or free in-app purchase.

And this would be repeated.

It is certainly not science or fancy and techy, but it will tell me more about what is resonating about the product than a bunch of statistically insignificant data. This is the real KYC.

The best thing data scientists did at Robinhood was sit in on user research sessions to hear directly from real customers. The second was spending more time deeply looking at detailed screen transition logs from a few users rather than focusing on aggregated metrics.

We think we know better. We think the data probably knows better. But the users know the best.