Measuring success in a non BS way
How not to get fooled by vanity metrics
We are on issue #8 of this newsletter, and by this point, it almost feels like a journal for the company. In fact, I often find myself referring it to interested new hires, to get a no-frills introduction into the inner going-ons of the company. It’s a time to reflect and look back at the last couple of weeks, and see what happened. It’s been immensely useful!
We have a bunch of VCs on the mailing list now (you know who you are 👀). Generally, subscriber count is slowly going up→So welcome to all new subscribers! I hope you get the chance to read the first seven issues of the newsletter. It is an ongoing, evolving story so please do take the time if you have it. On this one:
📏Measure your success, silly!
A friend of mine, who is a senior product manager, responded directly to my LinkedIn post where I publicized the last newsletter, asking for feedback:
She said:
I think u forgot to think about measuring the success… How are you measuring that you solved the problems you identified? How does success look like? And then map these tasks towards the goal that they are trying to achieve.
And she is absolutely right, of course. There is no point in doing all that planning if you’re not optimizing for some real-world goals that tell you whether you made the right choices. Indeed, I have talked about metrics before, but I wanted to take a slightly deeper dive today.
At ZenML, we’re optimizing for Weekly Active Users (WAU). DAU is too sporadic and hard to influence directly. MAU is too slow as it lags behind your efforts. WAU is a good balance→ We can look at WAU every Monday and adjust our activities accordingly.
However, just looking at WAU graph on MixPanel is not sufficient, if you do not define what “Active” means (Weekly Active Users, remember?). Defining what is active is dependent on the context of your application, your market, and ultimately your users. E.g. You could define an active user as someone who comes back at least twice a week, or spends over an hour on the app, or does an action a certain number of times. That is where you need to be critical and ask yourself what you’re optimizing for.
We made this mistake at ZenML at the start. We defined active users as users who just do any action with the pip package. However, that fails to account for many non-serious users who just run the quickstart and are curious. While that is also an important metric to track for our funnel, it does not qualify to be a North Star metric. We’re looking for users that are seriously testing us out in production. Therefore, we adjusted our metrics accordingly. By defining a more precise definition of active, our metrics fell a bit naturally but gave us a no-BS picture of where we are in our journey.
Once a key metric is defined, that number needs to go in the right direction at a certain growth level that you think is realistic and achievable, yet ambitious at the same time. For us, it’s about getting a certain number of qualified WAU by the end of this year to show significant adoption by the community. These numbers may not be a straight growth graph and may fluctuate, but needs to trend in the right direction. We then look at that number closely and any activity within the company is executed to influence that.
As eluded to last time, another metric we measure is Stickiness (DAU/MAU). This makes sure that DAU and MAU are not lost from our calculations:
Our stickiness tends to stay at 20%, which I have heard is good. We want to get this closer to 50% over the year, which would be excellent.
Lastly, another friend of mine, who is an executive at a high-growth startup, also gave me this sage advice:
When it comes to metrics, don’t overcomplicate it.
The team, investors (current and future), and you yourself need to understand these numbers quickly and act accordingly. In that case, simpler is better.
🏋️ Reinforcement Learning solves Atari
Ayush recently joined ZenML as our youngest team member. His first project involved getting his hands dirty with ZenML, and resulted in an awesome project where he showcased how to use reinforcement learning with ZenML to solve a classic Atari game!
Here’s Ayush talking about the project himself:
Or just read the blog post he also wrote, where he goes in-depth about the whole thing.
☁️Running Trainings on The Cloud
ZenML 0.6.3 was a solid release where the biggest feature was the ability to switch seamlessly between running steps on all the big cloud providers’ ML platforms (AWS, Google, Azure). Being cloud-agnostic is one of the key value-adds of ZenML, so I recorded a video to showcase it to the world:
I also wrote an accompanying blog that goes into more detail here. Hopefully, the community resonates with these much-requested features and we can crack on and make them even better in the coming weeks!
🦓MLOps at Stripe
Emmanuel Ameisen, ML Engineer at Stripe, graced the pod recently. Emmanuel wrote an excellent O’Reilly book called “Building Machine Learning Powered Applications”. Emmanuel’s book is packed with practical and useful examples of how to iterate quickly and deliver something that has value in the real world.
Listen to the full podcast over at the Pipeline Conversations page.
🛣️The Road to Zen with Fuzzy Labs
Our friends over at Fuzzy Labs have written the first part of a series about using ZenML! Matt Squire and Tom Stockton from Fuzzy Labs have been great early supporters, and this sort of content from the ZenML community really pushes us forward immensely.
P.S. Keep an eye out on the latest Pipeline Conversations, Matt might even make a special appearance soon!
🐦Twitter Spaces
Did you know that Twitter Spaces is a thing? I didn’t really. I’m not much of a Twitter aficionado. However, I joined a few in the last few days and they have been immense fun! So much so that we decided to do one of our own today! It went great at you can listen to the entire recording for the next 30 days. We had lots of prolific people joining and sharing their thoughts on MLOps and it was all good fun. Check it out now: https://twitter.com/i/spaces/1djGXPAjlPRGZ.
For those wondering how we publicized, we simply created this banner and posted it on the socials. It was low effort and easy to prepare, and definitely worth it:
Keep an eye out for the next one!
And that’s it for this time. As always, I will see you next time, and have a great weekend!
Cheers,
—H