Of metrics and orbits
How does one measure community and product traction?
Hey everyone!
We’re up to nearly a 100 subscribers on this newsletter and it’s nearly all been organic.
I wonder if those numbers are good or bad → the growth is pretty steady though! Anyway, on to what’s happened since we last spoke.
✨Metrics and Numbers
The eternal bane of any open-source startup is how the heck to measure who and where someone is using your tool. We have anonymous telemetry built-in to ZenML (that can be opted-out, of course) since late October 21. That gives us the ability to use MixPanel to do all sorts of fun stuff:
For me, one of the most interesting metrics is the Active Users (Daily, Weekly, Monthly). Retention is also an important one, while the pipeline runs per user is also a great indication. We are at an early growth phase of these metrics, and they give us exactly the information we need to validate our Go To Market. Let’s see how they develop over this year!
One interesting new metric that we discovered recently is “stickiness”, the ratio of the Daily Active Users (DAU) to the Monthly Active Users (MAU).
Stickiness = DAU / MAU
It was popularised by Facebook who have had an absolutely crazy stickiness factor ever since they launched. We are still assessing whether it applies to our use-case or not — it might not because its still hard to identify in our case whether a user is an ephemeral bot or a user. However, if we are able to distinguish that in our telemetry, it might be still a useful metric going forward.
🪐The Orbit Model for DevRel
Another new model we discovered recently is the Orbit Model to measure and grow communities. The GitHub explains it well:
The Orbit Model contains five fundamental concepts: Gravity, Love, Reach, Orbit Levels, and Presence.
Gravity is the attractive force of a community that acts to retain existing members and attract new ones.
Love is a member's overall level of engagement and impact in the community.
Reach is a measure of a community member’s sphere of influence.
Orbit levels are a practical tool for member segmentation and help you create different programs for different types of community members.
Presence is a measure of a community member's activity over a period of time.
We ended up signing up for the SaaS application that actually lets us connect Slack, Twitter, and LinkedIn to count the Gravity, Love, Reach, Orbit Levels and Presence of people ‘orbiting’ around ZenML. It is a way neater way of looking at developer relations (DevRel) than simple funnel calculus.
What does everyone here think about these models and metrics? Any success trying to find out how to measure ? If so, reply and let me know!
👪The Team Pulling Through
🕺 There is nothing that makes a startup more successful than a stellar team 🕺. I am privileged to be working every day with my colleagues every day: From discussing key strategic decisions of the product to brainstorming PC builds with them in a cooldown session, it’s the most fun I've had at work ever.
Last week was intense: I was to present at the MLOps Community to showcase our latest release. But disaster struck at the last minute: Things failing 🏴, red check marks ❌ in builds, error messages in notebooks🤞🏼! The presentation was at 6PM and I was sweating bullets. And then this happened:
The entire team came through, split up into squads and just executed. The day was a typical startup roller coaster. Commits left and right, pull requests, and so on. From 3PM onwards everyone simply got into one big Discord Voice Channel, and started a mass debugging.
At 5PM we made the release.
At 5:51PM the first pipeline ran on my machine.
At 6PM I was live with the community.
The demo went off without a glitch (see video below). It showcased perfectly the story we wanted to tell, and even more miraculously worked for the people who coded alongside me in the meetup!
Of course, none of this was visible to the actual viewers. All of it was behind the scenes before, and even during, the call. As Alexej put it in his blog:
"It is a real testament to how fast we can move as a team 💪. It is a real privilege working together with everyone here."
There was a moment afterwards where I got back on the call and everyone was still waiting and we had 15 minutes of celebration. No the world had not changed, the north star metrics had not 10x'd. But it was just 15 minutes of happiness for what we had pulled off, that only the team knew, and could be joyful about. And these are the victories we must take alongside the bumps in this startups journey. See the full video below:
You can also look at the accompanying blog post or repository to get straight into it.
🦾New Standup Bot
Every day, I was tasked with spinning a random wheel to find out which order we do our standups. So like any lazy engineer, I automated it with a bot (Similar to the KPI bot I spoke about a few weeks ago):
I know that some of you wanted me to open-source that KPI bot (@James is refactoring it for that purpose already). Do you also want this bot to be open-sourced? Let me know!
✍️The most successful blog post
We have a lot of new content for everyone to enjoy:
Alex wrote a blog post about our latest integration with evidently that got a lot of love. Read it here: 10 Reasons ZenML ❤️ Evidently AI's Monitoring Tool.
Ever wonder how to make Python more typed than it is out-of-the-box? Or why it’s even necessary to do so? Read Alex’s second blog about why type hints are good for the soul here.
Mohan Mahadevan was on the podcast to talk about NeuroSymbolic AI.
That’s a wrap. I bid you farewell, and hope to see you again in two weeks!
—H