How can I improve our Published Metrics?

I ask this question every week at my inworld office hours and in my regular blog posts, but there's no such thing as too much input on this, especially when I get reviews like this. (ouchie!) This post is fair criticism of our February stats, though certainly these past few months since have brought a lot of changes and improvements, on each of the concerns raised. Some bloggers are noticing the improvements. Each of these is a step forward, and I have lots more to come, but I would postulate that we are the most transparent and open company in both the Metaverse and MMO spaces in terms of real published metrics. I hope to stay well ahead of the curve with this, too.

Metrics Terms Definitions - our new wiki page for the Metrics Glossary will be the core updated source of explanation of the numbers. There is already a starter discussion happening about it, please contribute your questions/comments there, or comment here. The next step is to link the glossary terms from the headers of the actual reports. I have an interesting approach I'll be using for this- to be expanded in an upcoming post.

I have two new categories in the public JIRA, under the WEB heading. One is for our public reports and charts that we publish on a regular basis, and the other is for our data feeds. This is our tool for entering new feature (report) requests and to identify and track any bugs that you find. Please use this tool to vote for which new datasets you would like to see- and if there's one you want that isn't listed, please add it! There's a great guide for using JIRA for non-techies on Kala Bijoux's blog that's worth your time to explore.

Hey, you got your Virtual Chocolate in my Sculptey Peanut Butter! or, how to rank MMOs / Metaverses?

What a fascinating conversation has arisen from GigaOm's Top 10 MMO post. The ranking discussion has focused around reported figures of "active users" and I have a few comments on this ranking.

1. Apples (Registered Users) != Cherries (Active Users) : Most MMO systems report their userbase as Registered Users, which correlates to our public Residents metric. This includes Second Life users who registered on the website but have never logged in, for lots of reasons including "didn't want to download a client", "don't have fast enough video card to run", "don't have broadband", etc. Clay's well publicized criticism on valleywag calls this out - for us. What about the rest of the lot? For MMO Games that "never logged in" number would be much lower, as generally their login process includes a shrinkwrapped box with Minimum System Requirements printed clearly on the wrapper, and the product is marketed clearly at the Computer Gamer demographic- as opposed to the Metaverse target user, which is "everyone- who wants a Second Life", but that number of "never logged in" still exists for them- who reports it, besides us? It also includes users who logged in previously but have not logged on recently. There are lots of comments here, here, here, and here about how the reported numbers for other systems include inactive users. Personally, Registered users is one of the most irrelevant and distracting metrics you can discuss, as it is subject to the whelms of public attention, and will swell and shrink dramatically weekly in unexpected ways, based on Media focus. Try to relate Registered Users to the metrics of early-90s websites, and their long-term retention to get a sense of relevance.

2. Logged in Users (Oranges) != Active users (Cherries): Total hits of users on any social online system will be divided between those browsing it, and those contributing to it. Those contributing will invest more time and attention, and make the system more valuable. We report Logged in users overall for the month, and break it down by Gender. Active users we go much further and break down by Country and by Age Band. This is a more interesting set of data, on who is building our Better Life, in my humble opinion.

3. Hours spent inworld == more pertinent metric. Counts of Users, like Hitcounts, can always be gamed. Real attention is measured in real time invested. Second Life is much more than a website, it is a fully engaging Application with lots of Resources available. Residents who find the resources valuable will spend time here, and expand them. Browsers who don't find what they're looking for will look elsewhere. There are other online worlds experiencing this disparity. 

4. Active Users (Cherries) == "fully engaged users". This is always a grey area, as there is a gradient at play here on who contributes more, but there are obvious trends. I declare Active users as someone who spends more than a cumulative Hour on the system in a Month. These are our "sweet spot" users where the system meets their needs and they are investing their time and creativity to make the system better. They are contributing their communications, content, and social networks. They're Building, and Scripting, and creating groups. Active Users will include our community of blogging residents, designers, and otherwise Business Owners- whether through inworld products, services, or connectivity/marketing to outer web presences. Most are spending a lot more than 1 hour a month, once they break that barrier.

5. Metaverse != MMO Game: This topic has been flogged ceaselessly, but it's important to point out in this discusion. An MMPORG will have different demographics from a Metaverse. That is not to say that overall comparison isn't valuable, let's just make sure we're taking at least Oranges to Oranges. I don't expect every MMO worldwide to start disclosing their ratios of Cherries, but indeed, let's discard the Bad Apples from the conversation.

Mashups and Alternative views of Public metrics

Exactly as I hoped, when I published the specific figures for our May 2007 public metrics along with the percentages, now many more people are looking seriously at them from different angles, and asking really great questions.

First the folks over at MillionsOfUs directly divided the sum of the hours spent inworld by the count of active users, to find out the number of minutes per day per user per country. Cayman Islands comes in on top with a whopping 161 minutes daily average across 36 active users. Those users would be an obvious example of a statistical outlier, given that Cayman Islands only shows up in the Top 100 Countries by Active User Hours, and fails to show up in to Top 100 Countries by Active User Counts at all. We could potentially redefine our Active figure to use a 95th or 90th Percentile figure to remove anything outside of a reasonable Standard Deviation - thoughts? Discussion? What do you or other companies use for their public metrics?
I will make a couple small notes of correction for that post, technically "The average [active] SL user (without regard to geography) spends about 90[69] minutes in world every day. " I'll talk alot about our definition of active users in another post soon. I actually calculate our daily average for May Active users as (18.195M H / 507.844K Active Users)/31 Days in May * 60 Min = ~69 minutes. Extra eyes on that calculation are welcome...

Next Otenth looks at the metrics as a percentage of population, I love that he cites his source for population demographics and also publishes on google docs, and this time on the flip side of the scale, Antarctica pops up as an interesting phenomenon. There are a lot of remote places globally that are now getting fibre and other fast net connections, and I know lots of residents who love the social aspect of SL who live far from urban centres. Anyone have an appropriate dataset on "remoteness" to mix with this?

Then Erasmus takes it a step further to call this calculation Market Penetration and ranks the active user counts by 1000's of population. He doesn't cite his source for demographics, so I'm not exactly able to reproduce those numbers, although they look in the ballpark for what I know about the more active countries- particularly european. As they state, this ranking doesn't take economic or technological angles into account. I'd love to see these figures also joined up with perhaps some Broadband Penetration stats such as these to see how the rankings weight out in that dimension.

And to wrap up this list, Giff/Forseti comments on what I think is the most interesting angle of the change- Hours spent inworld. He shows that the ranking of avg hrs inworld by resident by country loosely follows the rankings shown by Erasmus, above, then specifically notes the increased Attention inworld by our older users - we might have a long list of young curious residents who come and browse briefly, but clearly there is something interesting enough in Second Life that our older users find reason to stay. I am likely to talk lots about the importance of Attention as a real measure of penetration. Other sites also use hours as a ranking of importance, I'd love to see this become a universal measure.
Also, under his "Data Accuracy" heading (ouch!) he points out that the counts and hours of the Gender tab don't match the counts and hours for metrics by Country or Age - which I had explained in my errata/changes section of my blog post.
To be clear here, under the category of "customer" or "residents" we really have 3 top-level sets of numbers. First is the raw Residents number, the often-debated figure of Signups who have accounts and can log into Second Life (7+M) - this includes many people who have never logged in, for lots of reasons, including "didn't want to download a client", "don't have fast enough video card to run", "don't have broadband", etc. Second is our "Logged in Users" figure which includes *everyone* who logs in during that month- those figures are on the Total Hours and Gender tabs of the report. Logged in users includes both "browsing" users who spend less than an hour inworld, as well as all of the Active users. Third is our Active Users, who each spends greater than an hour inworld during the month, which is divided by Country and by Age bracket. Statistically speaking, the 1-hour mark is a bit of a magick number, as it tends to mark a turning point where a resident finds something of interest inworld and begins to invest their time- often lots of time. Given the modern fact of short attention spans, if the system is too confusing, or the residents are unable to find (search and friendslist bugs?) or get to (teleport bugs?) anything of interest before that magickal hour is up, they are less likely to return in the near term.
Actually, I'll also note that the Gender breakdown is not about Avatars but is about the self-reported Gender of the customer behind the avatar at registration time. There has been speculation and discussion about the fact that "male" is the default selected during signup, which can have a major effect on the overall counts, which we do see level out for Active users. Fact: "browsing" users will generally get through the signup with as little updated info as possible, while those who will invest more time inworld take the time to fill out the signup more completely- and this is visible in the disparity between the Counts and the Hours.

If we were to include Avatar gender (which would be ferociously difficult- given that we're all "children of Ruth" I'd have to measure, what, the bust size slider measure for your active shape?) we'd have to include an "Other" bucket for robots, non-gendered Furries and other critters, Hermaphrodites, and those who choose to switch back and forth at will. What is the gender of the Flying Spaghetti Monster anyhow? I guess we do say "His noodly appendage" but I'll be darned if I can find anything I can measure for that avatar that tells me it's a male. :)