GameSetWatch writes:
"Recently, there's been a significantly greater profile for the video game chart compilation site, VGChartz.
As well as beginning to contact major news sites on a regular basis to disseminate its news, the site was also the subject of a positive article on O'Reilly Radar from Robert Passarella, comparing its open data dissemination method favorably with The NPD Group, the generally agreed 'canonical' source for North American game charts.
Indeeed, as the Wikipedia page for VGChartz notes, Forbes, Fortune, The New York Post, and The New York Times have all referenced the site. And since it's been more aggressively marketed by the site's creators - especially regarding the 'holy grail' of global sales figure comparisons, its references in the news are rapidly increasing in frequency.
But how is the site actually compiled, and is it a good source for reasonably reputable news websites such as Gamasutra to be citing? Thus far, we have referenced VGChartz data twice - once with regard to Xbox Live Arcade game sales, and more recently because Michael Pachter has started to cite the data in his NPD game sales previews.
The second citation provoked a number of reader queries about the veracity of the data, so we embarked on some detailed analysis of VGChartz, and followed it up with a long series of emails with the site's creator, Brett Walton."
This could be fun as they make great tables. Go big or go extinct. Prime your senses for a neural handshake and step into the cockpit of a Jaeger. It is on you to cancel the apocalypse when Pacific Rim Pinball comes to Pinball FX on May 16.
GL compiles a list of some of the most mind-blowing video game narrative twists in recent memory, from The Last of Us to Outer Wilds
With articles like these cant you tag the games mentioned so that we can know ahead of time if there’s a spoiler to avoid?
Not clicking on your article otherwise.
Gary Green said: We have a juxtaposition of 2D and 3D visuals, flashy turn-based combat, quirky anime characters with cheeky dialogue with plenty of partial nudity; Yes, this is a Compile Heart JRPG. Whilst the engine is borrowed from Hyperdimension Neptunia mk2, Mugen Souls is more of a Disgaea spin-off. It’s not a strategy RPG as such, it merely sits within Disgaea’s ever-expanding universe (Multiverse? Netherverse? Your guess is as good as mine). You won’t find cameos though, since Mugen Souls is a franchise which aims to stand on its own two feet.
For god's sakes, people! They spell charts with a 'z'!!!!
well it seems that vg is crappier than crap
But how accurate is it? If I was, say, writing a story for the New York Times, what proof do I have that the 'correct' numbers are displayed on the site? Obviously, as mentioned above, all sales figures are by necessity estimates, and that's the crux of the issue - we'll get back to that later. But I asked Brett Walton his methodology, and he gave me the following, quite impressive answer:
"The methodology we use for all of our charts in all regions is the same and our data is arrived at by a combination of the following:
- Sampled direct sell-through data
- Industry knowledge and experience - applying past trends in terms of marketshares, regional breakdowns, casual vs hardcore and so on
- plenty of statistical analysis, regression calculations, market projections
- Contact with industry figures - buy-side analysts (such as Pachter / Divnich), sell-side analysts who work with us on specific products / projects, manufacturers who work with us to project sales of their key titles
- Retail checks - we have a team who talk to stores and estimate shipment figures for low-stock and hard to find items which we struggle to track with our normal data samples.
Exactly how we get from these various sources of data to final figures differs from game to game and console to console and our exact methodologies are confidential for obvious reasons."
Essentially, Walton is saying that he uses a number of high quality factors to produce his estimates, but can't mention any of the retail sources, or companies that VGChartz works with. Well, fair enough. But did you realize that VGChartz estimates can retroactively change by 100% or more based on 'official' chart results?
Iron Man & Retrofitting
One of the most unexpected results in the recent NPD charts for May was the appearance of the poorly reviewed Iron Man game for PlayStation 2 in the Top 10 of the charts, with 130,000 copies sold.
Thinking about it carefully, with the movie rocketing to unexpected success during the month, it would make sense that the game would sell well. But it's not the kind of game that you're likely to estimate in the Top 10 - and indeed VGChartz did not, estimating 53,000 units in sales, according to VGChartz staffers.
But what's surprising is that Iron Man for PlayStation 2 has been adjusted in its official VGChartz page so that its first four weeks of sales (encompassing May) add up to 111,000 units.
Clearly, these numbers have been changed after NPD debuted, showing a couple of things. Firstly, if you were a journalist, you could have cited VGChartz as saying Iron Man was a flop on PS2, selling half as many units - when NPD vibrantly disagrees. In addition, and more interestingly, it shows that VGChartz trusts NPD over their own prediction data by retroactively changing things to better match.
Apparently, this has happened before, because in a FAQ about North American VGChartz numbers, Brett Walton addresses this precise subject:
"Do we adjust our data? Not as such. Do we adjust our methods then? Yes - which will of course alter some data. On what basis? If we believe that a particular data set differs significantly from other sources of data (data released into the public domain by tracking firms, manufacturers, analysts) then we do re-check our data and make adjustments to the methods / scaling factors used.
This happens on a fairly infrequent basis - less often than we adjust due to internal data changes - and is something that every tracking firm and analyst does. I personally have no issues with "benchmarking" our data from time to time against other sources of data - as long as it has been made public."
In other words, if they are sufficiently out, then VGChartz will retrofit their results - either weekly or monthly - to conform to the more 'official' data. But they won't credit those firms as the source of the retrofitting - they'll just bump their numbers around without saying why on the site.
As a result, we get to what VGChartz actually is - a strange mixture of a prediction market (as consensus prediction site TheSimExchange is) and a retroactive, but non-credited reflection of charts that have historically been known for having more concrete data.
Where's The Beef?
OK, so you might say - and a lot of VGChartz' forumgoers do - what's the problem with that? If VGChartz gets close enough, and can adjust if it's too far off when top-end data comes out, then why would there be a problem?
Well, because you then have a moving target for checking/reporting purposes, and particularly because there's a high probability that VGChartz figures will be significantly wrong for those titles on the lower end of sales - those that lurk outside the top of the charts.
In other words, for those high-selling titles, VGChartz is checking against public data, and they will change their estimates if they are majorly off. Most of the time, they are quite close compared to the worldwide charts. That's because VGChartz is - like services such as The SimExchange - using common sense, Internet buzz, real-time data such as Amazon.com and analyst commentary to synthesize a sensible estimate.
But in covering all games, they are doing readers a disservice, because it's clear from the Iron Man example that they simply do not have the direct sale retail contacts to extrapolate unexpected but nonetheless true results. And if a title spikes but is outside public data, VGChartz will never catch it.
And the amount of concrete data available to VGChartz is low - as is freely admitted in a recent interview, VGChartz had 2-3% of the North American market as a sample at the time, whereas by estimate, NPD might have 60-65%. If this 2-3% was clean and canonical, this might not matter - but how do you explain the big Iron Man discrepancy, if so? Wouldn't VGChartz' retail sources have picked it up too?
So, let's take a step back and concentrate on some games that have sold in significant numbers, but have never made it into the Top 20 in North America for a significant time.
One good example is the Ben 10 series of games from D3 Publisher. VGChartz has the series listed at 590,000 sold worldwide to date. But when Gamasutra interviewed D3's Yoji Takenaka last week, he specifically said: "Ben 10 is selling well over a million units right now, since last Christmas."
So sure, Takenaka could be conflating shipped with sold - making the number closer to the estimate. But that's an awfully large discrepancy - one that most people won't care about because it's not a prominent or critically acclaimed game, and there's no way to refute VGChartz on it, but a discrepancy nonetheless.
Unfortunately, we don't have lifetime NPD data for this set of titles - but in researching this story, we spoke to a third party who had access to NPD lifetime to date sales that are not normally disclosed to the public.
We picked two titles released for one of the next-gen consoles over the previous year, neither of which had been in the public NPD charts for more than a month, leaving VGChartz to make estimates based on their own sources on their selling curve over time.
Well, somewhat spectacularly, in both cases, NPD and VGChartz disagreed by about 100%. In one case, VGChartz was citing 300,000 sales, whereas NPD had the game at 150,000 units. And in the other case, it was inversed - NPD had the game at around 200,000, but VGChartz had it at 100,000.
"The headline actually originally read 1.5 million, but was changed by a not insignificant 200,000 units after publication. Even more surprisingly, the figure debuted just 48 hours after the launch of the game - not a lot of time to compile data from retail sources. "
The article was posted on the 14th June... MGS4 came out June 12. They even link to the article themselves. o_O
It's the best analysis and explanation of what VGChartz is and makes some sensible arguments about the misleading nature of some estimates being reported as fact. The altering of estimates post NPD, whilst an acceptable part of an analysts job, does raise a number of issues with transparency and whether VG should release any data before the release of Chart Track, NPD, etc...
Their lack of data sources at retail is a major concern for me to. Basing estimates off of 3% of retailers compared to the NPD's 60+% is a mighty difference. It may also explain why some people feel the site is bias. I don't believe that the people behind it are inherently bias as it goes against financial/goodwill incentives, but if you are only sampling 3% of a retail population you leave yourself open to a very biased methodology for estimates. For example, imagine your 3% sample comes mostly from lower income states of the US where consumers have a preference for cheaper consoles compared to the relatively expensive PS3, which may be preferred in more prosperous areas but is not included in your 3% sample!
Given that it seems clear that whilst not the best source of data for short term data, VGChartz should be reasonably accurate post NPD, Chart Track, etc, for monthly, quarterly and yearly figures when it takes the these sources much larger retail samples into account. But forget about day one and first week sales predictions as they could be well and truly out!
Kudos to the author for a very informative dissemination of VG. :)