Ask someone how the quality of a game is progressing, and there's a good chance the conversation will eventually turn to bug counts.
How many bugs are open? How many have been fixed? How many critical issues remain? Is the backlog going up or down?
These are useful questions. They're also relatively easy to answer, which probably explains why they've become such familiar measures of quality across game development.
But I've increasingly questioned how much they actually tell us.
A game with 500 open bugs isn't necessarily in worse shape than one with 100. A declining bug count doesn't automatically mean quality is improving. And a week with very few new issues might be a sign of progress, or simply that the team hasn't been testing the right things.
Bug counts tell us something about the work being reported. They don't necessarily tell us how good the game is.
I think there's an opportunity to do better.
The problem with measuring what we can count
Game development is full of information, but not all of it is equally useful.
Bug tracking systems are particularly good at producing numbers. We can count issues by priority, severity, status, component, build or assignee. We can track how quickly they're resolved and how the backlog changes over time.
Those measures are valuable for managing work. The problem comes when we start treating them as direct indicators of the game's overall quality.
Consider two development teams.
One has reported 300 issues over the past month. The other has reported 80.
Without additional context, it's almost impossible to say which project is healthier.
Perhaps the first team has been conducting intensive exploratory testing across a newly implemented feature set, while the second has been focused on a relatively stable area of the game.
Perhaps the first has excellent reporting discipline, while the second is struggling with incomplete submissions or limited coverage.
Or perhaps one project genuinely has more problems than the other.
The numbers alone can't tell us.
And yet, they're often some of the first things we look at when evaluating quality.
Quality needs context
I've come to think that one of the most important questions we can ask isn't simply how many issues we've found, but how much confidence we have in what we've tested.
That confidence depends on several factors.
What parts of the game have been exercised? Which platforms, devices and configurations have been covered? How recently were those areas tested? What kinds of issues were discovered, and how serious are they?
A game might have very few known problems on PC, but that doesn't tell us much about its quality on console if the latest build has barely been tested there.
Likewise, a feature might have passed its functional tests while still presenting performance problems, usability frustrations or inconsistent behaviour under different conditions.
These distinctions matter.
A meaningful assessment of game quality needs to consider both the problems we've identified and the strength of the testing evidence supporting our conclusions.
The absence of reported problems isn't the same as evidence that a game is working well.
Looking for patterns, not just totals
Another limitation of bug counts is that they tend to focus our attention on individual issues or aggregated totals, rather than the patterns emerging between them.
Imagine a project where the total number of open bugs is steadily decreasing.
On the surface, that's encouraging.
But what if the latest builds are also introducing an increasing number of regressions? What if performance issues are becoming more common on a particular hardware configuration? What if one major component is generating repeated problems despite receiving substantial development attention?
Those are important signals that might be difficult to spot in a traditional backlog report.
I think there's real value in looking at how quality changes across builds, environments and areas of a game.
Are fixes remaining stable? Are new changes introducing unexpected problems? Are issues clustering around certain systems? Is testing uncovering fewer problems because the game is improving, or because coverage has shifted?
The answers can help teams understand where risk is developing, rather than simply where work is accumulating.
This is where I believe quality reporting can become much more useful.
Not just describing what has already happened, but helping teams decide where to focus next.
We also need to understand the testing process
There's another aspect of quality measurement that I think deserves more attention: the effectiveness of the process producing the information.
A studio might have comprehensive bug dashboards, but if the underlying testing activity isn't visible, those dashboards only tell part of the story.
How much testing has actually taken place? Where has the team spent its time? How much effort is going into reproducing existing issues rather than discovering new ones?
Are reports reaching developers with the information they need? Are issues being repeatedly returned for clarification? Are verification queues growing because the team doesn't have enough capacity?
These aren't necessarily direct measures of game quality, but they influence how confidently a studio can assess and improve it.
A team struggling with incomplete bug reports may be generating plenty of activity without producing the same level of useful information.
Similarly, a project with insufficient testing coverage might appear stable simply because problems haven't yet been discovered.
Understanding the health of the testing process helps put the resulting quality data into perspective.
From reporting activity to supporting decisions
For me, the most important shift is moving away from asking what numbers we can report and towards asking what decisions those numbers can help us make.
A producer preparing for a milestone needs to understand whether the game is progressing towards an acceptable level of risk.
A QA lead planning the next testing cycle needs to know where coverage is weakest and which areas deserve attention.
A development lead needs visibility into recurring failures, unstable components and issues that are creating the greatest disruption.
These people don't all need the same dashboard.
But they do need a shared understanding of what's happening across the project, supported by information they can trust.
A more useful picture of game quality might combine bug severity and trends with testing coverage, regression frequency, performance stability, issue resolution patterns and outstanding risks.
None of these measures is perfect in isolation.
Together, however, they can provide a much more informative picture than a headline bug count.
And importantly, they can help teams decide what to do next.
Can game quality ever be objective?
There's an interesting challenge here.
Games aren't purely functional software. They're interactive experiences, and many of the things that make a game great are inherently subjective.
A game can be technically stable but frustrating to play. It can perform consistently while lacking the polish players expect. It can pass functional tests and still have fundamental problems with balance, pacing or usability.
I don't believe we'll ever reduce the entire quality of a game to a single objective measure, nor do I think we should try.
But that doesn't mean we can't make significant parts of quality management more evidence-based.
We can be more systematic about understanding coverage. We can track whether fixes remain stable. We can identify recurring technical problems and measure how effectively issues move through the development process.
We can also combine quantitative information with the qualitative observations of experienced testers, designers and developers.
The goal shouldn't be to replace human judgement with a number.
It should be to give people better evidence on which to base that judgement.
A more connected approach to game quality
This thinking has become an important part of the ambition behind TopMod.
We started by looking at how to improve the bug lifecycle, but the more we've explored the problem, the clearer it's become that individual bugs represent only one part of a much larger picture.
Testing sessions generate evidence. Bug reports capture discoveries. Review workflows tell us something about reporting quality. Development activity shows how issues are being resolved. Team operations reveal where resources and attention are being directed.
When that information exists across disconnected systems, it's difficult to understand how the different parts influence each other.
By connecting them, I believe we can start building a more useful understanding of quality throughout development.
Not through a magic score that claims to tell studios whether their game is good, but through clearer information about where quality is improving, where uncertainty remains and where teams might need to intervene.
It's an approach we're continuing to develop, and there's a lot still to explore.
Better questions lead to better decisions
Bug counts aren't going anywhere, and they shouldn't. They're a useful part of managing software development and quality assurance.
But I think we need to be more careful about what we expect them to tell us.
The number of reported issues is not the same as the number of problems that exist. A shrinking backlog doesn't guarantee a healthier game. And a project with very few open bugs isn't necessarily ready to ship.
Understanding quality requires context, evidence and a willingness to look beyond the most convenient metrics.
For me, the opportunity isn't about creating increasingly complicated dashboards or finding one perfect quality score.
It's about helping teams answer better questions.
What do we know about the current state of the game? What haven't we tested sufficiently? Where are problems emerging? How confident are we in the fixes we've made? And where should we focus our attention next?
If we can answer those questions more consistently, we can make game quality a more informed and manageable part of development.
And ultimately, give studios greater confidence in the games they're building.