"Me-too storms" in launchpad
I deal with a lot of X.org bugs in launchpad, and I often run into an intriguing phenomenon, the so-called "me-too storm".
Essentially, a me-too storm is a bug report which accumulates a large number of confirmation statements from different people, to the point that it actually hinders progress.
Knowing that a bug affects additional people does have some value to it. I particularly appreciate it when someone of a technical bent is able to reproduce a problem, because I can have confidence that they can test out patches or at least provide insightful technical analysis.
However, past maybe half a dozen comments, the value of an additional confirmation drops off to zero. So, comments in a healthy bug report would shift from confirmatory statements towards comparing data, discussing workarounds, identifying test cases, proposing patches, and so on.
Unfortunately, with a me-too storm, the confirmation statements come in at a faster clip, and their quality often drops further. Commonly, the confirmer will not provide log files or other proof that they do indeed have the bug. This is a particular problem with X because often there are different unrelated bugs that have identical symptoms (examples include X freezes, black screens, performance degradation), and so they might have a different bug; by "me-too'ing" their bug instead of creating their own bug report, it means their issue probably won't be investigated, or it can cause massive confusion or even sidetrack the discussion away from the original bug.
The bug suffers from a bad feedback loop at this point. The more comments a bug has, the more "important" it looks to launchpad's search engine, so it gets suggested to anyone with vaguely similar symptoms. New commenters notice that some old commenters provided data already so they don't bother putting it in.
Indeed, after a while the bug report can start accumulating negative comments, that have a value below zero - they actually detract from the discussion. These range from demanding "is it fixed yet?" questions to rude, inflammatory or borderline threatening comments "fix it now or I go back to windows, you insensitive clod!!!1!" These predictably stir up a variety of 0-value follow on comments, "No, it's not fixed yet, please be patient," "yeah +1 for fixing this soon," "did you try rebooting? it helped me," "plz unsub me from this list, to many emailz," "'doze sucks, get a mac," and so on.
Now, no one would argue that "me-too storm" bugs are not important. Obviously with so many people commenting, there must be some real problem that needs dealt with. It could be a real bug, or a class of bugs with similar symptoms, or a usability issue, or even just poorly set expectations...
However, the stormy nature of these high comment bugs tends to work against themselves, for a few reasons. First, as a developer it's just plain time consuming to read through a gazillion comments. Second, these bugs can be hard to summarize and send upstream, particularly if analysis data is coming from different people (with perhaps differing hardware). Third, if you post a proposed solution, you often get feedback from people having unrelated bugs that leads you to make erroneous conclusions about the fix.
The "me-too storm" is a fascinating phenomenon, but since it hinders bug solution it is interesting to consider ways that this could be prevented or mitigated in launchpad. I'll share my own thoughts in a future post. Meanwhile, I'd love to hear other's ideas.
In my experience not a single problem I have reported over the years has ever been proactively fixed. The only time fixes have ever happened is if upstream accidentally fixes it as part of normal progress and that usually takes years.
I don't know if you're referring to X.org bugs or bugs in other areas of Ubuntu, but I can only talk definitively about X bugs. I like to think X bugs are handled more quickly than "years" and that you're thinking of non-X bugs! If not, I apologize!
When I started at Canonical I looked at the huge pile of X bugs, and the rate of new bugs, and realized that I was completely outclassed. No chance in hell that I could personally fix that many issues.
But we're not individuals, we're a community. I recognized that the role I was hired into was not so much as a bug fixer as much a bug fix *facilitator*. That's my role in the community. In other words, my job is not so much to FIX bugs, but to help you, the bug reporter, to OBTAIN a fix (usually from upstream), by helping you get your bug report into a format that will make it easier to fix by upstream.
I think that most of the time a bug is transforming in a me-too storms is because there are no new on it.
Not necessarily; sadly there are tons and tons of neglected bugs. The vast majority do not turn into me-too storms.
From what I've seen, the preconditions for creating a me-too storm include some or all of the following:
- Bug's title describes a generic or common symptom
- Symptom is either severe or extremely irritating
- Symptom is newish, and being seen by many people
- It's hard to pinpoint what exactly is causing the issue
- Search engines seem to flag this bug report on searches of the symptom
Often when I see a bug report with a generic symptom, I deliberately change it to something more specific, to forestall the bug turning into a me-too storm. For instance, I'll retitle a bug from "X.org crashes randomly" to "X.org crashes in x11_foobar()".
Perhaps someone in the dev team can give some update to cool people. I found myself looking at the bug I submitted and it's very rare that a dev is showing is interest in it even with proposition to tackle it a little bit more.
It's been my experience that providing status updates on a popular bug does very little to prevent it from turning into a me-too storm; indeed, it seems to have the reverse effect - by adding interesting status info to it, it seems to just increase its ranking in search engines, so even more people run into the bug, me-too onto it, and add questions, comments, complaints. So rather than cool people, once a bug has hit the me-too storm level, adding status info is more a kiss of death.
I submit recently some bug about intel and uxa and I don't think that anybody look at it.
I won't claim that bugs in launchpad get ignored for long periods of time, but I can assure you intel/uxa bugs are as far from that class as can be. For several months I've been personally reading and following up on every UXA bug filed against xorg or xserver-xorg-video-intel.
It is important to understand that hundreds of new bugs get filed against Ubuntu's X.org components each week. It can be really challenging to respond to each and every bug personally to get it triaged. And honestly, my time is probably better spent fixing a small number of popular or severe bugs than triaging each individual bug that comes through. But my goal is to get you a response of some sort within a week or so, even if it is just a reply by an automated script.
Currently, there are 2000 bugs open against Ubuntu's X.org components. Following a release we often get 50-100 new bugs filed per week (between releases, this levels off somewhat). Responding to each of these new bugs can be a challenge (and quite time consuming).
Often I feel like I've the devil's choice of either responding personally to 10 bugs and fix none, or fix one bug and ignore 9. Or should I ignore all of them and write a blog post, and hope I can reach 100 people with some specific bit of info to improve the situation for everyone...?
I think that most of the time a bug is transforming in a me-too storms is because there are no new on it. Perhaps someone in the dev team can give some update to cool people. I found myself looking at the bug I submitted and it's very rare that a dev is showing is interest in it even with proposition to tackle it a little bit more. I submit recently some bug about intel and uxa and I don't think that anybody look at it. I can understand why but in this case don't ask for help or be ready to have me-too storms... People are ready to help with their capacity but if you ignore them they are getting frustrated.
Yep, that's another idea that has been floated quite a bit. Other folks have discussed it with more insight than I could, but essentially voting is probably not an ideal mechanism for bug prioritization, for a variety of reasons (including gaming).
Actually the "This bug affects me" functionality ought to serve the same need that a voting system would, if the data from that were more easily visible.
There are actually a LOT of different techniques at the user's disposal for improving attention on a given bug. Most involve some amount of effort (or paying money such as formal support), but there are things that can be done that really aren't that hard or time consuming that can make bugs a lot easier to solve. Maybe I'll plan on doing a blog post later to go into that more.
How about combining one of the useful features of Ubuntu brainstorm and other open source projects (such as wine / bugzilla) and have a voting mechanism rather than the 'this bug affects me' in order to show everyone who views the bug the total number of people it affects.
Two related issues I can see with this is that it would be possible to game the system or that a lot of bugs would end up with a very high number of 'votes'.
Two ideas:
1) Threaded comments. A fruitful discussion would stay in one branch, while 'me toos' or OS flamewars would stay in another.
2) Voting for the quality of comments and hiding comments of low quality - a bit like slashdot.
I can't vouch for your bugs but ones I have reported or experienced generally have zero feedback from Ubuntu. They just sit there, confirmed with all the log files and extra information.
As a user you feel powerless. Ubuntu has asked for your bug reports and relevant information, and so you and others have supplied that. Then nothing happens.
"Me too" storms are ways of users trying to express that the issue matters to them too and hoping that Ubuntu will notice and do something about it. It is about the politest way of doing that.
My suggestion is that once you are paying attention to a bug then post a comment making it clear that you are an "official" Ubuntu person, that you are working it, some sort of expected timeframe (eg days, weeks, months, next release) and that currently no more information is needed.
I do think bug reporting is broken in Ubuntu since there is usually some expectation that something will be done about them. (Given the staffing levels, finances etc the reality is that only a trivial portion will ever be addressed.) In my experience not a single problem I have reported over the years has ever been proactively fixed. The only time fixes have ever happened is if upstream accidentally fixes it as part of normal progress and that usually takes years.
Perhaps functionality could be added to the bug report page which would let you set/change the "Type" of a comment. Comments could default to Discussion, and perhaps certain people could flag more technical or helpful comments as Technical or Workaround. Each type of comment could have color coding so the differences were more apparent. Perhaps someone viewing the bug report could collapse comments of a particular type, so for example a developer could just hide all the "Me too!" comments by clicking the "Collapse Discussion Comments" button. Someone looking for workarounds would hide the technical comments and the discussion comments.
I understand your view as a developer: but from a normal user point of view, a bug report is also the best place to find informations on how to solve it, and to try to work around it while it hasn't caught developer's attention yet.
Oh I certainly agree I'm looking at it from a developer's perspective, but I think it ought to hold true for users as well. Wouldn't you rather have a concise bug to reference, with only a dozen or so comments to read, ending with the best (so far) workaround, or a link to a proposed solution?
Indeed, with some of the really bad me-too storms, you can tell users aren't reading all the comments because you'll see someone post a workaround and a while later someone asks for a workaround, to which someone repeats the workaround, then more discussion... and this cycle goes on and on.
Partly I think this is because launchpad displays only the first 80 comments by default, so if someone hasn't posted a good workaround within the first 80 comments, people may not notice that there are more comments to be read, and instead post their question. I've suggested the launchpad admins change it to display the _newest_ 80 comments instead, but I think they've got some more clever idea in mind, so we'll see how that goes.
You also show an interesting paradox: the more comments a bug has, the less likely will the developer want to get into it, and at the same time the lonelier the users will feel :P Pretty terrible when you see a X bug that prevents you from working normally that has been reported, commented 200 times, and there is absolutely no sign of anyone caring about it ^^
Yeah, I know, that's kind of sad.
Usually with the really bad X bugs, I know the issue well enough I don't need to track all the additional comments, so I often just unsubscribe from it and instead use my personal todo list to track the work, and try to update the bug report from time to time with status, although once commenters start getting surly and uncivil it seems adding status can just get people all the more stirred up. "Why isn't this fixed yet?!!?" "Will you backport the fix to Feisty?" "That's crazy, I think you should fix it some other way." "This is a total showstopper! Delay the release until it's fixed!" "I'm shredding all my Ubuntu CDs and going back to windows!!!"
In some cases I've maintained a "master" bug report where everyone can vent their "me-too's", but I actually do the work on other more focused bugs. Often the me-too storms actually represent a collection of different issues that share a common symptom but have different underlying causes. So the master bug serves as a kind of lightning rod, leaving the real work to be done in the quiet of other bug reports.
In theory this approach would leaves a lot of users less informed than they'd like, but I know the best way we can help users is to focus on FIXING the issue in the most efficient manner we can. Fortunately, many bug reporters tend to be good at sharing info around, particularly effective workarounds so in practice even if I don't update the master bug, someone else usually takes care of it.
Also, make it possible for admins or users request that the launchpad website email particular offenders a list of the rules and suggested actions (same one as above).
What I often have found is that many of these are "drive-by commenters", so it isn't like some spammer that you could shut down. It's a whole lot of newb's who just don't know the rules and whose first (and maybe only) bug comment is a mere "me too". I suppose once they've spent some time with bugs in launchpad they either get their problems sorted out and stop using the bug tracker, or they learn the ropes.
(In fact, this aspect gave me an idea which I'll write more on later.)
After a round of sound bugs that had gone the same path, I posted something similar:
http://ubuntulinuxtipstricks.blogspot.com/2008/11/tis-better-to-dup-than-to-convolute.html
I understand your view as a developer: but from a normal user point of view, a bug report is also the best place to find informations on how to solve it, and to try to work around it while it hasn't caught developer's attention yet.
This is even more true with X as as you say, it can have many different causes. So users end up comparing their configs to find the piece that may be present in all configs and therefore be at the origin of the bug - and thus facilitate the search for a workaround.
Finally, an X bug report is a lot less easy to report. It is usually more random crashes than "clic here, clic here, boum". So people post whatever information they think is relevant - depending on their experience it can be or not.
Anyway I understand your point, but also please keep in mind that while solving a bug takes time, reporting it in great detail, spending time to try to find out how to reproduce it, gather logfiles, is also time consuming - people don't do this to annoy and threaten but to try to help (and get helped, too) :)
You also show an interesting paradox: the more comments a bug has, the less likely will the developer want to get into it, and at the same time the lonelier the users will feel :P Pretty terrible when you see a X bug that prevents you from working normally that has been reported, commented 200 times, and there is absolutely no sign of anyone caring about it ^^
I've been a part of these bug reports, and it's indeed very annoying.
Would it be possible to put some sort of message on to launchpad by the comment field linking to a general set of "rules" that should be followed when posting to a bug report? Such as don't post here unless you're: 1. giving more info, logs, etc.; 2. saying "me too" and offering to make or test patches where needed; 3. etc. Then maybe suggest just using the "subscribe" button if the person wants to follow the bug report and/or clicking the "Me too" link if the bug also affects them. Direct them towards the ubuntuforums if they want to have a lengthy discussion that's not really relevant. Also, make it possible for admins or users request that the launchpad website email particular offenders a list of the rules and suggested actions (same one as above). And finally make it possible to ban people from commenting on a particular bug report or even from Launchpad until they decide to play by the rules (if that's not possible already).
Sorry. Couldn't resist!
Seriously though, how about pointing out the obvious, built-in solution on launchpad:
Educate people to use the "This bug affects me" button in Launchpad.
Admittedly there's a couple of things that could enhance this:
1. Moving it to the right column along with subscribe etc. It's pretty invisible unless you already knew it was there, where it is at the moment.
2. Put something into Launchpad that pops up if the length of a comment is very short (eg
The storm of me-too's is essentially a social conversation, and people are effectively mistaking the bug report for some sort of forum post.
I'd suggest some sort of integration with a 'real' forum; allow a bug maintainer to mark posts to a bug as 'social', at which point they are by default not displayed as part of the bug report, but instead join an automatically-created forum thread named after the bug report itself. Possibly also add a bug report option explicitly titled "me too" to allow people to self-categorise correctly.
That way people will be able to find a place to talk *about* their perceived problems in a more appropriate forum, and the more rigorous technical conversations can be kept in the bug.
Launchpad has a "This bug affects me too" button near the top. Would it be productive to try and encourage people to click that instead of "me-tooing"?
Is the list or number of people who have clicked on that exposed anywhere? I don't think I've seen it...
How about a big green "This bug affects me" button people can click, and bugs can each keep a tally of "affected users" or something, a bit like digg.