What is a Defect Report?
Once in a while in looking through bug reports I find one that makes me go, "Wow, that guy knows how to report bugs!" The report is clear, specific, and easy to drive to a solution. Then I look at the vast number of bug reports that languish in the bug tracker and realize that making good bug reports is a skill we need more people to have, if we're going to succeed at improving Ubuntu's quality.
First, I think we need a better term than "bug report". The way we use launchpad, "bug report" is a broad term which could include everything from a packaging change request to a support request to a plain old complaint. Let's just consider those bug reports which describe a distinct, confirmed breakage of the software in question. In software quality circles the term "Defect Report" is used, and that sounds suitable.
Lots of people have written about how to make a good bug report, but in thinking about it there's one thing above all which defines a good report: The original reporter is not needed for any further work on the bug. In other words, all the data necessary to characterize the bug is there, it's pretty clear what has broken, and the steps to reproduce it are known so we can verify the fix solves it. If there are bug reporting guidelines or troubleshooting procedures for the software package in question, they've been followed.
If you think about how often we have to ask the original reporter to supply more information, try testing some configuration variations, and so on, you realize that a lot of bug reports don't meet this criteria! In fact, if you think about it, these really are what you'd call support requests... where the (implied) request is for support in how to craft a valid defect report. ;-)
"Does strictly following the documentation for reporting a bug result in a usable 'defect report'?"
It will certainly get you a lot closer! This guide encourages filing bugs in a way that triggers apport, a handy tool which attaches various logs and files that are broadly of value in analyzing bugs. It aims at presenting a straightforward process which gets data that is useful *most* of the time, but every bug is a bit unique (if they weren't, they'd be a lot easier to fix!)
Usually you will want to do some additional basic analysis. Look at log files and highlight error messages that look interesting. Compare notes with others who do or do not see this bug. Try installing some different versions.
It can help if you imagine yourself writing a report for a lab experiment. Take good data, make good observations, write clearly, and don't jump too quickly to conclusions but do make well thought out hypotheses.
Great to hear. Unfortunately, to effectively write a good report one must be reasonably well versed in knowing what to look for.
For instance, the log file viewer in Ubuntu displays some 20 files in the left tab, and most of them are incomprehensible to someone who isn't familiar with them. In my 'messages' there are many different stack traces related to resuming from suspend (which actually works fine). If I had a crash and only knew the rudiments of what to search for, I would probably include this too.
If only there were people who are knowledgeable enough to assist in filing a bug report and who can't write code :P
I should add, I've too often seen unfortunate cases where the reporter did use apport to report the bug, but did not actually state what the problem was, or mentioned a problem but was too terse or general.
Sometimes I wonder if people get lured into a false sense of security when filing bugs with apport. They see a notice that the system is collecting information about the problem, and maybe assume that it's way smarter than they are at identifying the problem, so they really don't need to say anything further. That's unfortunate because apport is really supposed intended to augment the written portion of the bug report, not replace the need for it. Maybe one day it will be so smart that it can do this, but we're not there yet.
But I may be wrong. Maybe these reporters wouldn't have made any better of a report without apport anyway.
But what is the point of making good bug reports if so many decent reports go unnoticed by Ubuntu developers? Back when I used Debian, spending time on a bug report was actually worthwhile as I knew that it would be promptly reviewed by the package developer. In Ubuntu, good bug reports can languish for years with no follow up until it is closed because several new versions of the distribution have been released since the bug was originally reported. Either that or some over eager person marks it as a duplicate of a completely unrelated bug.
Yes, the problem is that Ubuntu just gets so many bug reports. I looked at the bug mail queue for Debian's X bugs and saw about 70 bug mails. Over the same period the mail queue for Ubuntu's X bug tracker saw approximately 2000 bug mails.
At the same time, from what I've observed, the average Debian bug report tends to be of much higher quality - closer to what I describe as a defect report. Is this because Debian has better skilled users? Or a higher threshold for accepting bug reports? Or a better bug tracker? Or fewer users? Or am I just looking at the wrong bug reports? I don't know. But it exacerbates to the problem because the lower quality bug reports require far more manpower to get fixed than well-reported bugs. So this means more developer effort required per bug, which translates into fewer bugs fixed.
Ubuntu bug reporters get angry that their bug reports are getting ignored, as if there's a bunch of Ubuntu developers ignoring bug reports spending their days on the golf course. In truth, the Ubuntu developers often are working into the wee hours of the night hammering on bug reports.
Ultimately, this gets back to my original post. We don't need *more* bug reports, we need *better* bug reports. Otherwise we're wasting developer energies on trying to help improve the low quality reports instead of giving attention to decent reports.
Does strictly following the documentation for reporting a bug result in a usable 'defect report'?
On some occasions I have been asked to attach a backtrace. On other occasions it has not been necessary. Perhaps it would be a good idea to add some more information to that guide?