Xoatlicue

In a recent post, dbaron suggested that people inappropriately judge Mozilla by, for example, the age of the bugs that one can see. I think he is right, but I think there is a reason for this. It is usually completely impossible to look at a bug and figure out whether it will be fixed or not. There are long-standing reasons for the use of nobody as an assignee, but this can be un-helpful.

Mozilla code development is socially driven. This may seem obvious, but the effects of this are not necessarily obvious, especially to someone in coming from commercial software. Look at the recent revisions of the module ownership system and you can see how tenuous the ownership of code can be. Nobody has to care about any bug. Or, to say it another way, only 'nobody' actually has to care about the vast majority of bugs.

It is very hard to look at a bug and see who actually cares about it. No. I will not gripe about bugzilla. Really. But I will just say that anything in a bug is not what it says, but how that is looked at by which group does the work. Priority, Confirmed, and many other terms mean different things to different groups of people. And everyone seems to be ok with that.

I have filed many bugs thinking that Firefox should just 'do the right thing' and that people would want to know. Is still have this idealism at times, but I recognize the other viewpoint as being valid. It may be true that one should not fix something that does not need to be fixed.

And as dbaron has pointed out to me many times, one is always welcome to file a patch. And, actually, that turns out to be the key to many things. I used to think there was value in asking about a bug to understand it. I used to think there was value in discussing a solution before creating a patch. But, as a practical matter, the number of people willing to talk about making changes is vastly more huge than the number of people who will make changes. So, considered questions almost always get silence while patches that are crap get responses. If I have 10+ years of database experience and ask a question about sqlite use by Places, and a 12-year-old asks a question with a patch he came up with in five minutes, who will get a response? Who will be more likely to get something started on a bug? It seems obvious to me now.

So when in doubt, submit a patch. Any patch.

So, this leaves a question. How does Mozilla want to manage bugs submitted by or read by or tracked by non-developers? It seems clear that bugzilla is really best only for developers. Asking users to look at bugzilla and make sense of what they see is, I would suggest, not realistic. Maybe the idea behind Hendrix needs to be developed further. A lot of what is in bugzilla looks like graffiti in a train station. So maybe something is needed manages that graffiti and relates it to actual bugs that people may actually work on. But then, the people who do the work do not mind looking at the graffiti and just working on what they talk about. They seem to know which ones are actually significant, so why fix it if it's not broken?

I had something happen yesterday which makes me think there might be hope. For my brain and how I sometimes fail to keep track of things. I was asked, in a bug I filed a while back, if the failure was reproducible.

I do not why it was not reproducible earlier. I hardly ever find out why people do not see the bugs I see. I filed the bug and it got the usual responses: "works for me", "what are the running? are you sure?", "i see something weird", "why are we checking this?", "can you attach more information?", "did you configure the thingamagig to build the whatamacallit?", "can you attach a lot more information?". It just gets really ... exhausting. Actually, this particular bug was not as bad as some others. And it is nice when someone can actually reproduce something I filed.

But right now, if you look in my ~/mo directory, you see 3 different builds of the browser, a half-dozen nightlies (some a few months old), a couple of calendar builds, a few xulrunners, an nspr build (what the heck was I doing with that?), a re-usable profile directory, and other random stuff. So, do I still see my bug? I look in the directory and and it is, again, exhausting, even to just think about.

Then I saw something.

tests_20080106_160400/

You know, I was talking about a re-usable test product, wasn't I? Oh, yeah.

% cd tests_20080106_160400
% ls
dist/ runReftests* testing/ xpcshell-simple/
%
% ./runReftests ~/mo/browser_20080327_235255/dist/MinefieldDebug.app/Contents/MacOS/firefox ~/mo/debug.mozilla_20080106_173138_PST

Holy test failures, Batman! It worked! I just gave it a fairly recent build and that re-usable profile directory I mentioned and there it is. Lots of failures.

Now, I am thinking again of that standalone test product I was mentioning. Once again, the buzzing in my head is beginning. There is apparently a new way to build a product out of the Mozilla build system. Create a directory, put some stuff in it, and voila! A new product. Now, to find the documentation for this wonderful new capability. Ok, I am sure that people are busy. Is it fair to ask them to document things? Should I ask a question in the newsgroups? I will just get that response that I really hate so much: "what exactly are you trying to do?". I think I was trying to ask a question. Again, it can just be exhausting. I even created an MDC page. Seeing a place for something like this to to be documented, someone who knows what is going on might, you know, document something. Not even a nibble on that yet. O well. You can lead them to water....

And, finally, I have to do homework today. My Stats and PDE homework is not doing itself. And why don't I get more Mozilla stuff done?

Are you participating in Earth Hour? I am, apparently. My wife says that we can and should turn off all lights in the house. We can then turn off all electrical devices (even my Server machine!) and, further, she is convinced we will survive and may perhaps even escape psychological damage.

I am dubious.

If you do not hear back from me, you know why. Good luck.

What does one need to do to help test Firefox. It turns out that to help very much, it takes a fair amount of effort. Just downloading the latest browser and "doing stuff" with it does provide testing, but it is not very complete. That kind of testing tends to be shallow at best.

Firefox has great automated testing tools built into it, and so one can build a copy of Firefox using '--enable-tests' as a configuration flag. One gets a bunch of standalone executable tests, some xpcshell tests, the mochitest functionality and reftest. There is a lot there to work with.

However, building Firefox can be a pretty hefty requirement. Is it really necessary? One benefit of automated testing is that the more of it that gets done, the more valuable it can be. There are lots of OS versions and machine configurations to worry about. MoCo can either try to have a copy of all those OSes and all those machines, or it can allow the community to meaningfully assist with testing without jumping through all the extra hoops needed to be able to build the browser. After all, the software required to just run tests is much simpler.

In pursuit of this, I am creating a standalone product which can be a test vehicle for testing Firefox. After all, if Firefox is changing every day, should one have to refetch all those tests? One would hope that when Firefox is changing a lot, the tests are not changing, so that the tests can provide confidence for the changes being made.

So far, I have a standalone product that I can use, on my Mac OS X system, to run reftests. There are issues with some of the other technologies that I will try to address. For now, I have taken the last 4 nightly builds and used the same test product on them and got the same result.

- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/bidi/bidi-000.html
- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/bidi/bidi-001.html
- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/bidi/bidi-001-j.html
- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/bidi/mixedChartype-02.html
- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/bidi/mixedChartype-02-j.html
- REFTEST UNEXPECTED FAIL: file:///.../testing/reftest/layout/line-breaking/currency-2.html

There may already be bugs on these test failures. It is hard to tell. I cannot seem to find the obvious way to locate bugs filed against automated tests.

The test product is based on the contents of the "_tests" directory as built with the '--enable-tests' flag, with some minor changes. Also, the contents of the directory are actually files, not links back into a source tree. The basic directory structure right now looks as so:

tests\
    Minefield.app/Contents/MacOS/chrome/reftest.jar
    Minefield.app/Contents/MacOS/chrome/reftest.manifest
    Minefield.app/Contents/MacOS/components/reftest-cmdline.js
    testing\
        mochitest\
        reftest\
    xpcshell-simple\

There are issues, already, that make it a little difficult to just grab this test product and put it up in a shareable location for other Mac OS X users. For one thing, script bits need to be put inside the application bundle of the app being tested. For reftest, it is only three files that need to be put inside the ".app/Contents" directory, but it is awkward to do manually.

Looking at mochitests, I can already see problems. The tests all assume that they can do something like "../../../dist/bin/testExecutable". There is no reason they need to assume that, but they all do. Does the xpcshell executable that runs the tests need to be the same build as the Firefox browser itself? I would say no, but that is the way it has always been done. Until issues like these are teased apart, there will have to be lots of workarounds.

If anyone has any suggestions, I am open to them. If MoCo wants to make automated testing available to a much larger group of people, without massively increasing their app download times, it seems that a standalone test product would be one way to do it.

Seeing the release of Firefox 2.0.0.10, followed by the rapid release of 2.0.0.11, made me wonder about how regression testing is going at Mozilla. Of course, there are no simple answers to be found for any of the issues. We all want less bugs and for bugs to get fixed more quickly. Nor am I feeling that there is some deep and dark hidden flaw that is coming to the fore here. Things could be better. No shock there. Things could be worse. Lots of people care enough to make sure they are not. And MoCo moves from one day to the next. We all love the products that come out of Mozilla, and as with anything one cares about, we are frustrated about some things. Such is life.

But with every new test harness being talked about and with every new set of tests being brought in, the same questions and issues seem to come up for me. Every new test harness I see being developed for MoCo technologies bring me hope, but it is rarely clear why the last one is not being talked about. The same issues seem to be stumbled over. What happened to the lessons learned in the past? One problem with wiki-based communities is that people talk a lot about what they are doing. People tend not to talk about what they are not doing. Search for any kind of solution in Mozilla's information space, in any problem area, and one finds at least four or five different efforts. Only one or two may be current, but nothing gets written about why the others are not moving forward, or why they failed. It may just be against our natures to talk about things that did not work. One just has to puzzle out the things that are real from the things that just look real.

In testing, it often seems to me that one needs to be clear about what kind of tests one wants, who can create, develop, or maintain what kinds of tests, and why different kinds of tests should or should not be run. If one is not clear about these things, tests get developed which do not get run, tests get run but they do not exercise the right things, or their results get ignored, and lots of people might be working hard to no good effect. The result is that there is no joy.

Questions:

(How hard is it to/Who can) develop a new test?
(How hard is it to/Who can) modify an existing test?
(How hard is it to/Who can) identify a feature or fault for which a test should be created?
(How hard is it to/Who can) evaluate the benefit of creating a test of the effectiveness of running a particular test over time?
(How hard is it to/Who can) run a test?
(How hard is it to/Who can) see that a test has failed?
(How hard is it to/Who can) identify the cause of a test failure, or interpret the failure?

I would suggest that each of these questions defines an axis in the "problem space", the space of possibilities to in which to locate problems and suggest solutions about Mozilla products. There are risks to be managed in the Firefox source base, and possibilities to be enabled. The cost of the risk to be averted and the possible improvement to the product that are enabled have to be measured against the costs of testing. I think that answers to these questions can be used to categorize various quality efforts and give a way to judge whether testing is doing what it should do and how much should or should not be spent on it.

I was going to take a stab at answering the one or more of questions above, but I will post this as a start. Answers can come later, especially since I do not know all of the answers. Nor am I sure I know all the questions. And if anyone wants to suggest either questions or answers, please do. And if anyone can think of a way to draw a representation of a, say, 12-dimensional space, so I can draw pictures based on something like the questions above, let me know that too.

Thanks. More to come.

Well, I am almost done with an extension I am developing for a third-party company. Wow! Was this ever harder than I thought it would be....

I had some slightly complicated things in the extension. 2.5K lines of javascript, a custom XPCOM component, a status bar icon, a toolbar popup button, a sidebar, and a partridge in a pear tree thrown in for good measure. The app was crashing on exit. Actually only FF 2.* was crashing. FF 3.* was fine.

So, I am doing a bunch of different kinds of allocations and there are listeners and so on. I figured I was doing something wrong with the memory. Much staring at the code. Much re-reading of dbaron's docs on this stuff. Much more staring at code....

Turned out it was this:

   <toolbarbutton id="id1" type="menu" popup="id2">
       <menupopup id="id2" />
   </toolbarbutton>

I thought that all this memory stuff might have a problem, and it turns out to be something bone simple! It is amazing what taking a machete to your code can show you. Along the way, an amusing thing happened. I took the image attribute off the toolbarbutton and I got this:



And someone said this was ok! I might try to guess at the logic, but I cannot make myself do it. Obviously, some people are very immersed in how XUL gets rendered. I hope I can learn enough to understand it. I hope I never forget that its contortions are, after all, contortions.

Next things to do now that this monkey is almost off my back:

- do an extension for rule-based qualification of XUL, because you need something to tell you that you can have A, B, and C together, but C makes B have no effect, and C without A does something totally different. Look at a given XUL reference. It inherits all its parent's attributes. Now, how many have any effect? How many break everything? There has got to be a way to get information about this.

- figure out something to help me search bugzilla. I looked for bugs on this many times and did not find it. Maybe it is because the relevant bug was marked for Windows. Maybe it was because the title mentioned <menupopup> and not <toolbarbutton>. Maybe it was because the title said it happened on window close and I was searching for "quit". I do not know. I just know I truly hate trying to find things in bugzilla. But when I suggest changes to bugzilla, they seem to hate that even more. I have no idea why. So, I just obviously need to construct tools to help me get around this thing. There has got to be a better way to find bugs than creating new bugs and letting gavin find the dupes. For one thing, this is not fair to gavin, or neil, or reed, or whoever it is.

- and some other things I am sure I have forgotten. We'll see.

Someone posted a while back about this icon. I thought they said they were not sure what caused it. I got one of these and saw what it was.



This is what happens when you have an app that is built only for Intel, and you are looking at it on a PPC system. It is not going to run. If it is built "fat", then it has both an Intel executable and a PPC executable hidden inside its bundle directory and it will run on either.

This particular app was the Mac OS X 10.4.x app put up by Chris Double for his SVG Video Demo.

Just FYI.