We test stuff internally. Eventually it’s released as a final product. But in between there, we do a thing that might be called a Beta or Private Test.
Why are we doing that? With the small group of people testing and developing it right now, we already have a large cache of bug fixes to work on. Outside feedback is crucial as well; we need feedback on the design now so we don’t spend a bunch of time polishing a potentially wrong design into perfection. That’s why we plan to start giving some folks access to OmniFocus 2 early.
Phase 1: Internal Test
For a few months now, we’ve been using OmniFocus ourselves. This has helped us shake out the most obvious and painful bugs. I hit several crashes (and wasn’t even able to report them fully, because our Crash Catcher wasn’t working properly either). I lost data a few times. And I had to flip back to OmniFocus 1 a few times when a feature was just completely missing. I know some of you wish you had access to OmniFocus 2 a month ago, but we don’t think it’s helpful to anyone if we ask you to put up with all of that. We got OmniFocus working well enough to show off at The Debut. Then we put in a lot more work to get it sturdy enough for some of you to start using it.
Phase 2: Private Test
There are now over 17,000 of you who have expressed interest in helping us test OmniFocus. And we love that you’re so eager to help us. But if we ask all 17,000 of you to take the app out for a spin all at once, we’ll suddenly have thousands of emails all reporting the same issue. So we’re going to do a phased roll-out. We’ll give access to a few (hundred) people at first. Once we’ve fixed the big issues they’ve found, and our Support Humans are caught up on emails, we’ll add some more folks to the testing team. This will help us get the feedback we need about the app while still providing the level of support we promise for our shipping applications. We want to know about crashes and other errors, of course. But we also hope you’ll tell us what’s confusing, annoying, or tedious – these emotions suggest that perhaps there’s a usability bug we can fix or a workflow we can simplify.
If you want to get in on the Private Test phase, make sure you sign up, and double-check for a confirmation email.
Phase 3: Public Test
When we’re close to shipping, we’ll fling wide the doors, and everyone can download OmniFocus 2 and take it for a spin. At that point, the functionality of the app will be all there, and we’ll be putting most of our energy into getting localizations and setting up the machinery for the formal release to both our store and the Mac App Store. However, this is also a last chance for you to let us know about any bugs that might really really need fixing before we ship.
How to help us test
So, once you get that exciting email saying “You can now download OmniFocus 2 test builds”, now what? Take a deep breath and think about whether this is really something you want to do right now. The builds will be generated and posted automatically, so you might run a particular build of OmniFocus before anyone at Omni does. There is a small but real chance that the build will destroy your OmniFocus data or worse. Make sure you’ve got backups of your computer data. And if you’ve got a really big assignment due on a Tuesday, maybe wait until Wednesday before taking a test build out for a spin. If all these warnings don’t scare you away, follow the instructions in the email to download and install the app.
We realize it’s probably futile to ask that you not talk about the app at all in this phase. So we will simply ask that if you do talk about or share screenshots of OmniFocus 2, please clearly indicate that this is not a final release. A great place to discuss the app is in the OmniFocus 2 Forum. (And please do NOT share the application itself.)
So you downloaded the app, and started using it, and something went horribly wrong. Now what? First check if this is on the “What’s NOT Ready” section of the Release Notes (available in the Help menu). For example, right now Review doesn’t look at all like the mock-up we showed at the Debut. Sending us an email telling us it’s unfinished won’t help, because we already know it’s a problem.
Otherwise, please tell us about it. You might be the first person to encounter it, or the first person to give us a critical clue we need to detect the root cause of the problem. Bugs are often best reported via email, because it is so frequently useful to include specific data about the issue. (We will still be answering the phones—during business hours—and twitter as well.)
If possible when reporting a bug please do the following:
- If this is a new issue, please start a new email thread. Replying to an old email thread to tell us about a new bug can make it more difficult for our Support Humans to respond efficiently. (If you’re continuing a conversation about an ongoing issue, it’s fine to reply to the email conversation already in progress.)
- Please mention what revision of the app you’re running. (Send Feedback will include this automatically in the subject line.)
- Tell us the story–what were you doing right before this happened, did you recently upgrade your OS, does this only happen at the coffee shop?
- Include a screenshot. A picture can be “worth a thousand words” even in a bug report.
- If there are error messages, a console log will often provide more detailed errors than what pops up in the alert box.
- If the app becomes unresponsive, including a sample can help us figure out what the app thinks it’s busy doing while it “beachballs” or “hangs”.
