QA Lessons From iOS 6.1.2

Shortly after the release of Apple’s iOS 6.1, reports appeared about issues with iOS Mail and Microsoft Exchange mail servers. They said iOS devices were generating excessive interactions with the server, resulting in huge log files, and there was talk of reduced battery life on the iOS device. In February, Apple released iOS 6.1.2 to address the issue.

iOS 6It turned out the excess Exchange activity only occurred after the user accepted an exception to a recurring calendar event. It seems that Apple tested the new mail app with an Exchange server and verified that it worked. Someone may have even accepted a calendar exception during QA, though I wouldn’t be surprised if this particular capability wasn’t explicitly tested. If it were, the reduced battery life would probably have been noticed.

When it comes to its internal processes, Apple is famously secretive. In this particular situation, though, it isn’t too hard to come up with a likely scenario: Apple didn’t check for problems on the Exchange server itself. And I can see why. Exchange, after all, isn’t one of its products, Apple probably has limited expertise in maintaining Exchange servers, and it may not have had the knowledge to include the Exchange server in its test plan.

During beta testing, the issue could have been discovered as well, but again I can see how it might have slipped through. Accepting this kind of calendar exception doesn’t occur often. Apple’s testing was focused on the parts that were updated, including its Mail app. In addition, the testers probably had only user access to the Exchange server. The problem showed up in the server log files that users normally don’t see.

We’ll probably never know whether Apple in fact checked the server logs as part of its testing. Even if it did, it’s rare that someone accepts an invitation to a calendar exception. It’s plausible that verifying the server logs was part of the test plan, but not in this particular context.

There are several lessons in all this.

  • First, when dealing with complex, interconnected systems, it’s important that testing verify both ends of each communication.
  • Second, all odd results reported by users should be investigated. A single person may have complained about reduced battery life, for example, and I can see how this was dismissed as an anomaly, but in the end it turned out not to be.
  • Finally, there must have been someone at Apple who knew that the code handling  this specific event was changed in the iOS 6.1 release. They should have ensured that accepting an invitation to a recurring calendar exception was part of the test plan.

As developers, we’re the ones who know best what’s changed in each release. We need to communicate what we’re changing so we can be sure these features can be thoroughly tested.

5 Responses to “QA Lessons From iOS 6.1.2”

  1. Clearly you are not a tester, and show signs of being an amateur programmer.

    First, to expect a tester to find some obscure issue such as this is unreasonable. This would be like finding out that when taking aspirin with tainted water and a burrito, you may get a headache. Is it reasonable to expect that this situation would occur? Maybe or maybe not, but there are better things to test first. If you hold up release until every obscure test can be verified, you’d never get anything out to market.

    Furthermore, your conclusions are out of left field. verifying both ends of a complex interaction is fine, but that’s not a conclusion from this issue, at least not as presented. A better conclusion would be to examine likely interactions and spot-check various parts of it to see what happens.

    And “all odd results reported by users should be investigated”? Again, this is typically a good thing, but not something that can be determined until it goes out to market so why is that a “lesson learned”? There was no indication that this was ignored at all.

    “there must have been someone at Apple who knew…” How is that a conclusion from the report as presented? How could this be a lesson learned.

    If this is the kind of thought process of people at your company, please let me know so I can avoid your infomercials and products since they will be presenting things that are irrelevant and try to promote them as evidence that your products are great.

    Sorry to be harsh, but this article took so many things in the wrong direction.

    • ROBS

      I am curious to see if you could shed some light on the QA process as I may be doing this for my company with our current app. Thanks if you can, and thank you for your comment it was most insightful.