Apple working on fix for iOS 6.1 bug that's choking Exchange servers

Apple has announced that a fix for an iOS 6.1 bug that's been choking some enterprise Microsoft Exchange servers is on the way. The deceptively minor problem in Exchange calendars triggers massive activity spikes on the Exchange servers, with ballooning transaction logs and CPU utilization. The server can sometimes block iOS 6.1 users.

Yet it's only one of a range of iDevice problems that have developed since the Jan. 28 release of iOS 6.1. Almost at once, those that upgraded to the new firmware began posting to Apple and Microsoft support pages, online forums and tech blogs about fast-dropping battery life, Wi-Fi and sometimes cellular connectivity problems, and Exchange snafus.

"Apple has identified a fix [for the excessive Exchange activity problem] and will make it available in an upcoming software update," according to an Apple support post. "In the meantime, you can avoid this bug by not responding to an exception to a recurring event on your iOS device. If you do experience the symptoms described above, disable then reenable the Exchange calendar on your iOS device...."

Microsoft has posted three temporary work-arounds for afflicted enterprises, one advising 6.1 users to only view their Exchange calendars. Another is simply blocking iOS 6.1 devices from accessing Exchange. It's difficult to determine how widespread the problems are. Network World polled a half dozen enterprises and universities, with varying numbers of iPhones and iPods, and only one was reporting some "inconsistent" issues with Exchange.

"I've read about issues too, but I haven't seen anything yet," says Benjamin Levy, principal, Solutions Consulting, a Los Angeles IT services firm that specializes in deploying Apple products in businesses. "Battery life is a different story. I've heard sporadic reports and noted some shorter run times myself. And I have a consultant friend who is feeling it big time, but I haven't quantified things and mostly I solve it by having a [charging] cable in my car."

"My guess, and it's only a guess is that there may be a new sub-routine that's polling some Exchange servers a little overly [aggressively]," Levy says. "This is not something I see as a big deal, because if it's real it's limited in its impact and I would see it as likely to be addressed quickly."

For end users, the problem with Exchange seems deceptively small. According to Apple's posting, "When you respond to an exception to a recurring calendar event [meaning "to a change to a single instance of a repeating calendar event"] with a Microsoft Exchange account on a device running iOS 6.1, the device may begin to generate excessive communication with Microsoft Exchange Server. You may notice increased network activity or reduced battery life on the iOS device. This extra network activity will be shown in the logs on Exchange Server and it may lead to the server blocking the iOS device."

"This can occur with iOS 6.1 and Microsoft Exchange 2010 SP1 or later, or Microsoft Exchange Online (Office365)," according to Apple.

Apple released iOS 6.1.1 version this week but that has a specific target: It "fixes an issue that could impact cellular performance and reliability for iPhone 4S."

For over two weeks, some users have reported that once updated, their iDevices now quickly drain the battery. Other users complained that their iDevices wouldn't connect over Wi-Fi or, if they did, quickly and repeatedly disconnected. Yet despite the litany of online complaints, it's difficult to tell how widespread these problems are among the total number of 6.1 updaters.

On the enterprise side, the heavy transaction logging on Exchange surfaced quickly for some companies.

"It may be a fluke, but something to watch for...," wrote Bobby Pendino, senior Microsoft Exchange administrator with Zachry Holdings, posting Jan. 31 at Microsoft Technet. One Zachry Apple user upgraded to 6.1 and "immediately after he finished, his phone/iPad started causing excessive logging on the Exchange server....His device caused over 50GB worth of logs for that particular database."

In a later post, Pendino added: "Had another one upgrade their phone and now their phone won't authenticate. Another iPad was updated, and it says it's connected but doesn't retrieve any data. Thanks Apple!"

The Exchange glitch was not a fluke. Based on similar posts, and direct contacts with other IT managers, Tony Redmond, who writes the Windows IT Pro blog, "Tony Redmond's Exchange Unwashed," wrote a week later about instances of "excessive transaction log generation after iOS 6.1 devices are introduced into Exchange 2010 or Exchange 2007 environments. I assume the same is true for Exchange 2013 as the underlying cause is likely to be in Apple's mail app code that calls ActiveSync to synchronize with a user's Exchange mailbox, with some indications being that the problem is once again associated with calendar events."

The Microsoft support posting for Exchange confirms that this problem can and has developed, and that Microsoft and Apple are partnering to sort it out. But IT managers at a half-dozen companies and universities say that, while they're tracking these reports, for the most part these problems have not affected their users or networks.

The only one of this group to notice possible Exchange issues is Boston Scientific Corp., a leading medical device manufacturer based in Boston. It has more than 5,300 corporate iPads and 1,000 personally owned iPads, with some percentage in the process of upgrading to iOS 6.1. "There's been no rash of problems but the Exchange group has issued an advisory around iOS 6.1," says Rich Adduci, the company's CIO. "I don't think we've seen a ton of issues here, but definitely have had some reports, pretty inconsistent, and almost all tied to accepting or declining Outlook meetings using an iOS device," he says.

Others seem to have sidestepped this problem at least by not having large numbers of 6.1 devices or by using email and calendaring alternatives to Exchange.

"We only have one iOS user on 6.1 who is using Active Sync and they have not reported any issues nor are we seeing any on our server," says Nelson Saenz, vice president of IT for Active Interest Media, in El Segundo, Calif. They have just over 120 iPhones and about 30 iPads. "We are standardized on Good Technology for [managing] all our other mobile devices and have not seen any of the unusual activity that is being reported."

Abilene Christian University distributes iPhones to incoming students, nearly 4,400 of them, and to all faculty and staff. There are thousands of iOS devices using the campus-wide Wi-Fi network. So far, ACU isn't seeing any issues, says Arthur Brant, ACU's director of networking services. They've sidestepped potential Exchange issue because the campus email system is based on Google's Gmail service. "Most of our iOS devices leverage the native mail client with their ACU gmail account," Brant says.

A thread at Apple's community forums on the battery life issue is still busy this week as users advise each other. There's also evidence that in some cases the battery drain may be caused by whatever is the calendar-Exchange glitch that triggers the endless transaction activity on Exchange Server.

One enterprise user, billkesh, on Tuesday posted to this thread the memo from his company's IT group about the 6.1 problems.

The IT recommendations for users actually don't solve the calendar-Exchange problems. It's a work-around to avoid the problem, but it comes with a price: "Do not create, accept, decline or modify calendar items on your iOS based devices. Use your device only for viewing your calendar. If you have not already upgraded, do not apply the 6.1 update to your iOS device."

The memo also noted that "Microsoft has identified that the recently released Apple iOS update version 6.1 could degrade performance and availability of email services. Technical details of these findings are published at http://support.microsoft.com/kb/2814847"

That Microsoft support page officially confirms the problem: "Apple and Microsoft are investigating this issue. We will post more information in this article when the information becomes available. Currently, we recommend that you open an Enterprise Support case with Apple, either through an Enterprise agreement or through a "Cross-Platform Integration and Command-Line Interface" case to report and diagnose the behavior in iOS 6.1."

Once the transaction "loop" is triggered, according to billkeller, "you should delete your Exchange account and recreate it in order to stop the looping. This is mentioned in the Microsoft link, but should be emphasized."

User frustration on the Apple forums is almost palpable, especially regarding Wi-Fi connectivity and battery life.

On one thread, RobertJCella, who has an iPhone 5, wrote "prior to 6.1 no issues at all but now wifi just drops repeatedly. Apple, do u not QA test before releasing software?? This is unacceptable."

Another user, zanviexx, with a fourth-generation iPod, wrote "Same problem here. iOS 6.1 *****. I honestly wish I could go back to iOS 4. Everything was great. So much for bug fixes."

Some users found that a "network reset" worked. A poster with the handle dumky2 wrote that "I was able to solve the intermittent wifi connection problem on ipod touch 5 with iOS 6.1 by doing a network reset, as suggested above in the thread. Go to Settings > General > Reset > Reset Network Settings." But others who tried it said it didn't work for them.

Some users this week sounded at the end of their rope. A Venezuelan user with the handle MstrJoey wrote in the same thread on Tuesday that his "wifi button is grayed out too, and updating my iPhone to iOS 6.1.1 didn't work at all. I ran out of MB [Mbytes] so now I can't even connect into the internet. I tried everything, from resetting network connections to restoring the iPhone completely, nothing works....At first wifi just dropped, now its grayed out. Lost all my apps on the restore process :/. I'm using an iPhone 4s, 32GB, bought it less than a year ago."

He concluded: "If someone on Apple is reading this, PLEASE work on it. Im desperate, really, I need my Wifi connection, otherwise i miss out on work stuff."

One user had a novel suggestion for him. "put your iphone 4S off, in the refrigerator for 30 minutes, then connect and wifi work again, I did and it worked," wrote brunosr.

And MstrJoey was desperate enough to try it. "Ok, so here's the deal, putting the iPhone in the fridge for half hour did work, it unlocked the wifi button so now I can turn it off and on," he reported. "Nevertheless, it's still not connecting to any network for more than 30 seconds, and now it doesn't even recognize any around me (and there are lots, trust me)."

Another user, Clairelh83, is so frustrated that she's considering switching from the iPhone when her data contract ends soon. "I am having major issues with this new update, and quite frankly it's very annoying," she posted. "In my household there are 2 iphones and 2 ipads, all of which were working fine before the update. Both iphones and ipads now won't connect to the wifi since the update."

"Why haven't apple come out with a statement or ways to fix the problem as looking through various threads it had been going on for some time," she wrote.

John Cox covers wireless networking and mobile computing for Network World.

Twitter: http://twitter.com/johnwcoxnww

Email: john_cox@nww.com

Read more about anti-malware in Network World's Anti-malware section.