FRAMINGHAM (09/26/2003) - To call the challenge facing Mark Flieger a scalability problem would be an understatement. When he came aboard as vice president of information services at Occupational Health + Rehabilitation Inc. three years ago, the Hingham, Massachusetts-based health care provider had nearly US$60 million in revenue, 35 offices from Maine to Missouri, 550 employees, a double-digit growth rate and a centralized software application running the core business built on top of dBase and MS-DOS.
OH+R owns and manages occupational therapy clinics that treat many patients with on-the-job injuries. Its information system must deal with patients as well as with the employers that are the payees under state workersí compensation laws. As OH+R expanded to more states, the company had stretched its software, Systoc, from Occupational Health Research Inc. in Skowhegan, Maine, to its limits.
OH+R had never upgraded its copy of Systoc to a newer, more network-friendly client/server version. As a result, the dBase version performed poorly over the wide area network (WAN) links connecting OH+R's offices. To work around that, OH+R added Citrix Systems Inc. MetaFrame servers, which ran user application sessions locally and sent only screen images and keystrokes back and forth to end users over the WAN. But Systoc wasn't designed to scale beyond a single state, so the business ended up creating and maintaining 13 separate databases as it grew.
"We've had all kinds of business issues around maintaining 13 data stores," Flieger says, not the least of which has been the need to maintain multiple sets of reports, more than 1,000 of them, due to duplication across databases. "Enterprisewide reporting was not possible," says chief executive officer John Garabino.
Billing couldn't run reports efficiently across the databases, and differing database schemas and payer-identification naming conventions made even manual reporting difficult, says Patti Walkover, vice president for contracting and reimbursement. And those reports that the staff could run bogged down system performance, says Flieger. "It doesnít take too many reports to slow down a transaction system, and people in the clinics would really feel that," he says.
Managing the system for high availability was difficult. "Our data center was more like a suite in an office building," Flieger says. "All the inherent maintenance of the back end was problematic." But since every office depended on the system for patient scheduling and billing, availability was a priority.
Today, OH+R has built its first high-availability data center in nearby Norwell, Massachusetts, and it's slowly consolidating its offices onto a new client/server version of Systoc. The new software includes a clustered SQL Server 2000 back end with a separate SQL Server data mart designed to speed financial reporting and move report generation off the production system. The client software, written in Visual Basic, is served up to Wyse 8235 Windows thin-client terminals by way of a MetaFrame XPe server farm.
Remote offices connect through frame relay over Asynchronous Transfer Mode WAN links, while remote workers equipped with laptops can access the system over the Internet through an integrated MetaFrame Secure Sockets Layer virtual private network feature called Secure Access Manager. Two years into the project, Flieger has offices in three states cut over to the new system. He hopes to have the rest of the offices switched over by year's end.
First Steps
Flieger started the process by reviewing Systoc. Occupational Health Research had a newer client/server version, but he was skeptical that the vendor, a small company used to dealing with small clinics, could support OH+Rís growth plans. But only one other small vendor had the capabilities he needed. "There are some unique business rules that cause us not to be able to use mainstream practice management software," he says. Users were also frustrated with Systoc, which they saw as slow and difficult to use. Flieger decided that better user training and a system upgrade could solve most of the problems, but scalability remained a concern. "Nobody had ever taken Systoc to the level where we had taken it or where we were planning to take it," says Garabino. OH+R would need to do its own scalability testing.
Infrastructure Basics
Next, Flieger turned to the infrastructure. He presented his case to Garabino for building a high-availability data center and began planning a highly redundant systems architecture that included uninterruptible power supplies and backup generators. He also wanted to keep the thin-client system in order to minimize field-support demands on a staff of seven.
"We're trying hard to keep servers out of our edge sites because of remote management issues," Flieger says. And he wanted to keep WAN costs, already more than $500,000 per year, under control.
Paul Venezia, senior engineer at Keene, New Hampshire-based IT service provider IniNet Inc., had something that seemed to fit: a Compaq-commissioned design for a high-availability application service provider reference architecture based on Citrixís MetaFrame XPe. IniNet agreed to adapt the design to OH+R's needs.
The new design stresses availability and addresses a potential weakness of using Windows terminals. "The loss of the data center means people are sitting on their thumbs. It brings added pressure on the core network to perform and maintain as high a level of availability as possible," Venezia says. Flieger shared this concern. "The con to thin client is that you put all your eggs in your network basket," he says. "If that network goes down, you're hosed."
The new architecture includes a 12-server front-end MetaFrame server rack, a back-end SQL Server and SAN rack, and a management server rack, each on its own virtual LAN segment. The rack servers, 1U (1.75-in.) dual-processor Hewlett-Packard ProLiant DL360s and DL760s, have redundant network adapters for each segment to which they connect, and servers on all three VLANs have six adapters. To minimize wiring problems, Venezia mounted dual Cisco Catalyst 3550 switches atop each rack, which allows OH+R to run just two fiber cables back to the core switches.
Although MetaFrame kept bandwidth requirements per session down to about 20Kbit/sec., Venezia added PacketShaper application-layer traffic-optimization appliances from Packeteer Inc. in Cupertino, California, to give Systoc traffic priority. The appliances give administrators more control of available bandwidth, but they required publishing individual applications as icons within MetaFrame's NFuse portal interface rather than publishing a single, more user-friendly Windows desktop. The PacketShaper appliances also control Web browsing bandwidth and smooth out report-printing traffic by giving those packet types a lower priority. The savings add up, Flieger says. "Adding a single-channel 64Kbit line is about $40,000. Weíre running our network probably at about half what others are," he says.
To address database scalability and performance concerns, the design includes direct-attached HP (Hewlett-Packard) StorageWorks MSA 1000 Fibre Channel storage arrays, one for each SQL Server system. Flieger also accepted Veneziaís recommendation to put the SQL Server database transaction log onto a 16GB RAM disk, called QikData X3, from Platypus Technology Inc. in Billerica, Massachusetts.
"That shifts the bottleneck out to the processor and memory and gets it off the disk I/O," says Flieger, who acknowledges that he was initially concerned about reliability and support of the 3-year-old vendor. But the X3's ability to partition memory banks into two mirrored 8GB virtual disks and to automatically write memory content to an internal hard disk in the event of a power failure allayed those concerns.
OH+R has 250 concurrent users today, but Flieger wanted to see Systoc scale to 600 concurrent users to support future growth. IniNet arranged for the use of servers and lab time at HP's benchmark center in Baltimore, while OH+R sent staffers to develop and administer the testing scripts. "They banged the hell out of those servers," says Flieger. The test system ran out of gas at 300 users.
The problem was a process that passes Windows authentication to SQL Server. Turning that off cut processor usage by 30 percent but required a generic SQL Server user ID and password to avoid an extra user log-in. In doing so, Flieger lost the ability to associate SQL Server database log activity with specific users. But when the number of concurrent users reached 600, he decided to continue with the migration.
Fresh Start
Flieger and his staff came up with a consistent schema for the new database but still faced integrity problems with the existing data. So OH+R broke with its past. The old systems would be maintained as "accounts receivable" databases for another 12 months, while the production database and schema would start anew. "It made sense for us to start fresh," says Walkover. After about a year, the old database will be phased out, says Ken Martin, vice president of customer relations at Occupational Health Research.
For its first migration, in Warwick, Rhode Island, OH+R spent two months doing eight test data conversions, going back each time to fix problems. On May 19, the "technical go-live" date, the final data conversion began. Users could enter patient information and scheduling data, but not billing information, for the first two weeks. The new system finally went live for billing on June 2. "It was the smoothest conversion I've ever seen. We didn't have a single phone call," says Martin.
Users now have access to Exchange e-mail, Microsoft Office and Systoc through the NFuse portal. Flieger will begin planning the migration of OH+R's human resources and accounting software next.
Garabino predicts "huge" savings in billing and collection, since that's where the system has been weakest. But he also expects smaller gains from improved business processes. "There will be productivity improvements related to the system all over," Garabino says.
Mainframe Redux?
As OH+R began designing its new data center and information system, chief executive officer John Garabino left most of the decisions to his IT staff, but past experience left him with a strong bias against one technology option. Garabino had been burned at his previous company after approving what turned out to be a very expensive system based on an IBM System/36 minicomputer. This time, he says, "the only thing I said was it absolutely had to be PC-based, and I didn't want to hear anything about minicomputers."
That didn't appear to be a problem at first. But Flieger's data center design, with its Windows terminals and MetaFrame back end, looked a lot like the old mainframe-based time-sharing model to Garabino, and Mark Flieger, vice president of information services, had to explain how it was fundamentally different.
The new system is, however, the same as a mainframe in one respect, Flieger says. "It's awesome when somebody can call you and say they need access to an application, and within 30 seconds, they're staring at it," he says.
But Garabino was put off by the "glass room" data center concept. "I said, 'Wait a minute, are we going back to the old computer room? If you start talking about raised floors and air conditioning, I'm going to be very nervous,' " he says.
So Flieger ended up explaining the intricacies and benefits of PC virtualization technology, client/server architecture, centralization of critical IT assets and how the new system is both similar to and different from the traditional mainframe architecture.
Garabino is now sold on the new system but acknowledges that it still sounds a bit like the old mainframe architecture to him. "What goes around comes around," he says.
Robert L. Mitchell
FACT SHEET
Occupational Health + Rehabilitation Inc.
Hingham, Massachusetts
Goals: Migrate practice management system from 13 dBase databases to a single data store. Build high-availability data center; minimize bandwidth requirements across frame-relay network.
Method: Use thin clients and Citrix MetaFrame server farm front end with back-end SQL Server database. Use Fibre Channel storage arrays and RAM disk to maximize database performance and packet-routing appliances to optimize wide area network bandwidth.
Project cost: US$1.6 million, including $400,000 in server gear.
Results: Improved reliability and uptime, consolidation of 13 databases into one, which reduced redundant reports and enabled comprehensive automated enterprisewide reporting for the first time.