Searching for that Next Generation PACS

I made the annual trek to Chicago this year, as usual right after Thanksgiving Day, to participate in the Radiological Society of North America meeting.  I don’t attend RSNA.  I “participate” in RSNA.  It’s like that adage that goes something like this “There are people who play the game, while there are people merely watching the game being played, and then there are those people who have no idea that a game is being played”.

I’m bringing up the adage, because I have seen it written and heard it said numerous times since the meeting that “There was nothing really new at this year’s RSNA”.  Anyone who actually holds that opinion must fall into that third group of people who have no idea that there is a game being played.

I had a number of reasons for checking out the current state of Radiology PACS.  I have clients with newly installed Vendor Neutral Archives that are now looking to replace their current PACS with a department-focused PACS that is a better fit with the VNA.  I had also recently read a very thought provoking article written by Lisa Fratt and published on line by Health Imaging on Nov 2, 2012, in which Lisa quoted lengthy comments on the state of Radiology PACS by Drs. Chang, Dreyer and Siegel.  I highly recommend reading that article, because my take away was that most of the current generation Radiology PACS, especially those in the larger booths, are not meeting the new list of expectations.

I think one of the problems that PACS vendors are having is investing development dollars at the same time they are being asked to reduce the sales price.  Today’s enlightened customer is asking for new features like multi-facility and multi-PACS workflow, powerful business analytics, and EMR integration to be packaged into a department PACS that should simply focus on what goes on inside the Radiology department.  A customer that has invested in a VNA and has already “image-enabled” the EMR with a universal viewer doesn’t need the department PACS to manage the data for its lifecycle, or provide the clinical viewer for the referring physicians.  We want the PACS vendors to add in the new workflow and analytics goodies, but reduce the overall price because of the removal of those applications that are no longer needed.  Apparently the call for a price reduction is falling on deaf ears.

Unfortunately that is the growing expectation among healthcare organizations that have invested in VNA and Universal Viewer solutions.  A growing number of providers believes that a department focused PACS is supposed to be significantly less expensive than the stand-alone, do-it-all model, that Dr. Chang refers to in the article as PACS 2.0.

Not only was I disappointed to learn after two days of RSNA booth visits that a department-focused PACS was merely 10% or so less expensive than the stand-alone model, but that the version of the PACS being demoed in the booths still does not include all the new goodies that are required.  That reminded me of yet another adage that was recently attributed to the new smaller packaging we see on the grocery shelves.  “Statements on the label such as New and Improved, really mean Smaller but just as Expensive.”

Based on what I saw at RSNA 2012, the major PACS vendors are failing to keep up with current market requirements.  So where is that next generation PACS, the one that sticks to the knitting in the department, includes the new Workflow, Analytics and EMR integration requirements, interfaces to the VNA, yet costs a good 20% less than the PACS 2.0 models?

They were there, on the floor, but you had to know where to look.  You had to be one of the people actually playing the game.

I’m going to do something I normally don’t do in my posts.  I’m going to name a few names along with the following disclaimer: I didn’t do an exhaustive search so I’m certain that I missed a few vendors and equally applicable solutions.   Therefore I am only commenting on the few that I did review.

If one were to describe the technology requirements for what we shall call the ideal next generation PACS, those requirements could be divided into three distinct subsystems: Workflow/Analytics, Visualization/Diagnostics, and Enterprise Archive/Distribution.  The adoption of the VNA and the Universal Viewer are well underway, and the arguments in favor of their deployment are well understood.  As for the other two subsystems that comprise what we think of as the department PACS, the three companies that I reviewed that offer commercially available Work Flow and Analytics Packages are: Compressus, Medicalis, and Primordial.  There are numerous companies that offer attractive Visualization/Diagnostics packages, but the application suite that I reviewed was developed by Visage Imaging.  Based on what I have seen and learned at RSNA 2012, in my opinion it is now possible to construct a next generation PACS by layering one of the three Workflow/Analytics packages on top of the Visage Imaging Visualization/Diagnostics package and interface this combination with a true VNA and the Universal Viewer package.  Once again, based on what I learned and in my opinion, that combination is substantially more potent than any of the PACS 2.0 solutions that the traditional PACS vendors are offering.

In my next blog I want to address numerous technology issues that had to be addressed by the participating vendors to make this best of bread combination work (like Directory database synchronization), and discuss some of the Service Level Agreement and Support issues that still need to be addressed.

Please stay tuned.  I think the Radiology PACS replacement market just got a whole lot more interesting.

Multi-Phase VNA Deployment Strategy

I have been preaching the merits of a multi-phase deployment strategy for a Vendor Neutral Archive for some time now.  This approach is very reminiscent of the deployment strategy for Radiology PACS adopted in the mid-1990’s.  A properly configured VNA is an even larger system than an organization’s largest PACS, so it too represents a large investment.

Nevertheless, a lack of funding to cover the deployment of the complete VNA should not deter the organization from making a start.  Every day going forward, the organization is forwarding a significant volume of new studies to their department PACS, which in the course of ingestion routinely makes proprietary modifications to the DICOM header.  Those PACS-specific idiosyncrasies in the headers are the primary reason behind the expensive and time-consuming data migration that is required when a PACS is replaced or simply upgraded to a next generation.  In effect, an organization is adding to its data migration liability every day that it continues to forward new study data to its department PACS.  At the very least, the organization should consider a pro-active data migration project…whereby all of the image data being managed by its department PACS would be converted from the proprietary format to a Neutral format and simply parked in a separate storage solution.

Most of the Vendor Neutral Archive vendors can configure entry-level versions of their products that will perform this pro-active image data migration and simply store the result for future use.  This approach makes an excellent Disaster Recovery solution that can be shared among multiple PACS.

I recently spoke at length about this subject with David Yeager,   who is a freelance writer and editor.  David spoke with several other individuals knowledgeable in this space and combined all of that information in a very comprehensive article on this subject.  The article appears in the October issue of Radiology Today.  David’s article is an easy read of a complex subject, so I encourage you to check it out here.  The online article also contains some very simple graphics to reinforce the concepts.

There are several entry points to a multi-phase VNA deployment strategy.  Replacing a PACS vendor’s DR solution with a Vendor Neutral DR solution is one such example that makes both strategic and financial sense.

The Real Reason for Deploying a Vendor Neutral Archive

I came across a PowerPoint presentation that I created in July 2006 that referenced a DICOM Migration Server, a term at that time referring to an “open” DICOM Part 10 Storage Solution.  I vaguely remembered the subject, so I opened the file and reviewed the slides.  I felt as though I had traveled back in time to the very earliest days of the paradigm shift that would one day be referred to as the Vendor Neutral Archive.  That’s six years ago last month.  Slide after slide contained bulleted descriptions of the numerous problems facing an organization that had managed to accumulate no fewer than five different department PACS.  Five separate silos of data that could not be exchanged between the PACS. Five different image viewers that the referring physicians had to toggle between.  The last few slides in the deck laid out a rather optimistic (at the time) plan for a strategic solution to the mess.  A grin spread across my face.

I closed the slide deck, assigned a loud red label to the file so I could easily find it again, and fast-forwarded to the present thinking, “You’ve come a long way baby!”

I have been intensely focused on both the concept and the reality of Vendor Neutral Archive for those last six years.  Perhaps that is why it seems so obvious to me why a healthcare organization should make the switch.  They should “take the A out of PACS”, move the data to a VNA, associate a universal viewer to the VNA and use this combo system to manage the distribution of that data to every other system and caregiver throughout the organization.  These are things that even the best of today’s department PACS are simply incapable of effectively doing in a multi-vendor environment.

Based on the questions I continue to see quoted in the print and electronic publications, posted on-line in the focus groups, and raised at the end of many of my webinars, there still appear to be a large percentage of both PACS admins and IT systems analysts that don’t “get it”.  They seem hung up on the technical features of the VNA and all of the potential snags that they fear are bound to occur when two systems, more importantly two vendors, are forced to work together.  The litany of both identified and suspected complications goes on and on.  No doubt the incumbent PACS vendors skillfully placed many of the items on these lists.

OK, it’s time to step back from the techy stuff for a minute.

It’s true.  Many currently installed department PACS are incapable of efficiently inter-operating with a foreign archive without help, simply because they were not designed to work with a foreign archive.  The installed base of VNA solutions would be a pitiful fraction of the real number, if the VNA guys had not developed some very clever workarounds to the inadequacies of many PACS.  Without help, most PACS could not be paired with a VNA. They lack the ability to store images to a foreign archive and then remember where they stored those images.  They are incapable of propagating ADT updates or Merge and Splits to an outside archive.  They have no concept of a deletion policy and therefore have no mechanism for reacting to an externally executed Purge Strategy.  Some PACS have no concept of a relevant prior coming from another PACS, and if the VNA suddenly delivers the study to its doorstep, the PACS thinks it’s a new study and puts it in a reading list.  As I have said, the litany of both identified and suspected complications goes on and on.  The naysayers apparently have not taken the time to read up and learn how all of these problems have been resolved.  As a consequence of those workarounds, the actual installed base of VNA solutions is actually a pretty impressive number.

My advice to those that still don’t get it is don’t get hung up on the technology.  The real argument for deploying the VNA is CONTROL.  It’s time for the organization to take control of its data.  Every day that goes by, another “x” gigabytes is forwarded to the department PACS where it is converted to an effectively proprietary DICOM format that the organization will eventually have to pay in time and money to move to yet another PACS with its proprietary format.  Regardless how soon the organization can afford to replace the incumbent PACS, it’s time to start migrating the data to a VNA…in effect it’s time to mitigate the cost of that future data migration.

What about future VNA migrations, when the first VNA has to be replaced with another VNA?  That’s a really good question.

The answer is actually quite simple.  The real objective in negotiating the contract for the VNA is to gain access outside of any confidentiality stipulation in the contract to the VNA database and all of the tools required to allow the organization to move its data from VNA 1 to VNA 2, at no cost.  Without that arrangement, you’ve missed the point.

Bottom line…initiate a pro-active DICOM migration of the PACS data to a entry-level VNA.  Take control of your data.  As soon as possible, replace the uncooperative PACS with a real PACS, one that fully interoperates with a VNA.

PACS / VNA Compatibility Issues

While much has been written and stated on behalf of the Vendor Neutral Archive being the ideal strategy for managing medical image data across the enterprise, little has been said about PACS Compatibility with the VNA Solution.

There’s a good deal more to this compatibility issue than the PACS being able to communicate with and exchange data with the VNA using DICOM.

Most department PACS, including Radiology and Cardiology solutions, were not designed to inter-operate with a foreign archive.  This is not to say that PACS systems were not designed to occasionally share study files with an external DICOM conformant system.  Most PACS can accomplish this using the DICOM communications protocol.  What I mean is that most PACS system designs are predicated on the assumption that the PACS will be the sole manager of the study data for the lifetime of the system.  And because of this design assumption, many of the current generation of department PACS are ill suited to the tasks required to fully inter-operate with a VNA.

Since most organizations probably did not include this compatibility issue in their PACS selection process, it may come as some surprise to learn that interfacing their existing PACS to a VNA is going to require solving a number of significant in-compatibility issues.

I thought it would be useful to present a summary of the more significant interoperability issues, because organizations need to be aware of the potential problems when they begin the process of planning for a VNA deployment.  The right VNA solution will have to be able to address these incompatibility issues.  It might also be prudent to consider these issues when planning for the next department PACS purchase, because sooner or later, odds are the organization is going to see value in data consolidation, and system interoperability.

Here is a list of the more critical PACS / VNA compatibility issues.

DICOM and IHE…The PACS should support a full suite of DICOM SOP Classes covering the full array of image objects that belong in the patient’s longitudinal record, not just those objects created in the imaging department.  This would include most of the DICOM Structured Report objects, image objects from Ophthalmology, Endoscopy, Pathology, and some of the non-image Cardiology objects like Waveforms and Hemodynamic data.  The PACS should also support image-related objects like Presentation States and Key Image Notes.  The system should also support a few key IHE profiles including Consistent Presentation of Images Profile, Presentation of Grouped Procedures Profile, Key Object Notes Profile, Simple Image and Numeric Reports Profile, and Transparent Query/Retrieve.

Foreign Study Support…The PACS should support the import and representation of Foreign Studies.  Ideally the PACS would directly accept from the VNA, studies originally acquired/processed by another (disparate) PACS or Image Source that are being managed by the VNA.  At the very least, the PACS would be able to receive from the VNA a non-billable order and use that order to aid in the acceptance of studies originally acquired/processed by another (disparate) PACS.

Store and Remember…The PACS should be able to “Store” (archive) DICOM objects originally acquired by itself to a foreign archive (VNA), and then “Remember” that the objects are stored on the VNA when the time comes to retrieve them.

Study Aggregation…The PACS should have the ability to automatically and pro-actively search for studies in the VNA that were originally acquired by another PACS and stored in the VNA (i.e. search the VNA for relevant priors originally acquired by another PACS).

HL7 Updates…the PACS should not only be able to accept HL7 updates from the local RIS or HIS and apply those updates to the metadata in the PACS that is associated with the image data the PACS is managing, but it should also be able to forward the same metadata updates to the VNA.

Object Versioning…The PACS should have the ability to forward to the VNA any updates or changes made to the study data (both pixel and meta data) after the initial “archiving” of the study data, effectively “re-archiving” the image or study.

Retention Messaging…The PACS should have the ability to accept and utilize the messaging from the VNA that is designed to communicate what image and study files have successfully been purged by the VNA’s Information Lifecycle Management (ILM) application.

The subject of PACS / VNA Messaging is actually the most critical of the PACS compatibility issues.  Perhaps one of the more challenging aspects of PACS / VNA interoperability is keeping the two disparate systems synchronized with each other.  Most but not all PACS accept and utilize HL7 updates from the HIS or RIS.  Many PACS do not have a reciprocal mechanism for updating a foreign archive (VNA) with changes that were made to metadata or pixel data in the PACS.

An even more challenging issue is presented by the advent of the purging mechanism that is supported by many VNA solutions.  This issue is referred to in the above list as Retention Messaging.  If a PACS is configured with a small local cache, and it is programmed to allow the oldest studies to fall off of that cache when the watermark is reached, how does it communicate to the VNA that it no longer has that study?  Correspondingly, if the VNA purges a study that has reached its retention limit, how does the VNA communicate to the PACS that the study no longer exists?

A number of the more advanced VNA solutions are attempting to resolve this “synchronization issue” through the use of a number of standard and not-so standard messaging techniques, because the VNA vendors recognize that few PACS vendors have considered VNA compatibility in their PACS designs and fewer still have implemented the appropriate IHE profiles.  Some of those solutions include:

  1. Private DICOM messaging
  2. Custom HL7 messaging
  3. The new IHE Profile called Imaging Object Change Management (IOCM), which is still in development

The Imaging Object Change Management Integration Profile will specify how one actor communicates local changes applied on existing imaging objects to other actors that manage copies of the modified imaging objects in their own local systems. The supported changes will include (1) object rejection due to quality or patient safety reasons, (2) correction of incorrect modality work list entry selection, and (3) expiration of objects due to data retention requirements. It will define how changes are to be captured and how to communicate these changes.

The successful assimilation of disparate PACS into an enterprise Vendor Neutral Archive configuration will have its challenges.  I think it is better to fully understand these challenges in order to better prepare for them, and I suggest that this knowledge play a key role in the VNA selection process.   It also makes sense to include the knowledge of these issues in the next PACS selection process, and thereby eliminate as many future interoperability issues as possible.

Pro-Active Image Data Migration to a VNA-enabled Storage Solution is a Strategic First Step in a Multi-Phase VNA Deployment Strategy

The concept of Vendor Neutral Archive (VNA) seems to be gaining traction in the medical Imaging market.  Why wouldn’t it?  It makes a good deal of sense…addressing problems directly attributed to the way department PACS manage image data.  Health Imaging & IT published a list of the top 10 PACS Problems in its July 2011 issue.  In my opinion, the VNA directly addresses 7 of the 10 problems listed in that article. Categories include Integration, Downtime, Hanging Protocols, Interoperability, Out with the Old (data migration), Whose PACS? (Radiology or IT), and Disaster Recovery.

Just to be clear on the concept, I offer the following succinct definition.  The Vendor Neutral Archive is an Enterprise-class data management system that consolidates primarily medical image data from multiple imaging departments into a Master Directory and associated consolidated Storage Solution, thus replacing the individual archives associated with departmental PACS…systems with unfortunate proprietary characteristics that limit their interoperability.   By virtue of it consolidating all of the enterprise image data, the VNA effectively becomes the unified image data repository for the Electronic Medical Record system.

The problem with VNA is not the concept, the problem is the expense.

The properly configured VNA is a BIG system.  It’s often nearly as expensive as the organization’s biggest PACS, since it will be managing all of the data from all of the department PACS, and the proper architecture is a mirrored, dual-sited configuration.

The point I want to make here, is that the properly configured VNA is big enough and expensive enough to require multiple years and multiple budgets…and this is very reminiscent of those first generation PACS deployments in the early 90’s.

Even as late as the mid 90’s, those first generation Radiology PACS required large investments.  And there was admittedly some doubt as to whether that generation of PACS would successfully replace film.  As a consequence, early PACS deployment strategies focused on solving more manageable, smaller problems with entry-level products that were modestly priced.  The mini-PACS for CT, MR, and Ultrasound, and Teleradiology systems were considered PACS precursor products that could ultimately be stitched together to create the full blown department PACS.  The typical PACS deployments in the mid-90’s were phased in over several years and several budgets.

Is there a multi-phase strategy for deploying a VNA?  I think there is.

As long as the organization’s image data is being managed in individual department PACS, the data format is (to some degree) proprietary to the PACS.  Consequently the organization faces the liability of future data migrations, whenever a department PACS is replaced with another vendor’s PACS, or even upgrade to a next generation PACS from the same vendor.  Therefore the first step in a multi-phase VNA deployment strategy should be to migrate all of the organization’s image data out of the individual PACS and into a VNA-enabled Storage Solution…converting the data into a “neutral” format in the process.  In this strategy, the organization is simply “parking” a copy of its image data for future use.  The hardware and software configuration does not have to support any department PACS operations.  This entry-level VNA is simply managing a copy of the image data “at rest”.

I refer to this first step as the Pro-Active Data Migration, and the ideal time to initiate this migration is at the mid-life of the organization’s largest PACS, which is frequently its Radiology PACS.  By the time all of the historical data has been successfully migrated and cleansed, its time to replace that largest PACS.   Now that the organization is in control of its image data, both the old and the new PACS vendors have much less leverage in the selection process.

Here are a few tips.  (1) Look for a VNA vendor that understands that the value of the VNA software license used to manage image data “at rest” (data that is not being accessed by a PACS or a Viewing application) should be a fraction of the value of the fully-activated VNA license.  (2) Consider forwarding the migrated image data to a vendor-managed Cloud Infrastructure, whose fee-for-study costs are probably going to be considerably less than those for an on-site, self-managed solution.

Once the organization has established a repository of one neutral copy of its image data, there are numerous second and third steps that can be chosen to complete the VNA build out.

Best-of-Breed VNA Looks Good on Paper, but can it be Built and Supported?

The initial focus of the Vendor-Neutral Archive concept was two-fold [1] to consolidate individual department PACS archives and [2] facilitate data exchange between those PACS.  As the concept evolved, so did the complexity of the configuration.  Today’s VNA configuration will probably include a pre-fetch and routing application, an HL7 interface, one or more methodologies for acquiring non-DICOM image objects, a QA/QC application suite, a universal viewing application…the list goes on.  It is highly likely that a second party is providing one or more of these numerous applications.  Different vendors will most likely provide the hardware infrastructure.  The system solution package will be comprised of multiple professional services components, which may very well be provided by different vendors.

The composition of the ideal VNA and the specific configuration that best meets the initial and long-term requirements of the healthcare organization will likely be an assembly of best-of-breed components and subsystems from multiple vendors. Therein lies a well-known problem.  Multiple vendors means multiple Service Level Agreements and multiple 800 numbers to call for support.  Is it possible for a single vendor to offer technology choices, multiple configurations, then actually build and support that custom solution?  Can a site-specific, best-of-breed VNA succeed?

Sponsored by an unrestricted grant from Dell Healthcare, I recently completed a white paper that addresses this very subject.   The paper was published/released November 27, 2011, on the opening morning of RSNA.  You can download a copy of the paper from this web site, or visit the Dell website through this link to find my paper and other related information from Dell.

The paper explores the complexity of the VNA and presents some of the options available to organizations looking for a best-of-breed VNA implementation.  It also looks at the profile of an ideal partner that has the software, hardware and services expertise to integrate, deploy and support multiple best-of-breed solutions.

A webinar also sponsored by Dell Healthcare and based on this paper was given on Thursday, November 17, 2011.  Follow this link to the Dell web site to retrieve the replay of that webinar.

 

True Business Continuity Requires Two Separated Instances of all Key Applications

I sat in on a webcast Wednesday, July 20, 2011, and noticed how the concepts of Disaster Recovery and Business Continuity were frequently paired…in the bullets and in the presentation…with the inference that if you had a DR solution, you had a Business Continuity solution.  That’s not the case.  A Disaster Recovery solution is basically a second copy of the data, usually stored remotely, so it can be accessed whenever it becomes necessary to replace the first copy of the data that might be lost or temporarily unavailable.  A DR solution could be as simple as an off-line digital tape cartridge or a DVD disk stored in a safe room.  It could be a near-line solution such as a tape library system or an on-line solution like a separate spinning disk system, either of which is directly attached to the PACS.  A second SAN or NAS storage solution paired with its front end gateway server could be located in a remote data center in order to increase its chances of avoiding whatever disaster might take down the PACS and/or its primary copy of the data.  The DR solution could be as elaborate as a Cloud-based storage solution, which often feature multiple data centers located in distant states.

In all of the above examples however, the original PACS application or a standalone display application like a web server is required to access and display that back-up image data. If the PACS application or any of the display applications are down, or in any way unavailable, the second copy of the data cannot be accesses and it cannot be displayed.  In this case, there is no Business Continuity.

A significant number of installed Radiology PACS are based on a single instance of the data management and display application, but they are configured with both a primary and a secondary storage solution, each located in separate data centers.  Depending on how geographically separate those data centers are, the secondary storage solution represents a solid Disaster Recovery solution.

On the other hand, very few Radiology PACS are configured with two instances of the data management application or the display applications.  And because there is only one instance of these applications, such a configuration does not have a Business Continuity solution.  With few exceptions, if the PACS application is unavailable, neither the primary nor secondary copy of the data is accessible.  If the display applications are unavailable, neither the primary nor the secondary copy of the data can be displayed.

A true Business Continuity solution requires two instances of the data management application, which can access either the primary or secondary storage solutions, and two instances of whatever display application the user prefers for accessing and displaying the data.  These two paired applications…data management and data display…should be geographically separated, so either of them can survive the disaster that might befall the other.

Since few PACS can be configured with multiple instances of its data management and display application, the current strategy for building a Business Continuity solution has shifted to deploying a dual-sited Vendor Neutral Archive and a dual-sited UniViewer.  The two separate instances of the VNA and the two separate instances of the UniViewer back each other up in the event of a disaster.  In the event that the only instance of the PACS and/or its only instance of the display application becomes unavailable, the new studies are probably interpreted at the modalities, and the priors are retrieved from whichever VNA is available and displayed using whichever UniViewer is available.  That is a true Business Continuity solution.  Take a look at my previous post regarding Failover Strategies, if you want to get a better idea of how primary and secondary subsystems back each other up.

Failover Strategies in Mirrored Configurations of Medical Image Management Systems

The subject of Failover and Resynchronization is near and dear, as I’ve been configuring mirrored systems for years.  I have become quite familiar with how various vendors address this requirement.  The principal reason for building a mirrored Picture Archiving and Communication System (PACS) or a mirrored Vendor Neutral Archive (VNA) solution is Business Continuity.  Most healthcare organizations realize that they cannot afford to lose the functionality of a mission critical system like a PACS and an Enterprise Archive, so they need more than a Disaster Recovery strategy, they need a functional Business Continuity strategy.   Unfortunately, it’s really tough to build a dual-sited, mirrored PACS that actually works.  The sync and re-sync process drives most PACS vendors nuts.  There are very few PACS that can support multiple Directory databases.  I think this shortcoming of most PACS systems is why we have been configuring mirrored VNA solutions from the beginning…if you can’t configure the PACS with a BC solution, then you should at least configure the enterprise archive with a BC solution.

In the dual-sited, mirrored image management system, there are two nearly identical subsystems, often referred to as a Primary and a Secondary.  The two subsystems are comprised of an instance of all of the application software components, the required servers, load balancers, and the storage solutions.  Ideally these two subsystems are deployed in geographically separate data centers.  While it is possible to make both subsystems Active, so half of the organization directs its image data to the Primary subsystem and the other half directs its data to the secondary subsystem, the more common configuration is Active/Passive.  In the Active/Passive configuration, the organization directs all of its data to the Primary subsystem and the Primary backs that data up on the Passive Secondary subsystem.

When the Primary subsystem fails or is off-line for any reason, there should be a largely automated “failover” process that shifts all operations from the Primary subsystem to the Secondary subsystem, effectively making it the Active subsystem, until the primary subsystem is brought back on-line.  When the Primary subsystem comes back on-line, there should be a largely automated “resynchronization” process that copies all of the data transactions and operational events that occurred during the outage from the Secondary back to the Primary.

Business Continuity operations can be even more complicated in an environment where there is a single instance of the PACS and a dual-sited, mirrored VNA configuration. In this environment, the failover and resynchronization processes can be somewhat complicated, giving rise to numerous questions that should be asked when evaluating either a PACS or a VNA.  I thought it would be beneficial to pose a few of those questions and my associated answers.

Q-1: If the hospital-based PACS and Primary VNA are down, how does the administrator access the offsite Secondary VNA and subsequently the data from the offsite VNA? Is the failover automated, or manual?  If manual, what exactly does the admin do to initiate the failover?”

A: The response depends very much on the VNA vendor and exactly how that VNA is configured/implemented.  Some VNA solutions have poor failover/resynchronization processes.  Some look good on paper, but don’t work very well in practice.  With some VNA vendors, system failover and resynchronization in a mirrored environment is a real strong suit, as they support many options (VMWare, Load Balanced-automatic, Load Balanced-manual, and Clustering).  Some VNA vendors have limited options, which are costly and actually create down time.  The better approach is a Load Balanced configuration with automatic failover (which requires certain capabilities existing on the customers network-VLAN/Subnet/Addressing), with manual failover being the second option (and more common).  VMWare is becoming much more common among the True VNA vendors, but many of these vendors will still implement the VMWare clients in a load balanced configuration until customers are able to span VMWare across data centers and use VMotion technology to handle the automatic failover.  There is also the option of using DNS tricks.  For example, IT publishes a hostname for the VNA which translates to an IP in Data Center (DC) A, the DNS has a short Time to Live (TTL), such that if DC A fails, IT can flip the hostname in the DNS and the TTL expires in 1-5 seconds, then all sending devices automatically begin sending/accessing DC B.

There is also a somewhat unique model that implements the mirrored VNA configuration in an Active/Active mode across both Data Centers – whereby the VNA replication technology takes care of sync’ing both DC’s, the application is stateless so it doesn’t matter where the data arrives, because the VNA makes sure both sides get sync’d.

The point in all of this is simply that the better and obviously preferred approach to failover is a near fully automated approach, ONCE THE SYSTEM IS SET-UP.  Resynchronization of the data should be automated as well.  Only updates/changes to the user preferences might require manual synchronization after a recovery.

Q-2: What do the UniViewer (zero client, server-side rendering display application) users have to do to access the secondary instance of the UniViewer? Do the users have to know the separate URL to login to that second UniViewer?

A: If implemented correctly, the UniViewer should leverage the same technology as described above for the VNA.  The user’s URL call goes to a load balancer, which selects the Active UniViewer rendering server.   If the Primary UniViewer (Active) has a failure, another node, or another data center takes over transparent to the end user. The rendering server in turn points to a load balanced VNA such that the users need to do nothing differently if the UniViewer servers or the VNA servers switch.

Q-3: Where do modalities send new studies if the onsite PACS and/or the Primary VNA are down?

A: Once again, this is highly variable, and there are several options.  [1] If the designated workflow sends new data to the PACS first and that PACS goes down, then I’d argue that the new data should be sent to the onsite VNA.  That means changing the destination IP addresses in the modalities.  [2] Vice-versa if the designated workflow sends the new data to the VNA first.   Most of the better VNA solutions can configure a small instance of their VNA application in what I refer to as a Facility Image Cache (small server with direct-attached storage).  One of these FIC units is placed in each of the major imaging departments/facilities to act as a buffer between the Data Center instance of the VNA and the PACS.  [3] In this case, the FIC is the Business Continuity back-up to the PACS.

If both the PACS and the local instance of the VNA are down, the new study data should probably be held in the modality’s on-line storage, for as long as that is possible.  The modalities could also forward the data across the WAN to the Secondary VNA in the second data center, but the radiologists would probably find it easier to access and review the new study data from the modality workstations.

Of course all of these back-up scenarios are highly dependent on the UniViewer.  In the case of those PACS with thin client workstations, if the PACS system goes down, the workstations are useless.  In the case of fat client workstations, most are capable of only limited interactions with a foreign archive.  See the next question and answer for additional detail.

Q-4: Do the radiologists read new studies at the modalities and look at priors using the UniViewer whose rendering server is located in the offsite data center?

While that is possible, my recommendation would be to use the UniViewer for both new and relevant priors.  Some of the UniViewer technology is already pretty close to full diagnostic functionality, some of the very advanced 3D apps being absent. There are already examples of this use of the UniViewer at a number of VNA sites…not only for teleradiology applications, but also diagnostic review if the PACS system goes down.  My prediction is that the better zero client server-side rendering UniViewer solutions are going to be full function diagnostic within a year.   This is a critical tipping point in the VNA movement…a real game changer.  Once the UniViewer gets to that level of functionality, the only piece of the department PACS that is missing will be the work list manager.   As soon as it’s possible to replace a department PACS with a solid [1] VNA, [2] UniViewer, and [3] Work List Manager, the PACS vendors will have a very difficult time arguing that their PACS (less the Archive and Enterprise Viewer) is still worth 90 cents on the dollar, as they are doing today.

Q-5: Does the EMR, if linked to the onsite UniViewer, have a failover process to be redirected to the offsite UniViewer so that clinicians using the EMR still have access to images through the EMR, or do the users need to have the EMR open in one browser and another browser open that points at the offsite UniViewer which they login to separately?

A: Failover from Primary to Secondary UniViewer should be and can be automated (see 1 and 2 above), if implemented correctly and support by the UniViewer technology.

In conclusion, most healthcare organizations are highly vulnerable to the loss of their PACS, because most PACS cannot be configured with a Business Continuity solution.  That problem can be remedied with a dual-sited, mirrored Vendor Neutral Archive paired with a dual-sited UniViewer.  While most VNA vendors can talk about Business Continuity configurations, their failover and resynchronization processes leave something to be desired.  The reader is encouraged to build a set of real-world scenarios, such as those presented here, and use them to discover which VNA will meet their Business Continuity requirements.  The Request For Proposal (RFP) document that I have created for VNA evaluations has an entire section on Business Continuity and the underlying functionality.

Total Cost of Ownership Models Favor Hybrid Vendor Neutral Archive Configurations

My recently completed white paper  The Anatomy of a Vendor Neutral Archive (VNA) Done Right: The Case for Silo Busting focuses on the subject of Vendor Neutral Archive, but it is more than just another rehash of the technical argument.  Well, there are the obligatory opening paragraphs that present the technical background, but that’s just to make sure we are all on the same page with respect to system descriptions and vocabulary.  The real meat of this paper is a presentation of system architectures; specifically architectures that support business continuity, and a brief look at a real world Cost Model.

Since a dual-sited VNA is both large and complex, requiring geographically separated data centers, the obvious questions are: [1] what are the best deployment options, and [2] what are the associated Total Cost of Ownership (TCO) figures?  The paper considers concepts like Cloud Infrastructure and Software as a Service, because they can have a significant impact on TCO.  In my opinion, organizations that do not already have a remote secondary data center and have limited IT resources need to seriously consider any strategy that simplifies system management and lowers costs.

The Cost Model is very revealing, as it compares a dual-sited, on-premise, self-managed VNA to a dual-sited, on-premise/off-premise, vendor-managed (SaaS) VNA.  The later is now being referred to as a Hybrid VNA.  The model was built for five different organization profiles using comparable configurations, and real world infrastructure and operational costs.

Comparisons - 5 year TCO for Capital and Hybrid VNA

In the Table reproduced here, you can see the encouraging results.  The paper was sponsored by an unrestricted grant from Iron Mountain, and I assure the reader that I was involved in assembling the components of the model and approved every one of the line item costs.

Organizations that are getting serious about deploying a VNA will need a positive cost model to win project approval.  As part of that process, I strongly encourage looking at the Hybrid VNA.

Putting Half of the Vendor Neutral Archive in the Cloud Makes Sense

Organizations looking at deploying a Vendor Neutral Archive have some hard decisions to make.  While there are several motivations for moving all of the enterprise image data, radiology, cardiology, endoscopy, etc. from disparate PACS archives to the consolidated VNA, the economic realities will make it a tough sell in many healthcare organizations. A properly configured VNA, one that provides reliable Disaster Recovery and Business Continuity, should have mirrored Primary and Secondary subsystems located in geographically separate data centers.  That will make a VNA is at least twice as big as all of the organization’s PACS combined!

Furthermore the VNA application suite is considerably more sophisticated than that of the department PACS.  Additional FTE resources, several with specialized expertise, will be required to administer the tag-mapping library, create and manage the retention policies, and monitor overall system performance, the security programs, and storage consumption.

All of which is to say, a properly configured VNA is going to be expensive to deploy and expensive to operate.  Making the economic argument for the VNA is going to be very difficult, because the Total Cost of Ownership of a VNA will almost always be higher than the TCO of a Heterogeneous PACS environment, simply because most department PACS have a weak DR solution, and no Business Continuity solution.  While the VNA will make a number of future data migrations unnecessary, the costs of those future data migrations avoided are typically not allowed in the cost models.

One possible solution to the economic challenge is to leverage Cloud Infrastructure and Software as a Service.  Rather than capitalizing and self-managing the Secondary VNA subsystem in a second geographically remote data center (which most organizations do not now have), the entire Secondary subsystem is operationalized and hosted in a Public (multi-tenant) Cloud Infrastructure.  Additional savings can be realized if the entire VNA, both the on-premise Primary and the off-premise Secondary subsystems, are managed under a Software as a Service contract.  In this scenario, both the on-premise and off-premise storage is delivered and billed on an as-needed basis, and all of the management resources and off-premise hardware infrastructure are shared across multiple organizations.

Hybrid VNAThe VNA configuration with the Primary subsystem on-premise and the Secondary subsystem off-premise in a Cloud is referred to as a Hybrid VNA.  If the organization does not believe that it has the IT resources to manage the on-premise Primary subsystem, there are Hybrid VNA vendors that will manage both on-premise and off-premise subsystems under a Software as a Service contract.  A Hybrid VNA managed entirely under a SaaS contract can have a 30% lower TCO than its capitalized, self-managed, on-premise counterpart.  That 30% savings can be used to make a positive economic argument for deploying the VNA.  For healthcare organizations with limited IT resources and no existing remote data center, the Hybrid VNA may be the only strategy that makes sense.