Archive for the ‘Workflow Issues’ Category

Failover Strategies in Mirrored Configurations of Medical Image Management Systems

Tuesday, June 28th, 2011

The subject of Failover and Resynchronization is near and dear, as I’ve been configuring mirrored systems for years.  I have become quite familiar with how various vendors address this requirement.  The principal reason for building a mirrored Picture Archiving and Communication System (PACS) or a mirrored Vendor Neutral Archive (VNA) solution is Business Continuity.  Most healthcare organizations realize that they cannot afford to lose the functionality of a mission critical system like a PACS and an Enterprise Archive, so they need more than a Disaster Recovery strategy, they need a functional Business Continuity strategy.   Unfortunately, it’s really tough to build a dual-sited, mirrored PACS that actually works.  The sync and re-sync process drives most PACS vendors nuts.  There are very few PACS that can support multiple Directory databases.  I think this shortcoming of most PACS systems is why we have been configuring mirrored VNA solutions from the beginning…if you can’t configure the PACS with a BC solution, then you should at least configure the enterprise archive with a BC solution.

In the dual-sited, mirrored image management system, there are two nearly identical subsystems, often referred to as a Primary and a Secondary.  The two subsystems are comprised of an instance of all of the application software components, the required servers, load balancers, and the storage solutions.  Ideally these two subsystems are deployed in geographically separate data centers.  While it is possible to make both subsystems Active, so half of the organization directs its image data to the Primary subsystem and the other half directs its data to the secondary subsystem, the more common configuration is Active/Passive.  In the Active/Passive configuration, the organization directs all of its data to the Primary subsystem and the Primary backs that data up on the Passive Secondary subsystem.

When the Primary subsystem fails or is off-line for any reason, there should be a largely automated “failover” process that shifts all operations from the Primary subsystem to the Secondary subsystem, effectively making it the Active subsystem, until the primary subsystem is brought back on-line.  When the Primary subsystem comes back on-line, there should be a largely automated “resynchronization” process that copies all of the data transactions and operational events that occurred during the outage from the Secondary back to the Primary.

Business Continuity operations can be even more complicated in an environment where there is a single instance of the PACS and a dual-sited, mirrored VNA configuration. In this environment, the failover and resynchronization processes can be somewhat complicated, giving rise to numerous questions that should be asked when evaluating either a PACS or a VNA.  I thought it would be beneficial to pose a few of those questions and my associated answers.

Q-1: If the hospital-based PACS and Primary VNA are down, how does the administrator access the offsite Secondary VNA and subsequently the data from the offsite VNA? Is the failover automated, or manual?  If manual, what exactly does the admin do to initiate the failover?”

A: The response depends very much on the VNA vendor and exactly how that VNA is configured/implemented.  Some VNA solutions have poor failover/resynchronization processes.  Some look good on paper, but don’t work very well in practice.  With some VNA vendors, system failover and resynchronization in a mirrored environment is a real strong suit, as they support many options (VMWare, Load Balanced-automatic, Load Balanced-manual, and Clustering).  Some VNA vendors have limited options, which are costly and actually create down time.  The better approach is a Load Balanced configuration with automatic failover (which requires certain capabilities existing on the customers network-VLAN/Subnet/Addressing), with manual failover being the second option (and more common).  VMWare is becoming much more common among the True VNA vendors, but many of these vendors will still implement the VMWare clients in a load balanced configuration until customers are able to span VMWare across data centers and use VMotion technology to handle the automatic failover.  There is also the option of using DNS tricks.  For example, IT publishes a hostname for the VNA which translates to an IP in Data Center (DC) A, the DNS has a short Time to Live (TTL), such that if DC A fails, IT can flip the hostname in the DNS and the TTL expires in 1-5 seconds, then all sending devices automatically begin sending/accessing DC B.

There is also a somewhat unique model that implements the mirrored VNA configuration in an Active/Active mode across both Data Centers – whereby the VNA replication technology takes care of sync’ing both DC’s, the application is stateless so it doesn’t matter where the data arrives, because the VNA makes sure both sides get sync’d.

The point in all of this is simply that the better and obviously preferred approach to failover is a near fully automated approach, ONCE THE SYSTEM IS SET-UP.  Resynchronization of the data should be automated as well.  Only updates/changes to the user preferences might require manual synchronization after a recovery.

Q-2: What do the UniViewer (zero client, server-side rendering display application) users have to do to access the secondary instance of the UniViewer? Do the users have to know the separate URL to login to that second UniViewer?

A: If implemented correctly, the UniViewer should leverage the same technology as described above for the VNA.  The user’s URL call goes to a load balancer, which selects the Active UniViewer rendering server.   If the Primary UniViewer (Active) has a failure, another node, or another data center takes over transparent to the end user. The rendering server in turn points to a load balanced VNA such that the users need to do nothing differently if the UniViewer servers or the VNA servers switch.

Q-3: Where do modalities send new studies if the onsite PACS and/or the Primary VNA are down?

A: Once again, this is highly variable, and there are several options.  [1] If the designated workflow sends new data to the PACS first and that PACS goes down, then I’d argue that the new data should be sent to the onsite VNA.  That means changing the destination IP addresses in the modalities.  [2] Vice-versa if the designated workflow sends the new data to the VNA first.   Most of the better VNA solutions can configure a small instance of their VNA application in what I refer to as a Facility Image Cache (small server with direct-attached storage).  One of these FIC units is placed in each of the major imaging departments/facilities to act as a buffer between the Data Center instance of the VNA and the PACS.  [3] In this case, the FIC is the Business Continuity back-up to the PACS.

If both the PACS and the local instance of the VNA are down, the new study data should probably be held in the modality’s on-line storage, for as long as that is possible.  The modalities could also forward the data across the WAN to the Secondary VNA in the second data center, but the radiologists would probably find it easier to access and review the new study data from the modality workstations.

Of course all of these back-up scenarios are highly dependent on the UniViewer.  In the case of those PACS with thin client workstations, if the PACS system goes down, the workstations are useless.  In the case of fat client workstations, most are capable of only limited interactions with a foreign archive.  See the next question and answer for additional detail.

Q-4: Do the radiologists read new studies at the modalities and look at priors using the UniViewer whose rendering server is located in the offsite data center?

While that is possible, my recommendation would be to use the UniViewer for both new and relevant priors.  Some of the UniViewer technology is already pretty close to full diagnostic functionality, some of the very advanced 3D apps being absent. There are already examples of this use of the UniViewer at a number of VNA sites…not only for teleradiology applications, but also diagnostic review if the PACS system goes down.  My prediction is that the better zero client server-side rendering UniViewer solutions are going to be full function diagnostic within a year.   This is a critical tipping point in the VNA movement…a real game changer.  Once the UniViewer gets to that level of functionality, the only piece of the department PACS that is missing will be the work list manager.   As soon as it’s possible to replace a department PACS with a solid [1] VNA, [2] UniViewer, and [3] Work List Manager, the PACS vendors will have a very difficult time arguing that their PACS (less the Archive and Enterprise Viewer) is still worth 90 cents on the dollar, as they are doing today.

Q-5: Does the EMR, if linked to the onsite UniViewer, have a failover process to be redirected to the offsite UniViewer so that clinicians using the EMR still have access to images through the EMR, or do the users need to have the EMR open in one browser and another browser open that points at the offsite UniViewer which they login to separately?

A: Failover from Primary to Secondary UniViewer should be and can be automated (see 1 and 2 above), if implemented correctly and support by the UniViewer technology.

In conclusion, most healthcare organizations are highly vulnerable to the loss of their PACS, because most PACS cannot be configured with a Business Continuity solution.  That problem can be remedied with a dual-sited, mirrored Vendor Neutral Archive paired with a dual-sited UniViewer.  While most VNA vendors can talk about Business Continuity configurations, their failover and resynchronization processes leave something to be desired.  The reader is encouraged to build a set of real-world scenarios, such as those presented here, and use them to discover which VNA will meet their Business Continuity requirements.  The Request For Proposal (RFP) document that I have created for VNA evaluations has an entire section on Business Continuity and the underlying functionality.

Posted in PACS, PACS Optimization, PACS-Neutral Archive, Strategic Planning, Workflow Issues

Next Step in Image Sharing…Beyond the CD/DVD

Tuesday, January 12th, 2010

The task of getting a patient’s medical images into the hands of the Specialists, Surgeons and Primary Care physicians becomes considerably more complicated, when those images are produced in an “outside” Organization.  The practice of forwarding film-based images ahead of or with the patient is increasingly rare, having been replaced with the conveyance of digital copies of the patient’s images on CD or DVD.   While this method of data exchange between organizations is considered more efficient and less expensive than forwarding films, the problems associated with data exchange using the CD/DVD are now legendary.

Aside from such obvious issues as viewing software and media compatibility, the principal problem is frequently basic data incompatibility.  The DICOM standard allows a significant degree of “customization” of the DICOM image data header by the PACS vendors.  In a white paper recently written by Dr. Wayne DeJarnette, titled Context Management and Tag Morphing in the Real World and posted on their informational web site, there are 10 examples sited where certain key pieces of information stored in the DICOM header need to be created, modified or moved in order for one PACS to be able to properly interpret the data created by another PACS.  I highly recommend reading this paper to catch up on the subject generally referred to as Tag Morphing.

Apparently the problems associated with sharing medical image data using CD/DVD media has reached critical mass, because a number of solutions in the form of Data Exchange Servers and Data Exchange Services have recently entered the market.  The focus of these new products and fee-per-study services is clearly to provide an end to the pains of data exchange.  Unfortunately there is now yet another set of issues.

Clearly the  most exciting solution to the data exchange issue is the “Image Share Service in a Cloud”.  How can one not get excited about anything in a cloud?  I counted a half dozen such “cloud” service solutions being exhibited at the 2009 RSNA or being advertised since.  The simple, high-level summary description of this fee-per-study service is as follows.  An authorized organization/user accesses the upload application through a secure web site.  A couple of simple clicks and data insertions later and a patient’s medical image data is uploaded to a central server in the cloud.  There are a number of methods for announcing the availability of these images in the cloud to the intended recipient, ranging from email notification to a phone call.  The authorized organization/user then accesses the secure web site hosted by the cloud server and is granted access to only those images intended for their use.

Here is the interesting part.  The intended user then downloads to their PC a very small piece of client software, in some cases there is no client software (zero), and this allows the user to view the images on their PC.  Most of the display applications I have seen associated with this version of the cloud service are based on what is called “server-side rendering”, meaning all of the image rendering, processing, etc. being directed by the user is actually being executed on the server in the cloud.  The result of this rendering, an HTML page, is all that is actually downloaded to the user’s PC.  The actual image data itself is not downloaded to the user’s PC.  The actual image data itself does not leave the secure server in the cloud, making this a very HIPAA-compliant application.

The current state of server-side rendering display applications allows for support of full-fidelity (loss-less) images and a full range of image processing features (2D, 3D, even Orthopedic templating), so the display application associated with most of these cloud-based image exchange services should be well received by a wide range of physicians seeking access to a patient’s images that were produced in an “outside” organization.

What I find most interesting about this approach to image sharing is that this solution totally avoids the data incompatibility problems encountered when an organization attempts to actually import digital image data from an “outside” PACS into their local PACS.  Instead of importing “outside” study data into the local PACS, so the images can be accessed and viewed by the physicians using the local PACS web viewer, the cloud solution depends on its own embedded display application to access and display the image data.  Just like all PACS that customize the image header of incoming image data, the cloud server only has to make the incoming study data produced by the contributing PACS completely compatible with its own display application.  Moreover these new server-side rendering display applications frequently offer a wider range of features and functions than the incumbent local PACS “web Viewer”.   It’s a clever solution that simply avoids the data incompatibility problem.  As mentioned, this version of image sharing is available as either a purchased/leased “appliance” or a fee-per-study cloud-based service.

However clever this solution appears, it is important to remember that this version of image sharing does not solve the data incompatibility problem.  If an organization wishes to assimilate a patient’s image study data created by an outside organization into that patient’s local longitudinal medical record (acquire the outside study data into the local PACS and add the study to the patient’s local folder); the data must first be modified,  more specifically the DICOM headers must first be modified, to satisfy the idiosyncrasies of the receiving PACS.  That means executing Tag Morphing of the type and complexity mentioned in the DeJarnette white paper.  Unless the contributing and receiving organizations have only a few studies a day to exchange, a manual approach to this Tag Morphing would be too labor intensive to be practical, not to mention fraught with the potential for human error.   In short the exchange of study data between two different organizations, and especially between disparate PACS requires an appliance or a fee-per-study service that can automatically execute Dynamic Tag Morphing on the incoming DICOM image data headers, prior to exporting the data to the recipient PACS.  Any solution that does not support this key process, is naively  relying on “DICOM conformance”, and we already know the problems with that approach.

In summary, I think an appliance or a cloud-based service that can provide the physicians with HIPAA-compliant internet access and display of a patient’s outside images is a significant advance over the CD/DVD method of data exchange.  I think the display-only approach is a clever way to avoid the problems inherent in exchanging data between disparate PACS.  The participating organizations simply need to understand their needs and make sure that the chosen solution will meet their expectations.  Products or Services that suggest that actual data exchange between the PACS is an option should be expected to provide evidence that their product or service supports Dynamic Tag Morphing.  Otherwise the organizations will likely end up right back where they are today with their CDs and DVDs.

Note: Currently there’s not a lot of information available on DICOM Tag Morphing out there on the web.  In addition to the DeJarnette paper already mentioned, you might want to focus your favorite search engine on “vendor-neutral archive”, as I’m sure any of those vendors can provide additional info on this subject.

Posted in Image Sharing, News, PACS-Neutral Archive, Workflow Issues

New Breed of Teleradiology Poses Challenging Technology Issues

Thursday, June 18th, 2009

Radiology Groups reading studies forwarded from multiple, often remote facilities is not a new concept, but technical challenges frequently limit the effectiveness of this service and the resulting product of the effort is typically the final report and nothing else.

One of the major benefits of a Radiology Picture Archiving and Communications System (PACS) is its ability to preserve all of the work products associated with the study created by the technologist and radiologist during study preparation and interpretation.  Paper-based information from requisitions to consent forms can be scanned into the PACS and associated with the study.  The window/level settings, graphical- and text-based overlays created by the radiologist can be preserved as the Presentation State  of the images.  Key images can be flagged and shorthand text notes conveying the gist of the report can be created and saved as Key Image Notes.  And of course the radiologist’s final report completes the package. A PACS can preserve all of this clinical information along with the image data in an electronic file that can be accessed and viewed simultaneously by all of the caregivers responsible for the patient’s care.  And all of this radiology information can be combined with a patient’s cardiology studies, laboratory results, medication history and case summaries via the Physician Portal of the Electronic Medical Record System.

If the resulting product of a remote interpretation is nothing more than the final report, all of the caregivers are being deprived of the wealth of clinical information contained in those work products created by the radiologist during interpretation, the Presentation States, Key Image Flags and Key Image Notes.  Furthermore, it is not unusual for that final report to be delayed by several hours at best, while it loops its way through the editing and sign-off process.  That short-hand Key Image Note might easily be the first piece of clinical findings that reaches the referring physician.  In my opinion, a teleradiology solution that promises to deliver more than a preliminary finding should also deliver all of the work products along with the final report.

The technology challenges actually start at the very beginning of the teleradiology process.

It is well known that even current generation PACS are far from being truly open systems.  Idiosyncrasies in the DICOM headers can affect the way the images acquired by one PACS appear on a display screen of another PACS.  The teleradiology system needs to be able to correct for these idiosyncrasies.

Admittedly not all PACS support DICOM Greyscale Softcopy Presentations States (GSPS) or Key Image Notes (KIN), but that is bound to change in the near future, so a new Teleradiology system should support both of these DICOM SOP Classes on day one.

My point is that the deliverable product of a remote interpretation should be the final report AND all of the work products associated with that study.  That means returning the new version of the study, along with all those additional work products, back to the originating PACS.  That brings up another technical challenge.  The originating  PACS will most likely match this new version of the study with the original version, based on the patient Name, Accession Number, etc., but how does the originating PACS determine that the study status has changed from unread to read?  Hopefully the originating PACS can accept an HL-7 update from the local RIS when the associated report is received.  IF not, this is a bit of a loose thread.

Another issue is that of the relevant priors.  Does the technologist have to manually forward the relevant priors along with the new study?  Is the originating PACS capable of auto-forwarding both the new study (based on predefined meta data criteria) and the relevant priors to the teleradiology system?  And at the end of the interpretation, what becomes of the relevant priors, and for that matter the new study?  Is all of this study data simply deleted from the teleradiology system?  Seems like a waste of bandwidth to keep forwarding relevant priors over and over again, each time a new study is generated for the same patient. Wouldn’t it make sense to “archive” all of the studies received by the teleradiology system, so they are available for comparison purposes?  That means the teleradiology system would have to be able to partition its Directory database by originating facility, and possibly deal with multiple Medical Record Numbering Systems.

My argument is simply this, the product of a remote interpretation should be just as inclusive as the product of an in-house interpretation, for the benefit of the caregivers and the patient.  The technology required to achieve this application is considerably more than yesterday’s teleradiology system.  In fact the technology is beyond most current generation PACS.  The ability to accept, display, and manage radiology study data from disparate PACS and return an interpreted study with all the associated work products to the originating PACS in a format that that PACS can recognize as its own is the purview of the PACS-Neutral Archive.

Radiology Practices and Health Systems interested in remote interpretation of Radiology studies would be well served if they carefully consider their respective expectations of such a service and then fully investigate the claims of the system providers, many of which may not fully appreciate the technical requirements of such a system.

Posted in News, PACS Technologies, Workflow Issues

Tackling the Exam Room Display

Thursday, August 4th, 2005

Consider marketing the use of CDís as the means of transferring study data from radiology to the referring physicianís office. All studies required for a dayís worth of office visits are ordered in advance and placed on one CD. The referring physicianís office staff transfers the studies from the CD to specific desktop or lap top computers in the offices and exam rooms. This strategy minimizes network complexity, eliminates time-consuming steps required to access studies, utilizes existing office workflow by assigning all time consuming tasks to the office staff.

Posted in Workflow Issues

Document dilemma

Thursday, August 4th, 2005

It might be useful to re-open the subject of Document Scanning: [1] Where should the document objects be stored (PACS vs RIS), and [2] What would be the best file format for the scanned document object?

A number of radiology departments with PACS are currently managing scanned document objects in the PACS. Many of these PACS are managing the document objects as a DICOM Secondary Capture objects and storing the objects as a new series in the study file. This makes it easy for the radiologist to access and view the document along with the images. This also assures that documents can be migrated along with the study data, whenever necessary.

One of the problems with this method includes having to accommodate this new study series (document objects) in pre-existing hanging protocols. Another problem is the fact that all of these document objects subsequently travel everywhere the study file travels, ending up on display screens all over the enterprise.

Comments in favor of or against this approach are welcomed.

Posted in Pitfalls, Workflow Issues