High Availability and Rapid Recovery Protection (HARRP) Reference Guide 8.0, 8.1, 8.2 June 2006

Abstract

The HARRP Reference Guide (P/N 100020700-03) provides reference information and procedures for using the High Availability and Rapid Recovery Protection (HARRP) application.

Manufacturer and European Representative

Manufacturer:

European Representative:

Varian Medical Systems, Inc. 3100 Hansen Way, Bldg. 4A Palo Alto, CA 94304-1030, USA

Varian Medical Systems UK Ltd. Gatwick Road, Crawley West Sussex RH10 9RG United Kingdom

Notice

Information in this user guide is subject to change without notice and does not represent a commitment on the part of Varian. Varian is not liable for errors contained in this user guide or for incidental or consequential damages in connection with furnishing or use of this material. This document contains proprietary information protected by copyright. No part of this document may be reproduced, translated, or transmitted without the express written permission of Varian Medical Systems, Inc.

FDA 21 CFR 820 Quality System Regulations (CGMPs)

Varian Medical Systems, Oncology Systems products are designed and manufactured in accordance with the requirements specified within this federal regulation.

ISO 13485

Varian Medical Systems, Oncology Systems products are designed and manufactured in accordance with the requirements specified within ISO 13485 quality standards.

CE

Varian Medical Systems, Oncology Systems products meet the requirements of Council Directive MDD 93/42/EEC.

HIPAA

Varian’s products and services are specifically designed to include features that help our customers comply with the Health Insurance Portability and Accountability Act of 1996 (HIPAA). The software application uses a secure login process, requiring a user name and password, that supports role-based access. Users are assigned to groups, each with certain access rights, which may include the ability to edit and add data or may limit access to data. When a user adds or modifies data within the database, a record is made that includes which data were changed, the user ID, and the date and time the changes were made. This establishes an audit trail that can be examined by authorized system administrators.

WHO

ICD-9 codes and terms used by permission of WHO, from: ■ International Classification of Diseases for Oncology, (ICD-O) 3rd edition, Geneva, world Health Organization, 2000. ICD-10 codes and terms used by permission of WHO, from: ■ International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10). Vols 1-3, Geneva, World Health Organization, 1992.

Trademarks

ARIA is a trademark of Varian Medical Systems, Inc. Microsoft and Windows are registered trademarks of Microsoft Corporation. © 2005-2006 Varian Medical Systems, Inc. All rights reserved. Printed in the United States of America.

ii

Contents CHAPTER 1

INTRODUCTION ...1-1

Visual Cues ... 1-1 Associated Publications ... 1-2 Contacting Support ... 1-2 Ordering Additional Documents ... 1-3 Communicating Via the World Wide Web ... 1-3 Sending E-Mail... 1-3 About HARRP ... 1-4 How HARRP Works ... 1-4 HARRP Configurations ... 1-5 CHAPTER 2

PERFORMING THE FAILOVER PROCESS...2-1

About Failover ... 2-1 Initiating Failover ... 2-2 CHAPTER 3

PERFORMING THE FAILBACK PROCESS ...3-1

Before you Start ... 3-1 Initiating Failback ... 3-2 CHAPTER 4

PERFORMING THE RESTORATION PROCESS...4-1

About Restoring Data... 4-1 Before you Start ... 4-2 Restoring Data ... 4-2 GLOSSARY... GLOSSARY-1 INDEX...INDEX-1

iii

List of Figures CHAPTER 1 One-to-One LAN Configuration...1-5 One-to-One WAN Configuration ...1-6 Two-to-One LAN Configuration...1-7 Two-to-One WAN Configuration ...1-8 CHAPTER 2 Failover Control Center ...2-2 Failover Condition Met ...2-3 Failover Complete ...2-4 Failover Complete on Two Machines...2-5 CHAPTER 3 Failover Control Center ...3-3 Failed Over Source Machine...3-4 Monitored Machines/Names List Expanded...3-5 Failback Complete ...3-6 CHAPTER 4 Management Console ...4-3 Restoration Manager...4-4 Management Console, Restoration Complete ...4-5 Connection Manager, One To One Mapping ...4-6 Connection Manager, Mirroring Tab ...4-7 RT Chart, History Tab ...4-8

v

Chapter 1

Introduction

The High Availability and Rapid Recovery Protection (HARRP) Reference Guide includes information and instructions for using HARRP, an optional product that provides a solution for disaster recovery and fault tolerance for the ARIA system. The High Availability and Rapid Recovery Protection (HARRP) Reference Guide is written for ARIA network administrators responsible for performing disaster recovery processes and procedures. In This Chapter Topic

Page

Visual Cues

1-1

Associated Publications

1-2

Contacting Support

1-2

About HARRP

1-4

HARRP Configurations

1-5

Visual Cues This manual uses the following notational conventions to help you locate and identify information: Note:

A note describes actions or conditions that can help the user obtain optimum performance from the equipment or software.

CAUTION:

A caution describes actions or conditions that can result in minor or moderate injury to personnel, damage to equipment, or loss of data.

1-1

In addition to the notational conventions shown above, this manual also uses the following: ■

Italic text - Identifies manual titles, new terminology, alternate terms, information you type in during procedures,

■

Bold text - Used for names of dialog box options (command and option buttons, check boxes), window options (menus and menu options, and toolbar buttons), names of keys, controls, and switches.

■

File > Exit - This sequence defines selecting a menu, then option selection from that menu. The sequence of selections are two or more selections deep into a menu. The selection here means to click on the File menu, then click the Exit option from that menu.

Associated Publications For additional information about HARRP, refer to the ARIA – HARRP Release Notes (100020520). For detailed information about Double-Take®, refer to the Double-Take User’s Guide, which is available at http://www.doubletake.com/_docs/pdf/double-take-user-guide-4.4.pdf.

Contacting Support Support services are available without charge during the initial warranty period. If you seek information not included in this publication, call Varian Medical Systems support at the following locations:

1-2

■

North America toll-free telephone support

+1.888.827.4265

■

Global telephone support

+1.702.938.4807

■

Global telephone support for treatment planning

+1.702.938.4712

HARRP Reference Guide

Ordering Additional Documents To order additional documents, call the following:

■

North America

+1.800.535.5350 (Press 1 for parts)

■

Global

+1.702.938.4700

Communicating Via the World Wide Web If you have access to the Internet, you will find Varian Medical System support at the following location: Oncology Systems - http://www.varian.com/oncy/ Then click Support from the menu list along the left side of the window.

Sending E-Mail Send your e-mails to the following locations for support:

■

Information Management Systems

■

Digital Imaging Management [email protected] Systems

■

Delivery Systems

■

Treatment Planning Systems

■

Brachy Therapy Systems

Introduction

[email protected]

[email protected] [email protected] [email protected]

1-3

About HARRP HARRP is an optional product that provides a solution for disaster recovery and fault tolerance for the ARIA system. HARRP uses a third-party application called Double-Take® to continuously monitor the Varian System database and image server and to replicate all of the data to a designated backup server. In case of a disaster or a fault to one of these servers, Double-Take provides semi-automated failover to the backup server with minimal interruption to department operations and treatments.

How HARRP Works Double-Take continuously monitors mission-critical data on your production machine, known as the source machine. Any changes to the data are immediately sent to the designated backup machine, known as the target machine. The target machine includes a dormant copy of Sybase with the exact geometry and installation paths as the source machine(s). All specific services and daemons are installed and configured as well. These programs and services are left in a dormant state with no functionality provided to network users. The target machine simply stores a copy of the critical data from the source machine. This real-time replication keeps the data on the target machine up-to-date with the data on the source machine. Double-Take also monitors the status of machines by tracking network requests and responses exchanged between the source machine and the target machine. When the source machine misses a specified number of requests, Double-Take assumes that the machine has failed and notifies you by e-mail or notifies everyone within the domain by a net send command (depending on your configuration) to initiate failover. During failover, the target machine assumes the network identity of the failed source machine so that all user and application requests destined for the source machine are routed to the target machine. When the source machine is repaired and ready to be brought back online, you manually initiate a failback, which reverts the target machine to its original identity. Once failback is complete, you perform a restoration, which copies the up-to-date data from the target machine back to the source machine so that both machines are once again in sync.

1-4

HARRP Reference Guide

HARRP Configurations A Varian service representative performs the initial HARRP installation and configuration at your facility. HARRP can be configured in one of the following two ways: ■

One-to-one configuration – One source machine is protected by one target machine (see Figure 1-1 for LAN configurations and Figure 1-2 for WAN configurations). In this configuration, the Varian System database server (which also includes the ARIA applications) and image server are all on the same source machine. There is one failure scenario with this configuration. The source machine fails.

Source Server Target Server

SYBASE_RS VA_APP_RS VA_DATA_RS

Hospital Network

Workstation

Workstation

Workstation

Workstation

Workstation

Varis/Vision Clients

Figure 1-1 One-to-One LAN Configuration

Introduction

1-5

Source Server Target Server

Internet

SYBASE_RS VA_APP_RS VA_DATA_RS

Hub

Hospital Network

Workstation

Workstation

Workstation

Workstation

Workstation

Varis/Vision Clients

Figure 1-2 One-to-One WAN Configuration

1-6

HARRP Reference Guide

■

Source Server

SYBASE_RS VA_APP_RS

Two-to-one configuration – Two source machines are protected by one target machine (see Figure 1-3 for LAN configurations and Figure 1-4 for WAN configurations). In this configuration, the Varian System database server (which also includes the ARIA applications) is on one source machine, and the image server is on another source machine. There are three failure scenarios with this configuration. 

The source machine containing the Varian System database server fails.



The source machine containing the image server fails.



The source machine containing the Varian System database server fails, and the source machine containing the image server fails.

Source Server

Target Server

VA_DATA_RS

Hospital Network

Workstation

Workstation

Workstation

Workstation

Workstation

Varis/Vision Clients

Figure 1-3 Two-to-One LAN Configuration

Introduction

1-7

Source Server

Source Server Target Server

SYBASE_RS VA_APP_RS

Internet VA_DATA_RS

Hub

Hospital Network

Workstation

Workstation

Workstation

Workstation

Workstation

Varis/Vision Clients

Figure 1-4 Two-to-One WAN Configuration

The failover, failback, and restoration processes differ slightly depending on the network configuration and the failure scenario.

1-8

HARRP Reference Guide

Chapter 2

Performing the Failover Process

When you are notified by e-mail (or by a net send command) that a source machine has failed, you need to perform the failover process to instruct the target machine to take over for the failed source machine. Note:

If you are not comfortable performing this process, contact your local Varian service representative for assistance.

In This Chapter Topic

Page

About Failover

2-1

Initiating Failover

2-2

About Failover Failover is a process in which the target machine takes over for the failed source machine. Failover allows user and application requests directed to the failed source machine to be routed to the target machine. Double-Take monitors the status of machines by tracking network requests and responses exchanged between the source machine and the failover target. The time between requests and the number of allowable responses that can be missed are combined to create a timeout period. When the source machine fails to respond before the timeout period has expired, Double-Take determines that the source has failed. At this time, you will be prompted by e-mail to initiate failover. During failover, the target server assumes the network name and IP address of the failed source server. All user and application requests destined for the source machine are routed to the target machine.

2-1

Initiating Failover You use this process to initiate failover when one or more source machines fail. Note:

User and application requests destined for the source machine will not be delivered until the failover process is completed.

To initiate failover: 1. On the target machine, open the Failover Control Center by clicking Start > Programs > Double-Take > Failover Control Center. Failover Control Center opens (see Figure 2-1).

Select target machine then click Login

Status bar

Figure 2-1 Failover Control Center

2. From the Target Machine list, select the target machine.

2-2

HARRP Reference Guide

3. Click Login. The Monitored Machines/Names list displays a list of source machines. Failed source machines display with a yellow bullet. The Manual Intervention Required message box also opens with a message stating that a machine has met a failover condition (see Figure 2-2).

Yellow bullet indicates a failed source machine

Click OK to initiate failover

Figure 2-2 Failover Condition Met

4. In the Manual Intervention Required message box, click OK to initiate failover.

Performing the Failover Process

2-3

5. In the Monitored Machines/Names list, click the plus (+) sign to the left of the failed source machine to expand the list and view more of the machine’s IP address and status. When failover is complete, the source machine displays with a red bullet and the text Failed Over (see Figure 2-3).

Red bullet and the text Failed Over indicates that failover is complete

Figure 2-3 Failover Complete

6. If you have a two-to-one configuration, and both source machines failed, from the Monitored Machines/Names list, select the second failed source machine, and click Failover. In the confirmation dialog box, click Yes to continue. Note:

2-4

Only failed servers need to be failed over.

HARRP Reference Guide

Failover begins. When failover is complete, the source machine displays with a red bullet and the text Failed Over (see Figure 2-4).

Failover complete for both source machines

Figure 2-4 Failover Complete on Two Machines

7. On the target machine, open Windows Services by clicking Start > Settings > Control Panel, double-clicking on Administrative Tools, and double-clicking on Component Services.

Performing the Failover Process

2-5

8. Start the following services by right-clicking on each service and selecting Start. Notes:

■

Some of the following services may not display in the Windows Services list because they are installed on a client machine connected to the server instead of the server itself. You need not do anything about those services at this time. When you reboot the client machines in step 11 of this procedure, any services on those client machines will automatically start.

■

Your system may be using failover scripts to automatically start the following services in the event of a failover. In this case, you can simply verify that the services have started successfully.

■

Apache2 Services

■

Dispatcher (you only need to start this service if you have Eclipse)

■

File Server Host Server (you only need to start this service if you have Dynamic Documents or Document Manager)

■

Loader (you only need to start this service if you have Eclipse)

■

Sybase BCKServer_<server name>

■

Sybase SQL Server_<server name>

■

Task Scheduler

■

Varian Flexnet License Manager

■

Varian Syslog Daemon

9. Close the Component Services and Administrative Tools windows. 10. Choose File > Exit to close Failure Control Center.

2-6

HARRP Reference Guide

11. Reboot all of the client machines connected to the failed source server(s). 12. When the failover process is completed, do the following: CAUTION:

To prevent any data loss, you must perform the following steps.

■

Ask your users to re-evaluate their patient plans from the time the server shut down to one hour prior to shut down to ensure that all of their data was saved in the Varian System database.

■

If Eclipse is integrated with your system, and a failover occurs during the calculation process, recalculation may be necessary. Ask your users to verify that their calculation processes were completed and to recalculate if necessary. If users experience an error during recalculation, your beam data may need to be reconfigured. Contact your local Varian service representative for assistance.

Performing the Failover Process

2-7

Chapter 3

Performing the Failback Process

Once you have resolved the problems on your source machine, you are ready to perform the failback process. The failback process reverts the target machine back to its original identity so that the source can be brought online. Note:

If you are not comfortable performing this process, contact your local Varian service representative for assistance.

In This Chapter Topic

Page

Before you Start

3-1

Initiating Failback

3-2

Before you Start Before you begin the failback process, keep in mind the following information. ■

You must correct the source machine problems while disconnected from the network to avoid a name or IP address conflict. After the problems are corrected, you can initiate failback.

■

Do not connect the source machine to the network until failback has completed. For Windows, this means that the source’s identity is completely removed from the target.

■

You must perform the failback when no users are using the system; therefore, we recommend you perform this process during off-hours and that you notify your users that the ARIA system will be unavailable during that time.

3-1