EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 0 Abstract This white paper provides an overview of the architecture used to show that automated global failover and recovery over nearly any distance is a reality. The white paper will address the issues around designing and deploying a global recovery solution, including: considerations, architecture, and the operation of the failover process. The paper will review and conclude that the integration of SRDF/A and PRIMECLUSTER enables organizations to provide a robust foundation to support highly available enterprise applications. These products provide a real-world option for any organization developing long distance business continuity plans. Technical White Paper Global Recovery Demonstration: SRDF/A and PRIMECLUSTER - EMC Remote Data Facility/Asynchronous Fujitsu Siemens Computers PRIMECLUSTER EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 1 EMC2, EMC, Symmetrix, AutoIS, Celerra, CLARiiON, CLARalert, DG, E-Infostructure, HighRoad, Navisphere, PowerPath, ResourcePak, SnapView/IP, SRDF, VisualSAN, WideSky, The EMC Effect, and where information lives are registered trademarks and EMC Automated Networked Storage, EMC ControlCenter, EMC Developers Program, EMC Enterprise Storage, EMC Enterprise Storage Network, EMC OnCourse, EMC Proven, EMC Snap, Access Logix, AutoAdvice, Automated Resource Manager, AutoSwap, AVALONidm, C-Clip, CacheStorm, Celerra Replicator, Centera, CentraStar, CLARevent, Connectrix, CopyCross, CopyPoint, CrosStor, Direct Matrix, Direct Matrix Architecture, EDM, E-Lab, Enginuity, FarPoint, FLARE, GeoSpan, InfoMover, MirrorView, NetWin, OnAlert, OpenScale, Powerlink, PowerVolume, RepliCare, SafeLine, SAN Architect, SAN Copy, SAN Manager, SDMS, SnapSure, SnapView, StorageScope, SupportMate, SymmAPI, SymmEnabler, Symmetrix DMX, TimeFinder, Universal Data Tone, and VisualSRM are trademarks of EMC Corporation. All other trademarks used herein are the property of their respective owners. PRIMECLUSTER, PRIMESERVER Trademark and Copyrights of Fujitsu Siemens Computers. Copyright 2003. All rights reserved. Part number H1063 EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 2 Table of Contents Executive Summary....................................................................................................... 3 Summary............................................................................................................................3 Purpose..............................................................................................................................3 Scope.................................................................................................................................4 Audience ............................................................................................................................5 The Global Recovery Demonstration........................................................................ 6 Major Components..............................................................................................................6 Summary.........................................................................................................................6 Cluster Nodes..................................................................................................................6 Storage Systems..............................................................................................................6 Storage Area Network (SAN) ............................................................................................6 DMX Fibre Channel and GigE...........................................................................................7 Remote Data Replication (Symmetrix Remote Data Facility/Asynchronous) .........................7 SRDF/A Architecture........................................................................................................9 WAN Testing .................................................................................................................10 Local Area Network (LAN) .........................................................................................11 Wide Area Network (WAN) .............................................................................................12 QoS Details...................................................................................................................13 Enterprise Application (SAP/R3)......................................................................................14 Driver System (SD Benchmarking)..................................................................................14 Cluster Software (Fujitsu Siemens Computers PRIMECLUSTER).....................................14 Strategy............................................................................................................................14 Application Availability....................................................................................................14 Data Protection..............................................................................................................15 Network Configuration....................................................................................................15 Putting It All Together........................................................................................................15 SRDF/A Infrastructure ....................................................................................................15 PRIMECLUSTER Network Infrastructure .........................................................................16 Logical Storage Architecture...........................................................................................16 Final Architecture...........................................................................................................18 Cluster Operation ..............................................................................................................19 Preparation (both nodes are available)............................................................................19 Site Failure (Hopkinton node is not available)...................................................................19 Site Failover (transition to Cork and bring Cork node online).............................................19 Summary........................................................................................................................20 Appendix ........................................................................................................................21 DMX Configuration for Hopkinton and Cork.........................................................................21 Definitions Used in this Document ......................................................................................25 References.......................................................................................................................25 EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 3 Executive Summary Summary This whitepaper was developed in partnership between the EMC Corporation and Fujitsu Siemens Computers. The objective of this effort was to demonstrate a new paradigm in application availability, information protection, and business continuity. By integrating EMC’s recently announced Symmetrix Remote Data Facility/Asynchronous (SRDF/A) with Fujitsu Siemens Computers’ PRIMECLUSTER technology, we have been able to demonstrate that unlimited distance application availability is now a practical undertaking. EMC’s SRDF replication technology has evolved as the industry-leading information protection solution in the market today. The market has matured to produce various deployment options so that customers may achieve the appropriat e service level for information protection relative to performance, distance, and cost. The integration of replication technology and clustering technology has enabled organizations to automate application availability for production and disaster recovery purposes. Previously, these integration efforts resulted in solutions that were limited to no more than 200km of separation between the primary and alternate processing location. This limitation is due to the following: a) as distance increases between primary and alternate/backup locations, the propagation delay caused by the speed of light negatively affects local application response time; b) clustering technologies available previously were limited by the physical distance limitations imposed by the underlying synchronous data replication method. EMC and Fujitsu Siemens Computers have created a new service level alternative that eliminates distance as a consideration in achieving global business continuity capabilities. In the face of new global risks, organizations desire to leverage existing personnel, facilities, and networks to contain costs while also seeking greater geographic diversity between processing locations. Organizations now have the ability to achieve metropolitan-class application availability across previously impossible distances. A distance-independent and application-transparent remote clustering solution has been developed by Fujitsu Siemens Computers and EMC. This solution combines Fujitsu Siemens Computers’ PRIMECLUSTER technology with EMC’s SRDF/A replication technology. This integration enables geographically-distributed and application-transparent availability with very small RTO and RPO solution attributes. The PRIMECLUSTER technology exploits the features of EMC SRDF/A replication technology to meld the benefits of clustering and replication in a revolutionary manner. This integration enables a network-effici ent unlimited-distance application availability solution for UNIX and Linux environments. SRDF/A’s Delta Set technology enables bandwidth-effici ent remote replication over any distance with no performance impact on production applications. This whitepaper details how this was accomplished in a demonstrated scenario between EMC solution centers located in Ireland and the United States. SRDF/A is the world’s highest performing asynchronous remote replication technology supporting unlimited distances with no host impact. SRDF/A offers long distance replication without the production application latency common to other remote replication technologies. This patented network-effi cient technology ensures a consistent point in time of customer data at the remote location. Together, EMC and Fujitsu Siemens Computers have exploited this capability by extending the benefits into the clustering market. This integrated clustering capability creat es synergy between information protection (EMC SRDF/A) and automated application availability (Fujitsu Siemens Computers PRIMECLUSTER). Purpose The purpose of the demonstration is to show that the integration of EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER products work seamlessly in a real business scenario to achieve global application availability and continuity. The scenario chosen consisted of an enterprise business application (SAP) connected via a wide area network (WAN) between the United States and Ireland. This WAN supporting this demonstration is also supporting daily production activity. SRDF/A and PRIMECLUSTER products provide the high availability functionality. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 4 Scope The scope of the demonstration was to demonstrate the seamless integration of the two products and the automatic nature of the failover process. There are many ways to charact erize any IT system and articulate various operating scenarios. We have focused the emphasis of this demonstration to illustrate a site failure in Hopkinton, MA with application resumption to Cork, Ireland. The primary objectives are the protection of application availability, information integrity, and the confidentiality of these major components. IT infrastructure components 1. Storage 2. Server and OS 3. Application and Database 4. Access Infrastructure and Middleware 5. People The business challenges are: • Business is global - Business environments must run 24x365 around the globe • Distance matters - Longer distances between recovery sites is becoming a requirement • The environment has changed - Natural and mechanical disasters have, and will continue to occur To provide a highly available system the goal is to ensure there is no single point of failure for each component. It should be clear that building a highly available cluster will address the first three components. There are many methods, products and technologies available for protecting the other components. Fujitsu Siemens Computers Professional Services provide products and services to ensure protection for all five components. This is essential when building a real-world business continuity plan. This demonstration does not therefore address the development of a comprehensive disaster recovery plan (DRP), nor the wider topic of a business continuity plan (BCP). Such plans must address the related areas of business impact analysis, legal obligations, assignment of business priorities, the role of insurance, service level agreements, staff availability, additional security and costs, etc. These topics are outside the scope of this demonstration. There are usually two distinct drivers for building highly available systems which require different strategies. The first driver is to ensure the system can cope with normally expected peak loads. In this scenario, locally redundant paths and systems are usually sufficient with some form of load balancing. The second driver is the requirement to ensure the system can cope with a complete failure of the primary site, i.e., a complete disaster. In this scenario remote redundancy is required. This demonstration addresses the DR scenario only. Strategies for building highly available systems: 1. Use local redundant components 2. Use remote redundant components 3. Use manual activation on failure of primary component 4. Use automatic activation on failure of primary component Providing high levels of availability implies redundant components geographically distributed to remote sites. Furthermore, activation of the secondary site (server, application, database, and storage) should be automatic. There are many products and technologies available today to provide the required level of redundancy for each component. What is not widely available is the ability to provide automatic and robust remote redundancy at the storage level. In many organizations this means that bringing the remote site online was not an automatic process, it required reconfiguration of network and server systems and often, restoration of files and databases. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 5 With clustering and remote data replication, you avoid all of these issues. The primary objectives of the storage system in a DR scenario with automatic failover are: a) Failover of storage to secondary site is automatic. b) Storage at secondary site is consistent c) Storage at secondary site is restartable. In achieving these objectives a number of immediate benefits are realized: a) An orderly transition of the application to the secondary site is achieved. b) Data integrity is achieved during and after transition. c) Automatic implementation of management’s desired DR procedures without human intervention. d) Full use of the secondary site is possible until it is required for DR. e) The ability to fully and repeat edly test the DR architecture with confidence to ensure it will work when you need it most. Audience This white paper is targeted for IT analysts, managers, and engineers who are responsible for planning, implementing or operating a high availability IT environment using Fujitsu Siemens Computers PRIMECLUSTER and EMC SRDF/A products. The focus of this scenario is to demonstrate the growing market demand for extremely long distance replication and automated application availability. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 6 The Global Recovery Demonstration Major Components Summary Component Speci fication Nodes Platform: Fujitsu Siemens Computers PRIMEPOWER 400, Solaris 8 HBA: Emulex LAN Cisco Gig-E Switches WAN (major components) Router Cisco 7200 ATM Switch Cisco 8510 EMC Hopkinton Campus SONET ATM Switch Cisco 8510 Eircom ATM Net SDH/SONET ATM Switch Cisco 8510 Enterprise Application SAP R/3 IDES V4.6C Oracl e V8.17 DRIVER MS Windows 2000 SD Bencmarking PRIMECLUSTER Admin GUI EMC management software Cluster Fujitsu Siemens Computers PRIMECLUSTER SAP R/3 Wizard SRDF/A Wizard RMS Customization Remote Data Replication SRDF/Asynchronous Storage Symmetrix DMX800 2 X Gig-E Multi-Protocol Channel Directors (MPCD) Brocade Fibre Channel Switch DS16B2 Cluster Nodes The platform chosen for the cluster nodes is the PRIMEPOWER 400 system running Solaris 8. The SAP/R3 system is installed on each of these hosts, along with the supporting software, such as Oracle. Each system boots from the SAN via two host bus adapters (HBAs) which are used to provide redundancy at the HBA level. Each host also has five 10/100MB Ethernet interface cards (NICs) to provided connectivity to the three networks required. Storage Systems The storage systems chosen were two EMC Symmetrix DMX800 series systems. Storage Area Network (SAN) The storage area network consists of a single Fibre Channel switch at each site. These were Brocade DS16B2 switches. The SAN provided the cluster nodes with the means to access their boot and data volumes on the DMX systems. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 7 DMX Fibre Channel and GigE Fibre Channel over IP was chosen as the transport for SRDF/A protocol for this demonstration. To support this there are two GigE Directors, (MPCD or Multi-Protocol Channel Directors) on each DMX system. These operate over 1 Gb/s fiber to the local router. Two GigE Directors are used to provide redundant paths. The Symmetrix GigE Multi-Protocol Channel Director (GigE MPCD) on the DMX enables you to connect 2 DMX systems into a network switch and route SRDF/A data over any TCP/IP network. This eliminates the need to purchase, install and maintain separate conversion devices. It also allows you to maximize the efficiency of existing IP links. And, because the GigE director uses onboard compression it can further increase throughput and utilization of communication links. Remote Data Replication (Symmetrix Remote Data Facility/Asynchronous) Symmetrix Remote Data Facility/Asynchronous (SRDF/A) is an innovative approach to asynchronous remote replication. SRFD/A supports both open systems and mainframe environments with a single solution. It delivers a consistent and restartabl e remote copy of your production data at all times, over any distance, with no host application impact. Only SRDF/A can achieve the disparate goals of multi-volume sets over unlimited distances without impacting production performance or recoverability and without consuming vast amounts of network bandwidth. Unlike traditional time-stamped, ordered write, asynchronous architectures, SRDF/A uses a new cache-based Delta Set architecture to send data sets to the remote secondary node. Delta Sets maintain an atomic copy of the data at the remote sites at all times. SRDF/A is used to ensure that no matter how the primary node or storage subsystem fails, there is always a consistent restartable copy of the data at the secondary location. This ensures that the secondary node can always take over the primary processing loads with the most up to date data from the primary. SRDF/A’s innovative Delta Set architecture utilizes cache-based Delta Set technology that manages all write activity to volumes participating in the SRDF/A group. Delta Sets allow applications to rewrite to tracks numerous times during the Capture Delta Set. This ensures only the last set of writes is write-folded and transmitted to the target location, reducing bandwidth requirements and significantly lowering overall TCO. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 8 The four Delta Sets are described below: Local System 1. Capture Captures, in cache, all incoming writes to the source volumes involved in the SRDF/A group. Upon completion of the set, the Capture Delta Set is “write folded” and promoted to a Transmit Delta Set and begins. (A new, separate Capture Delta Set is then created to maintain the next Delta Set of writes.) It is important to note that the Capture Delta Set is not copied in cache, it is simply promoted in-place). 2. Transmit Transfers its contents—only the very last set of writes—from the source to the target system. Once the Transmit Delta Set is transmitted to the target system, we have the following: Remote System 3. Receive This occurs on the target system where the target receives the Transmit Delta Set. Once the Delta Set is received in its entirety, it is promoted. Again, this happens in-place, in cache, to the Apply Delta Set. 4. Apply Applies the Apply Delta Set’s writes to the target volume to create the consistent, restartable remote copy. This finishes one Delta Set cycle. This cycle is then repeated. SOURCE TARGET Figure 1. CAPTURE Collects application write I/O TRANSMIT Sends final set of writes to target RECEIVE Receives writes from Transmit Delta Set APPLY Once Receiv e is complete, data is applied to disk EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 9 SRDF/A Architecture The following shows the use of the major components required to operate SRDF/A over the corporate wide area network. 12 3 4 5 6 78 9 10 1 12 A B 1 2x 6 8 2 9 3 1 0x 4 1 x 5 7 1 Ethernet A 1 2x 6 8 2 9 3 1 0x 4 1 5 7 1 C 1 2 3 4 5 6 7 8 9 10 1 2 A B 1 2x 6 8 2 9 3 1 0x 4 1 5 7 1 Ethernet A 1 2x 6 8 2 9 3 10 4 1 5 7 1 C EMC Corporate I S 34mb pvc FA 1 FA16 Hopkinton DMX800 Cork Gigabit Layer 2 S witch Gigabit Layer 2 Switch Fibre Channel Gigabit RF2 RF15 RF2 RF15 FA16 FA1 DMX800 DS 16B2 DS16B2 Figure 2. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 10 WAN Testing To ensure the proper configuration of the WAN to support the use of SRDF/A, a latency test was performed to provide actual network latency results. SD Benchmarking was used to simulate a defined number of users and run transactions under these user IDs. For this particular test, we had a test cycle of five loops. We see the following utilization on the corporate PVC over the length of our testing. This testing shows us how the current network is performing before configuring SRDF/A: Figure 3. Note that this cannot be considered to be representative of all similar production customer environment as there are many factors which contribute to a load on the wide area network. These will include other uses of the WAN, the processing power of the SAP R/3 hosts, the number of users, the primary role of the SAP R/3 system, etc. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 11 Local Area Network (LAN) Network Assignment Five networks were required at each site to support the demonstration. These networks were: Name VLAN Hopkinton Cork admin 1 10.241.1.0/24 172.30.216.53 /24 public 2 10.241.2.0 /26 10.241.3.0 /26 private 3 10.241.2.64 /26 10.241.3.64 /26 dmx 4 10.241.2.128 /26 10.241.3.128 /26 corp 5 10.241.2.192 /27 10.241.3.192 /27 Network Description admin This was used to managed the host and for all initial installation and configuration of the OS, HBA, SAN connectivity, database, SAP/R3, etc. This network operates over 10/100Mb/s Ethernet to the local router. public This is the network which all user and client systems must use to access the nodes. The PRIMECLUSTER product is responsible for making IP aliases available for users to connect to the cluster, the provision of aliases is done independently on each node in our demonstration scenario. The concept of virtual IP addresses which failover from one node to the other is not used in this demonstration. To simulate a user system the DRIVER system is on this network so that it can drive/load the SAP R/3 application. This network operates over 10/100Mb/s Ethernet to the local router. private The cluster foundation component of the PRIMECLUSTER product specifies a private redundant network to provide a highly reliable communications channel between the two nodes. This is also commonly called the cluster interconnect. This network is used to send and receive status requests, heartbeat, updates, etc. between the two nodes. This network operates over 10/100Mb/s Ethernet to the local router. It is used to support the coordinated failover of the cluster from one node to the other. dmx The SRDF/A uses Fibre Channel over IP. To support the IP, a dedicated network is provided. This network operates over fiber and two interfaces are used from the DMX to the local router to provide performance and high availability. corp To ensure we have an IP route to the corporate WAN, and to ensure the corporat e WAN can make routing decisions, a separate network was requi red. This network will carry all the traffic, from the local networks, to the remote networks via the corporat e WAN. This network operates over 10/100Mb/s Ethernet from the local router to the corporate rout er Implementation Details 1. We will essentially ignore the admin network as its only function is to provide access for administration and is in no way required for operation of the cluster or data replication. 2. Each network is duplicated at both sites and the same host ID was used on each side. This is merely to ensure that the network configuration is intuitive and easily learned. 3. Each site uses a single class C network which has been subnetted to provide the required number of networks. The subnetting was chosen to ensure we had scalability in terms of adding more nodes, IP aliases to cluster, DMX systems, etc. 4. The use of a single class C network at each site also simplified the routing required to be configured on the corporate routers. They needed to know that the 10.241.2.0/24 network was in Hopkinton and the 10.241.3.0/24 network was in Cork. There is no need for the corporate routers to know anything about the local subnetted networks. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 12 5. The corp network was kept small as its only function is to provide connectivity to our corporate WAN. Normally only two IP addresses are requi red to provide this connectivity, one for the local network, and one on a corporate router. 6. A local router was used at each site. This served primarily as a switch to provide the VLAN functionality and to allow connectivity of 10/100Mb/s and 1 Gb/s Ethernet and 1 Gb/s fiber. The router also served to route all traffic via the corporate WAN destined to/from the remote site. 7. Virtual Private Networks were used to allow us to build the five IP networks on a single device. The VLANs also serve to reduce broadcast domains and to ensure that there is no cross traffic between the networks. This segregation also helped when troubleshooting network connectivity issues. Wide Area Network (WAN) To ensure a realistic demonstration and to provide a scenario close to real-world situations, the EMC corporate WAN was used to provide the WAN link between the sites. Due to the confidence EMC has in the operation of this new technology we had no concerns regarding deployment over our own business-critical infrastructure. Major WAN Components The EMC corporate WAN consists of a number of major components as follows: Router Cisco 7200 provides connectivity to corporate network from Hopkinton site ATM Switch Cisco 8510 provides ATM connectivity to the Hopkinton SONET network EMC Hopkinton Campus SONET provide ATM connectivity within Hopkinton area ATM Switch Cisco 8510 provides ATM connectivity to the Eircom SDH network Eircom ATM Net SDH/SONET provides transatlantic ATM connectivity ATM Switch Cisco 8510 provides ATM connectivity from Cork to Eircom SDH/SONET net. Router Cisco 7200 provide connectivity to corporate WAN from Cork site Implementation Details 1. The same issues that were addressed during the planning and implementation of this link are the same issues any organization will have when deploying their own WAN link. Due to the reliance on the WAN, for critical business operations and the high cost of downtime, the primary concern of the corporate network team was to protect the integrity and stability of the WAN. Before any connectivity was considered the corporate network team developed a solid understanding of the protocols to be used, the network configuration, and the expected levels of traffic. 2. It was very important to clearly identify the requirements of the PRIMECLUSTER and the SRDF/A products. 3. All clusters use some form of ping request (often referred to as a heartbeat) sent between the nodes to determine if the node is available. If a response to the request does not return within a specific time, the cluster assumes the remote node is not available. It was therefore important to know the upper limit in terms of roundtrip delay that the PRIMECLUSTER could tolerate. This was important because if the WAN could not provide the performance, then each node would decide the other is not available when in fact it is. The PRIMECLUSTER product was designed with global distances in mind and roundtrip delays of 200+ milliseconds can be tolerated. Our corporate WAN provided a roundtrip delay in the order of 95ms to 110ms. As part of the project Fujitsu Siemens Computer tested the operation of PRIMECLUSTER with a roundtrip delay in the order of 250ms, and the bandwidth required to support the communications was negligible. 4. A key factor in the project was how much data the SAP R/3 application generated to the local DMX, which would then need to be transferred to the remote DMX via the WAN link. This was an unknown, and to test this a WAN simulator was used in the early stage of the project to investigate various loads on the SAP R/3 system. The main objective of the simulation was to simulate the corporate WAN characteristics in terms of bandwidth, latency for a quality of service (QoS), and virtual private circuit service (PVC). EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 13 5. The last issue was that of ensuring the two cluster nodes and the DMXs could communicate with each other, and that the DRIVER benchmark system could communicate with both nodes. The task was to ensure the traffi c between the two sites and the local networks was possible. The options identified were: a) Bridge the local networks over the corporate WAN This option was ruled out due to the fact that introducing bridging traffic to a large corporate WAN can affect the stability of the entire network. The corporate network team was not prepared to accept this level of risk. b) Tunnel the local networks over the WAN using a VPN appliance This seemed to be the most obvious solution due to the fact that, once encapsulated, the corporate WAN just had to route traffi c from one part of the network to the other. Also we could have this specific traffic prioritized relative to existing traffic. The problem here was that it required an extra component in the architecture to troubleshoot, to maintain, etc. Also the fact that the need for encryption ensured that there would be an extra delay introduced in the transfer and the actual data transferred would increase due to the encapsulating protocols, authentication, integrity checks etc. required for VPN implementation. c) Route the local networks over the local WAN The routing option seemed to be the most practical but we were initially not sure of the effect this would have on the existing traffi c on the WAN. We were aware of the fact that the SRDF/A protocol could consume sizeable amounts of bandwidth if it needed it. This led us to investigating the use of PVCs and QoS to guarantee bandwidth for both our SRDF/A and ensuring a minimum required bandwidth for existing traffi c. Option (c) using a QoS was chosen as the most robust and practical solution. QoS Details EMC uses industry-standard marking techniques in conjunction with Cisco Systems proprietary queuing and congestion avoidance processes to provide QoS for the current EMC data services network. Implementation of QoS on the corporate WAN involved the following steps: 1. Configure the traffic priority policy a) Define the policy to identify which traffic has priority Traffi c IP Precedence (0-5, high priority to low) VOICE 5 HIGH 3 DEFAULT 2 LOW 1 SRDFA 0 b) Assign traffi c to each Label using IP source and destination or type of traffi c. The traffi c assigned to the SRDFA label included all traffi c flowing between the networks identifi ed in section c) Apply the policy to appropriate router interfaces 2. Configure the queuing policy a) Speci fy the queuing algorithm to be used. WFQ (Weighted Fair Queuing) is a flow-based algorithm that prioritizes and schedules traffi c into eight weighted queues based on the IP Precedence QoS settings. b) Speci fy the congestion-avoidance algorithm to be used. EMC use Weighted Random Early Detection (WRED), which is an enhanced version of RED that is aware of IP Precedence, where packets are dropped proportionally based on precedence values. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 14 Enterprise Application (SAP/R3) SAP R/3 is the leading enterprise management system in Europe. It provides subsystems for areas such as Human Resources, Manufacturing, Sales and Distribution, Purchasing, Finance, etc. SAP R/3 is transaction-based and requires a full relational database. All objects including users, printers, access rights, sales orders, invoices, etc. are stored in this database. The architecture of SAP R/3 does not necessarily lend itself well to clustering so it will serve as a good test of PRIMECLUSTER. The Sales and Distribution (SD) subsystem was chosen to be loaded during the demonstration. A tool provided by SAP was configured to allow an interactive load to be put on the cluster. Oracle provided the relational database functionality required to support SAP R/3. Driver System (SD Benchmarking) The DRIVER system is a Microsoft Windows 2000 system. This system is used to simulate interactive users accessing the SD component of SAP/R3. The tool IS called SD Benchmark (Sales and Distribution) and is used to simulate the login of multiple users and run any number of tasks on behalf of those users. This tool was configured to put a load on the SAP R/3 system before, during and after the failover. In this way we can show that the load is first processed by the primary node and after failure the load is processed by the secondary node. This load ensures that there is a realistic traffi c load on the SAN to the storage system (DMX), which generates a traffi c on the WAN over SRDF/A to the remote DMX. The DRIVER system is also used to monitor the status of the PRIMECLUSTER using the cluster PRIMECLUSTER Administration GUI. This is accessed using a standard Web browser and graphically shows the status of all components in the cluster, on both nodes. There are various colors to indicate OK, warning, or error status for all components in the graph. The DRIVER system is connected to the SAN and using EMC Symmetrix Administration software it is possible to manage the DMX. This functionality was used to establish, split, and query the status of all the volumes on the DMX. Cluster Software (Fujitsu Siemens Computers PRIMECLUSTER) The PRIMECLUSTER software suite is used to manage and monitor the availability of SAP including software and hardware components used by SAP. PRIMECLUSTER manages switching the SAP application and all the resources it uses, from the primary site to the secondary site when an uncorrectable problem occurs (e.g., node failure). It manages the execution of remedial actions when a component failure is detected that can be corrected on the local machine. The software and hardware components used by SAP are also monitored and managed by PRIMECLUSTER. This includes the Oracl e database, numerous filesystems, disks, network interfaces, and compute nodes. Strategy Application Availability An enterprise application has many dependencies and SAP R/3 is no exception. It requires: a database which is fully operational, file systems, network connectivity, an operating system, and storage. The objective is to build a system where the SAP R/3 system is available at all times. The practical option is to build a cluster. If there is a disaster at the primary site then there are two metrics to measure how available the system is. These are: Recovery Time Objective: RTO: This is the time it takes to get the application and users online at the secondary site. This objective is fully dependent on the nature of the business application, but there will always be a desire to keep this to a minimum. The objective in this demonstration was to keep the RTO to less than ten minutes. Recovery Point Objective: RPO: This is the earliest point in time, before the failure of the primary node, to which the application and data can be recovered. Simply put, how much data can be lost. The objective of this demonstration was to keep all data up to and including the last transaction performed on the primary node. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 15 Data Protection The SAP R/3 application was analyzed and it was determined that the most representative configuration was to provide local volumes which would be used to install the boot system, operating system, and any applications required. This includes Solaris, Oracle, and SAP R/3. These volumes should be unique to each node due to the fact that the OS, network, and application configuration and licensing are di fferent for each node. The volumes used to store the SAP R/3 database and log files will reside on the primary DMX and be accessed by the primary node. These volumes will then be synchronized with the data and log file volumes on the DMX at the secondary site. In this way both DMX systems will maintain a consistent and restartable copy of the SAP R/3 database and log files at all times. From a data protection perspective the policy was as follows; Operating System and Application volumes: • Maintain local copies of all local volumes. • Maintain copies of all local volumes at the remote site. Data, Log volumes: • Maintain local copies of all data and log volumes. • Maintain continuously consistent copy of all data and log volumes at remote site. Network Configuration Due to the nature of any disaster, the best policy with regard to networking was to use the same devices at both sites. This resulted in an intuitive network infrastructure which had total symmetry at both sites. This essentially means that the host part of all IP addresses for both nodes was the same. This also extended to the WAN configuration. Due to the fact that we were using a full-class IP address on either side of the WAN, it was a simple configuration change to ensure routing between these two networks. Putting It All Together SRDF/A Infrastructure The SRDF/A was configured to use Fibre Channel over IP. The gigabit Ethernet connectivity was provided by two Gig-E directors which were installed in the Symmetrix DMX systems. The use of two Gig-E directors is standard practice to provide redundancy at this level. The Gig-E switch was then used to provide the connectivity to the WAN. A corporate router was used to provide the routing between the primary and remote sites. This is a native IP infrastructure. Fibre Channel Fibre Channel over IP over IP over Gig-E over Gig-E DMX Gig-E SWITCH Gig-E SWITCH DMX WAN Gig-E IP Network Gig-E EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 16 PRIMECLUSTER Network Infrastructure public public private private These are the same Gig-E switches that are used for the SRDF/A infrastructure. We need IP networks for each. Support for multiple IP networks on the switch is supported by using VLANs. Logical Storage Architecture DMX: 0672 HOPKINTON DMX: 0363 CORK SRDF R1 R2 SRDF R2 R1 SRDF/A R1 R2 BCVs STD STD BCVs Use of Storage: Use Mount Point Solaris 8 / Oracl e Binaries /content Oracl e Data /content Oracl e logs /logs SAP /services Hopkinton Gig-E SWITCH Gig-E SWITCH Cork WAN hoplocal Corklocal hoprdf Corkrdf EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 17 DMX: 0672 Hopkinton Mount Point Device Slice / 1 s0 These are local and are /var 1 s3 not established to Cork. /opt 2 s0 /usr 2 s3 /services 3 s0 /logs 4 s1 /content 5 s1 Device Capacity Device Grp Volume ID BCV RDF (0363) 1 9GB hoplocal ED 33 1 2 9GB hoplocal EE 34 2 3 9GB hoplocal EF 35 3 4 16GB hoprdf B9 CB 39 5 50GB hoprdf AD BF 2D DMX: 0363 Cork Mount Point Device Slice / 1 s0 These are local and are /var 1 s3 not established to Hopkinton. /opt 2 s0 /usr 2 s3 /services 3 s0 /logs 4 s1 /content 5 s1 Device Capacity Device Grp Volume ID BCV RDF (0672) 1 9GB Corklocal E 11 E 2 9GB Corklocak F 12 F 3 9GB Corklocal 10 13 10 4 16GB Corkrdf 39 4B B9 5 50GB Corkrdf 2D 3F AD EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 18 Final Architecture Router Cisco 7200 ATM Switch Cisco 8510 ATM S witch Cisco 8510 Router Cisco 7200 ATM S witch Cisco 8510 management interface HOPKINTON CORK Network: ATM Capaci ty: Q0S A vailable: > 10Mb/s is pos sible up to a maximum of 34M b/s approx FSC Clus ter node FSC c lus ter node managem ent interfac e S D Benchmarking Driv er m anagement interface Fibr e Channel Switc h DMX 800 DM X 800 GIG-e Switc h GIG-e Swtitch GIG-e public cluster access private/cluster interconnect DMX Fibre Channel S witch public cluster access private/cluster interconnect FC FC FC FC FC public cluster access SONET SONE T WAN Fujitsu Siemens Computers cluster node Fujitsu Siemens Computers cluster node EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 19 Cluster Operation Preparation (both nodes are available) The primary objective of this stage is to demonstrate that the primary node is operational and is processing the SD Benchmarking load. At this stage the cluster GUI is used to monitor the status of the cluster showing that both nodes are available, but the secondary is not active. The cluster GUI clearly shows that SAP R/3, Oracle, Oracle network services, the OS file system and the storage is online on the Hopkinton node (primary) and offline on the Cork node (secondary). It should be noted at this stage that both the Hopkinton and Cork nodes are available to the cluster. This means that both are fully operational at the OS level and both have the Cluster Foundation subsystem. This means they are part of the cluster and the cluster can chose which one to make active. Site Failure (Hopkinton node is not available) We created a failure situation in the Hopkinton node (primary) by terminating electrical power to the server and disconnecting the remote network connection from the DMX storage subsystems. This is only one of many scenarios which PRIMECLUSTER can be configured to detect and respond to. At this stage all processing ceases on the Hopkinton node. At this stage also no processing can take place on the Cork node due to the fact that the cluster has not brought SAP R/3 online yet on that node. Site Failover (transition to Cork and bring Cork node online) In a very short time the secondary node detects the failure of the primary node and determines that it is the only available node in the cluster. At this stage the cluster will bring up each resource required in the correct sequence to bring the SAP R/3 application online. Due to the nature of SAP R/3 there is no possibility of maintaining process state between the two nodes, which means there is no way to automatically take users that were logged into the Hopkinton node and transfer their session to the Cork node. They simply need to restart their SAP instance. Using the cluster GUI it is easy to show that the primary node is not available and that the secondary is. It is also obvious what is happening on the secondary to bring the application online. The sequence has a lot of dependencies, but will generally follow the following sequence: 1. Bring storage online 2. Bring filesystems and network services online 3. Bring Oracle database online 4. Bring the SAP R/3 application online Once the cluster GUI shows that the application is online, we can use SAP GUI to demonstrate this. It can be clearly demonstrated from the DRIVER using SAP GUI logon that the SAP R/3 application can be accessed by users when steps 1-3 above are complete. Once the application is online we can run another SD Benchmark load from the DRIVER to ensure data is written to the local Cork DMX. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 20 Summary This demonstration has shown that global recovery of both information and infrastructure is achievable. When the primary node server fails for any reason then the Recovery Time Objective was in the region of 5 .5 to 6 minutes. The WAN used in this demonstration provides up to a maximum of 34Mb/s, however it should be noted that the full 34Mb/s could never actually be available as this is used for normal business use between the Cork manufacturing plant and the numerous Hopkinton plants. The success of the demonstration with this modest bandwidth further shows the efficiency of the SRDF/A Delta Set technology that enables significant bandwidth efficiencies. Primary Objectives: a) Failover of storage to secondary site is automatic. b) Storage at secondary site is guaranteed to be consistent and up to date. c) Storage at secondary site is guaranteed to be restartabl e. Primary Benefits: 1. An orderly transition of the application to the secondary site is achieved. 2. Data integrity is achieved before, during, and aft er transition. 3. Automatic implementation of management’s desired DR procedures without human intervention. 4. Full use of the secondary site is possible until it is required for DR. 5. The ability to fully and repeat edly test the DR architecture with confidence to ensure it will work when you need it most. By integrating EMC’s Symmetrix Remote Data Facility/Asynchronous (SRDF/A) with Fujitsu Siemens Computers’ PRIMECLUSTER technology, we have been able to demonstrate that unlimited distance application availability is now a practical undertaking. Organizations now have the ability to achieve metropolitan-class application availability across previously impossible distances. Through careful integration, PRIMECLUSTER exploits the features of EMC SRDF/A remote replication technology to meld the benefits of clustering and replication in a revolutionary manner. This integration enables a network effi cient unlimited distance application availability solution for UNIX and Linux environments. This distance-independent and application-transparent remote clustering solution has been developed, deployed, and demonstrated by EMC and Fujitsu Siemens Computers and is available today using generally available products from both companies. EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 21 Appendix DMX Configuration for Hopkinton and Cork Different modes of SRDF exist within the infrastructure. Only the data volumes from the SAP banking application are enabled for SRDF/A. Boot volumes are protected via local TimeFinder business continuance volumes and are transmitted via SRDF/A only when updates to the PRIMESERVER operating system occur. A look at the device groups on each side shows us the following: Hopkinton Hopkinton>symdg list D E V I C E G R O U P S Number of Name Type Valid Symmetrix ID Devs GKs BCVs VDEVs srdfa RDF1 Yes 000187900672 2 0 0 0 Hopkinton>symdg show srdfa Group Name: srdfa Group Type : RDF1 (RDFA) Valid : Yes Symmetrix ID : 000187900672 Group Creation Time : Wed Jul 16 17:00:32 2003 Vendor ID : EMC Corp Application ID : SYMCLI Number of STD Devices in Group: 2 Number of Associated GK's: 0 Number of Locally-associated BCV's: 0 Number of Locally-associated VDEV's: 0 Number of Remotely-associated BCV's (STD RDF) :0 Number of Remotely-associated BCV's (BCV RDF): 0 Number of Remotely-assoc'd RBCV's (RBCV RDF): 0 Standard (STD) Devices (2): { -------------------------------------------------------------------- Sym Cap LdevName PdevName Dev Att. Sts (MB) -------------------------------------------------------------------- DEV001 /dev/rdsk/c1t17d153s2 00B9 (M) RW 17263 DEV002 /dev/rdsk/c1t17d151s2 00AD (M) RW 51789 } Device Group RDF Information { RDF Type : R1 EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 22 RDF (RA) Group Number : 1 (00) Remote Symmetrix ID : 000287900363 R2 Device Is Larger Than The R1 Device : False RDF Mode : Asynchronous RDF Adaptive Copy : Disabled RDF Adaptive Copy Write Pending State : N/A RDF Adaptive Copy Skew (Tracks) : 65535 RDF Device Domino : Disabled RDF Link Configuration : GigE RDF Link Domino : Disabled Prevent Automatic RDF Link Recovery : Disabled Prevent RAs Online Upon Power ON : Enabled Device RDF Status : Ready (RW) Device RA Status : Ready (RW) Device Link Status : Ready (RW) Device Suspend State : N/A Device Consistency State : Disabled RDF R2 Not Ready If Invalid : Disabled Device RDF State : Ready (RW) Remote Device RDF State : Write Disabled (WD) RDF Pair State ( R1 <===> R2 ) : Consistent Number of R1 Invalid Tracks : 0 Number of R2 Invalid Tracks : 0 RDFA Information: { Session Number : 0 Cycle Number : 4489 Number of Devices in the Session : 2 Session Status : Active Session Consistency State : Enabled Tracks not Committed to the R2 Side: 0 Average Cycle Time : 00:00:30 Time that R2 is behind R1 : 00:00:50 } } Hopkinton> EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 23 Cork cork1:Sol8_0202:/> symdg list D E V I C E G R O U P S Number of Name Type Valid Symmetrix ID Devs GKs BCVs VDEVs srdfa RDF2 Yes 000287900363 2 0 2 0 cork1:Sol8_0202:/> cork1:Sol8_0202:/> symdg show srdfa Group Name: srdfa Group Type : RDF2 (RDFA) Valid : Yes Symmetrix ID : 000287900363 Group Creation Time : Fri Sep 5 14:40:30 2003 Vendor ID : EMC Corp Application ID : SYMCLI Number of STD Devices in Group : 2 Number of Associated GK's : 0 Number of Locally-associated BCV's : 2 Number of Locally-associated VDEV's : 0 Number of Remotely-associated BCV's (STD RDF): 0 Number of Remotely-associated BCV's (BCV RDF): 0 Number of Remotely-assoc'd RBCV's (RBCV RDF) : 0 Standard (STD) Devices (2): { -------------------------------------------------------------------- Sym Cap LdevName PdevName Dev Att. Sts (MB) -------------------------------------------------------------------- DEV001 /dev/rdsk/c1t0d50s2 0039 (M) WD 17263 DEV002 /dev/rdsk/c1t0d48s2 002D (M) WD 51789 } BCV Devices Locally-associ ated (2): { -------------------------------------------------------------------- Sym Cap LdevName PdevName Dev Att. Sts (MB) -------------------------------------------------------------------- BCV001 N/A 004B (M) RW 17263 BCV002 N/A 003F (M) RW 51789 } Device Group RDF Information { RDF Type : R2 EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 24 RDF (RA) Group Number : 1 (00) Remote Symmetrix ID : 000187900672 R2 Device Is Larger Than The R1 Device : False RDF Mode : Asynchronous RDF Adaptive Copy : Disabled RDF Adaptive Copy Write Pending State : N/A RDF Adaptive Copy Skew (Tracks) : 65535 RDF Device Domino : Disabled RDF Link Configuration : GigE RDF Link Domino : Disabled Prevent Automatic RDF Link Recovery : Disabled Prevent RAs Online Upon Power ON : Enabled Device RDF Status : Ready (RW) Device RA Status : Write Disabled (WD) Device Link Status : Ready (RW) Device Suspend State : N/A Device Consistency State : Disabled RDF R2 Not Ready If Invalid : Disabled Device RDF State : Write Disabled (WD) Remote Device RDF State : Ready (RW) RDF Pair State ( R1 <===> R2 ) : Consistent Number of R1 Invalid Tracks : 0 Number of R2 Invalid Tracks : 0 RDFA Information: { Session Number : 0 Cycle Number : 4498 Number of Devices in the Session : 2 Session Status : Active Session Consistency State : Enabled Tracks not Committed to the R2 Side: 0 Average Cycle Time : 00:00:30 Time that R2 is behind R1 : 00:00:47 } } cork1:Sol8_0202:/> EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER 25 Definitions Used in this Document RMS Reliant Monitoring System. This is a major component of the PRIMECLUSTER product. It is responsible for monitoring resources and initiating actions to repair the primary system or to failover to the secondary system. SRDF/A Delta Set ATM Asynchronous Transfer Mode. This is a networking protocol used to provide high levels of services on a network. It is primarily used in wide area networking applications. References EMC: www.EMC.com FUJITSU SIEMENS COMPUTERS: www.PRIMECLUSTER.com