Add your Article
Business Recovery Overview
By CPACs, LLC Published: 07/07/2009
TABLE OF CONTENTS
Introduction 1
Responsibilities of the Business Recovery Coordinator 2
Budget Considerations 2
BRC Plan Development, Testing, and Maintenance Functions 3
Status Reporting Function 4
Liaison Function 4
Overview of BRC Functions During a Recovery 4
Establishing a Recovery Control Center 5
Advisor to Management 6
Liaison To Recovery Teams 6
Recovery Plan Training Program Overview 7
Orientation Training 7
Recovery Team Training 8
Training Schedule 9
Recovery Plan Testing Program Overview 10
General Testing Guidelines 10
Component Testing 11
Test Components 12
Integrated Testing 14
Recovery Drill Testing 18
Budgeting for Testing 18
Recovery Plan Maintenance Overview 20
Scheduled Maintenance 20
Unscheduled Maintenance 21
Recovery Plan Manual Distribution 23
Introduction
Business Resumption Planning, Disaster Recovery Planning, Business Continuity Planning, Corporate Contingency Planning, Business Recovery Planning - any of these terms can be used to describe the process of developing, maintaining and testing business recovery plans to ensure the survival of an organization following a serious disruption of its operations. The term "business recovery planning" will be used throughout this topic, which describes the functions and responsibilities of a Business Recovery Coordinator (BRC). The BRC is responsible for managing the business recovery planning process, and for coordinating recovery activities after a serious disruption to operations. Working with the managers of all of the business processes that are included in the plans, the BRC will develop procedures and policies that address recovery planning administration requirements. The specific functions and responsibility of a BRC will vary depending upon the size of the organization, the scope of the business recovery planning process, and the planning methodology that is selected. The recommended approach to business recovery planning is to assign responsibility for the development and maintenance of each segment of the plan to the recovery team leaders and members who are responsible for restoring the business processes after a disaster. The BRC, working with the recovery organization throughout plan development, maintenance, and exercises of the plans, will have the most thorough knowledge of all of the plans. Since there is no guarantee that the BRC will be available when the plan is activated following a serious disruptive incident, an alternate BRC must be appointed. The alternate BRC must also participate in all business recovery plan development, maintenance, and testing activities. This is very important since relying upon one person to have thorough knowledge of the organization's plans is extremely unwise.
Business recovery planning is not a trivial process. Development of a business recovery plan for just one business process can take several months. Development of enterprise-wide business recovery plans, a complex process involving many departments and business processes, will generally be accomplished over a period of at least nine months, possibly a year or more. Business recovery planning is not a one-time project. It is an on-going process, and it must become an integral part of the management of the organization's operations. Changes made within the organization must be incorporated in the business recovery plans. The impact of a proposed major change upon the organization's ability to survive after a serious disruption of its business processes should always be considered before the change is implemented.
The need for efficiently managing a process as complex as business recovery planning is self-evident. The BRC is responsible for administration of the entire business recovery planning process, and will also serve as the focal point during recovery from a serious disruption of the organization's business processes. Enterprise-wide business recovery planning generally requires a full-time BRC. The BRC must have a thorough understanding of all of the information contained in the plans.
Responsibilities of the Business Recovery Coordinator
The primary responsibilities of the BRC are:
business recovery plan development, testing, and maintenance;
business recovery team member training;
business recovery awareness training for all employees;
recovery coordination and liaison functions during a recovery.
Budget Considerations
Management must allocate funds for a comprehensive business recovery program. Some of the expenses that must be included in the budget are:
Alternate Sites: Monthly subscription fees and recovery activation fees for alternative sites.
Off-Site Storage Sites: The cost for maintaining off-site facilities for storing critical records and emergency supplies that will be required for recovery.
Telecommunications Recovery Network: The cost of establishing or maintaining recovery telecommunication networks.
Testing: The cost of testing business recovery plans, which includes the cost of alternate site tests, personnel, equipment usage, supplies, transportation, lodging and meals, special materials, and off-hour access to critical records stored at an off-site locations.
Business Recovery Planning Software: The one-time cost to license business continuity planning software, such as HaXers RecoveryPAC, and the annual maintenance fees associated with the software license.
Training: The cost of providing business recovery training and RecoveryPAC product training to recovery team members and other employees.
Training Seminars, and Conferences for BRC Education: Since business recovery planning and recovery management expertise is essential to the organization, funds must be allocated to ensure that the BRC is familiar with the latest business recovery planning methods and tools. The BRC should attend one or more business recovery planning training sessions, and should be certified as a qualified business recovery planning specialist. The budget should include funding for the BRC to attend at least one business recovery planning conference or seminar each year. Note: In addition to allocating funds for the BRC and an alternate BRC, enterprise-wide planning may also require funding for training administrative or support staff.
Business Recovery Planning Consulting Services: Development of enterprise-wide business recovery plans is a lengthy and complicated process. If the BRC has not had previous recovery planning experience, the process of recovery planning may suffer. Experienced consulting firms can provide training and guidance to the BRC. If development of plans, or modification of out-of-date plans is a high priority project, and the organization's employees do not have sufficient time to devote to the business recovery planning process, the plans will not be completed in a timely manner. Allocating budget funds for experienced business recovery consulting services should be considered.
BRC Plan Development, Testing, and Maintenance Functions
The primary responsibility of the BRC is to develop and maintain comprehensive, tested business recovery plans so that an organization can continue to function following a disruption of any magnitude. The BRC also coordinates periodic plan testing, and ensures that employees receive training regularly to promote enterprise-wide awareness of the business recovery plans and the recovery responsibilities of all employees. The BRC develops administrative programs that encompass all aspects of the business recovery planning process. The BRC should:
Coordinate or perform a risk assessment, coordinate or perform a business impact analysis, and manage the development of business recovery plans based upon the results of the business impact analysis.
Coordinate tests (or exercises) of the business recovery plans, report test results to management, and update the plans as required to correct any deficiencies found during the tests.
Establish a business recovery training program to ensure that employees are aware of the planning process, and their responsibilities with respect to implementation of the plans.
Print and distribute individual recovery team plans and a complete, consolidated business recovery manual.
Establish a maintenance program that ensures that plans are updated on a scheduled basis, as well as when organizational changes that have an effect on the business recovery plans occur.
Act as a liaison to all business processes whose functions are included in the business recovery plans, as well as to support functions such as Human Resources, Finance, and Facility Services.
Meet regularly with recovery teams to review their recovery plans, and verify that recovery activities, resources, and procedures are accurately defined and documented in the plan.
Maintain contact with government agencies that may be involved following a serious disruption of the organization's operations.
Provide input and support to the business processes for projects that relate to business recovery planning. For example, updating documentation, creating procedures, evaluating security systems.
Research, evaluate, and recommend internal and external solutions for recovery related issues.
Assist in maintaining contracts for alternate facilities and services.
Status Reporting Function
Management makes a substantial commitment of resources to develop and maintain comprehensive, tested business recovery plans. The BRC is responsible for providing periodic status reports to management with respect to development, testing, and maintenance throughout the business recovery planning process.
Yearly Planning Cycle: Once the plan is developed, establish a yearly planning cycle that outlines training, testing and maintenance functions. Report briefly each month about active and completed projects and any difficulties that are impeding progress.
Monthly Reports: Prepare monthly reports about significant problems that could interfere with efficient implementation of business recovery plans. For example, equipment that has become obsolete and would be difficult or impossible to replace; operating procedures that are not routinely maintained as business processes change; inadequate off-site storage policies for critical records.
Project Status: Report on projects that the BRC has initiated. For example, documentation of procedures, or establishment of off-site records rotation programs for critical records.
Regulatory Status Reporting: Complete regulatory status reporting about business recovery planning as required.
Liaison Function
Liaison to Auditing: In many organizations, internal or external auditors are responsible for reviewing business recovery plans and determining if the plans are adequate to protect the organization's assets. The auditors should have frequent input throughout the business recovery planning process to ensure that the completed plans meet the auditor's requirements, and thus avoid extensive rework.
Liaison to Business Process Managers: The BRC acts as the liaison to the business process managers and employees who are included in the scope of the business recovery plans.
Liaison to External Organizations: The BRC acts as the liaison to the vendors that supply business recovery services and products.
Overview of BRC Functions During a Recovery
The BRC is the most knowledgeable person in the organization about the business recovery process and plans. Therefore, the BRCs value during a recovery will be as an advisor and liaison to management, recovery team members, support staff, and vendors. The primary functions that a BRC will perform during a recovery are:
Recovery Team Leader Notification: Upon being notified of a serious disruption of operations, the BRC contacts the Recovery Team Leaders to inform them of the situation.
Recovery Control Center Establishment: Aiding and advising in the efficient establishment of a Recovery Control Center.
Advisor to Management: Advising management about the details and use of the business recovery plan, the responsibilities of recovery team members, support services, vendors - and about recovery progress.
Liaison to Recovery Team Members: Working with recovery teams to enhance coordination, resolve problems, and facilitate the acquisition of the resources that are required.
Reminder: The alternate BRC must participate in all aspects of the business recovery planning program in order to be able to substitute for the BRC if necessary.
Establishing a Recovery Control Center
The BRC may be assigned to the task of establishing a Recovery Control Center during recovery from an extensive disaster. A Recovery Control Center (or Command Center) is the focal point for communications between all of the people who are involved in a recovery. The location of a Recovery Control Center, and the resources that are required to operate it, must be determined as an integral part of the business recovery planning process. The BRC continues to assist in the Recovery Control Centers operations throughout the recovery period, coordinating recovery operations between management, business recovery teams, vendors, and alternative sites.
Some of the activities that must be addressed when establishing a Recovery Control Center are:
Select a location for the Recovery Control Center from the sites that have been identified in the recovery plans. Contact the selected location and arrange for the establishment of the Recovery Control Center.
Retrieve resources from off-site storage locations.
Contact the team leaders of the teams that will support establishment of the Recovery Control Center.
Travel to the site and establish a general message board, recovery status charts, recovery team status boards, recovery team member location status board, and any other status boards and charts that may be required.
Distribute supplies and telephone logs to Recovery Control Center team members.
Activate the RecoveryPAC business continuity planning software tool on a notebook PC.
Note: Recovery Control Centers are often established at a commercial site, such as a hotel, immediately after a serious disruptive event. If possible, several locations that are appropriate for establishing a Recovery Control Center should be identified. For example, identify facilities that are located within one, three, and five miles from the primary site. The cost of maintaining a commercial Recovery Control Center is not insignificant. It may be advisable to reestablish the Recovery Control Center at a non-commercial location when the immediate emergency period has passed.
Advisor to Management
The BRC coordinates and leads the development and maintenance of all portions of the business recovery plan, and therefore knows each element of the plan and how the plan coordinates the work of all recovery personnel during a recovery. This understanding of both the individual elements and the overall functions of the plan uniquely qualifies the BRC to advise management throughout a recovery effort. Using the business recovery plan and information provided by recovery personnel and service providers, the BRC provides management with clarification of information presented in the plan and the status of the recovery efforts. The BRC can identify problem areas and, if necessary, seek guidance and support from management to resolve the problems.
Liaison To Recovery Teams
Given that the recovery team members have been involved during the entire process of developing, testing and maintaining the business recovery plan for the business processes for which they are responsible, they will be very aware of their own responsibilities. During integrated tests of the recovery plans, team members will also gain knowledge about the responsibilities of other recovery teams. Since the BRC has the most extensive knowledge of the recovery plans, the BRC can serve as liaison to the recovery teams, management, and support personnel. The BRC is responsible for overall progress of the recovery, and will monitor the progress of all of the people who are involved. Problems resulting from coordination of recovery team and vendor activities can be resolved by the BRC. Communications will be facilitated by using the BRC and Recovery Control Center as the focal point for many of the communications between the recovery organization, vendors, and management.
Recovery Plan Training Program Overview
The training programs suggested in this document are designed to familiarize personnel with the business recovery planning process, and to train recovery team members to develop, maintain, and activate a business recovery plan. The BRC is responsible for developing a training program, publishing a training schedule, and providing both scheduled and unscheduled training.
Orientation Training: The orientation training program is designed to provide a basic understanding of the need for business recovery planning and an understanding of the business recovery planning process. Orientation training should be provided to management, recovery team members, and employees who, while not involved in developing, testing, maintaining, or activating business recovery plans, need to understand the concepts of and reasons for recovery planning.
Recovery Team Training: In order to develop recovery plans that will work as advertised, recovery team members and their alternates must participate in plan development, testing and maintenance. During plan development the BRC is responsible for training the recovery teams and ensuring that a transfer of knowledge occurs. Scheduled recovery team training sessions help the teams to focus on the plans and their responsibilities. Distributing the training session schedules to all business process managers will reinforce the need for training. It will also serve as a reminder that there are scheduled training sessions, thus reducing or eliminating the need for unscheduled training sessions.
Scheduled Training: Depending upon the size of the organization, and the scope of the recovery planning process, it may be appropriate to provide training classes on a scheduled basis. This will be especially important when major changes that impact the business recovery plans occur. The schedule should be published and sent to all of the managers of the business processes that are included in the recovery plans.
Unscheduled Training: Business process managers should request orientation training for new employees. The BRC will provide the training as soon as possible
Orientation Training
Recovery planning orientation training is designed to familiarize employees with the business recovery planning process, and with their roles and responsibilities with respect to business recovery following a serious disruption of operations. The BRC is responsible for conducting orientation training sessions, and creating and updating training materials. The main points to be discussed during orientation training are:
The definition of a disaster: A sudden calamitous event bringing great damage, loss, or destruction (Webster's Dictionary).
The definition of business recovery planning:
- Development of a plan of action that protects the organization's assets and safeguards employees by providing a means for continued operation after a disaster;
- Identification of the most expedient method of recovery from a disaster;
- Documentation, prior to a disaster, of contact information for the recovery organization and vendors, all recovery procedures and tasks, required resources, critical records, and other information required to recover from a disaster.
The main elements of the business recovery planning process.
The business recovery team concept, and an overview of team responsibilities.
An overview of the business recovery plan.
Business recovery procedures and flowcharts.
Business recovery plan training, testing, and maintenance requirements.
Recovery Team Training
Recovery team training is conducted for recovery team leaders, members and alternates. The training sessions are held with individual teams or with groups of teams. The BRC is responsible for scheduling and organizing training sessions and providing the attendees with all required materials.
New recovery team members: Hold a training session to familiarize new team members with the business recovery planning process, the functions that the teams are responsible for recovering, and the specific tasks that will be assigned to each member. The following points should be addressed during recovery team training for new team members.
Discuss the key points of business recovery planning, maintenance and testing.
Distribute completed recovery team plans to the new members. The complete, consolidated business recovery plan manual should also be given to new recovery team leaders.
New or updated plan information: Hold an informal meeting to present new or updated information to recovery teams affected by changes. To save time, discuss changes that apply to all teams at a meeting attended by all teams. (Generally, if one team member has been replaced with another person, the plan will have to be updated to transfer responsibilities from the previous team member to the new team member.)
Test preparation: Conduct individual or group team training in preparation for a test. Discuss the following points in these training sessions:
1. Review test objectives.
2. Review team functions, tasks, resources and procedures.
3. Review other portions of the plan that may be used during the test.
Remember that the alternate BRC and recovery team leaders and members should be involved as much as possible in recovery plan development, testing, maintenance, and training. This helps to ensure that the business recovery plans can be executed by alternate team members if necessary.
Training Schedule
A Training Request and Training Schedule form should be completed by the business process managers when staff training is required. Training request guidelines:
All requests for training should be made at least one month prior to the requested training date if possible.
Employees should attend a regularly scheduled training session if possible.
The BRC should notify the manager of the date and time of the training session. Training sessions should take approximately two hours.
Recovery Plan Testing Program Overview
A viable business recovery plan is one that is maintained and tested, or exercised, regularly. An out of-date or untested plan is not much better than no plan at all since it cannot be relied upon to work when needed. In a controlled and monitored environment, regularly scheduled Business Recovery Plan exercises validate the accuracy of information contained in the plans, provide experience for recovery teams, and ensure that plans are revised as necessary. During the tests, all of the employees and vendors who will be involved in recovery from a serious disruption of operations perform the activities that have been assigned to them. The Business Recovery Coordinator is responsible for the recovery plan testing program. The BRC should recommend testing schedules, develop scenarios for the test, develop test objectives, develop the strategy for the test, and provide post-test evaluation forms. It is important to remember that the word "test" does not imply pass or fail. There are no grades - the purpose is to exercise the plan to determine if:
Critical records (including computer media), supplies, and other materials that are stored off-site are adequate;
The Recovery Organization understands the plans, and is ready to respond appropriately;
The business recovery teams have identified the tasks and procedures that are required to recover mission critical business processes and the computer application systems that support these business processes;
All of the information contained in the plans is accurate and up-to-date;
The activities required for an effective recovery can be accomplished in the required time.
Testing is a major component of an on-going business recovery program. It involves a commitment of time, money, and resources across the entire organization. Despite the considerable burden that testing recovery plans places on the organization, it is difficult to overstate the value of a comprehensive testing program.
General Testing Guidelines
Periodic review (test, exercise) of recovery plans is required to ensure that the plans will work as intended when a disaster occurs. Recovery plan testing is generally done incrementally. Phasing the tests ensures that components are tested before the entire plan is tested. Since "walk through" or "desk top" tests usually involve a single Recovery Team, testing can begin as soon as a component of the plan is completed. "Live" tests can be quite expensive and are not a trivial exercise. Conducting a "live" test should not be undertaken until the plans are completed and have been tested in other phases.
Decide which testing phase to use for the test.
At least two weeks in advance of the test, notify test participants of the test date. Tell them what type of testing is planned. Remind participants to bring their copy of the recovery plan with them. Have extra copies available to distribute if necessary.
Develop test objectives, plans, and scenarios.
Conduct and monitor the test.
Document problems as they occur, document telephone calls, document the times required to execute the recovery activities.
Evaluate the test.
Provide a written evaluation of the test.
Develop maintenance schedules and modify the plans based upon the results of the test.
The recovery organization gains knowledge and confidence through exercising the recovery plans. The testing program must be simple at first to build a solid foundation, and then gradually become more detailed. This section provides a three phased, progressive testing methodology that moves from testing individual elements of the plan to a comprehensive recovery drill.
Testing Phases:
Component Testing: The first phase of the testing program is to test the individual elements that comprise the total plan.
Integrated Testing: The second phase of the testing program combines a number of related elements of the plan in the order that would occur during recovery operations. Integrated testing starts with simple tests and progresses to more complex tests.
Recovery Drill: The third phase of the testing program tests and exercises the entire Business Recovery Plan, and is scheduled only after thorough component and integrated testing. This phase is sometimes called a "live" test since relocation to a hot site may be required.
Depending upon the scope of the plans and the time required to develop the plans, all three testing phases may overlap. For example, component testing of an Information Systems plan may begin before plans for a Finance Department have been developed.
Remember - the testing process is a learning process and an opportunity to revise and improve upon recovery plans. No test is a failure! Frequent testing also increases enterprise-wide awareness of the Business Recovery Plan.
Exercise the recovery plans during peak load times as well as during non-peak load times.
Component Testing
Component testing focuses on the parts of the Business Recovery Plan that can be tested individually. Recovery team members gain a greater understanding of the entire plan by testing its individual components. Component testing also prepares the recovery organization for the more complex integrated testing, and will generally have less of an impact on operations.
Component testing is often conducted using a "walk through" scenario. Procedures are examined, such as those required to accomplish recovery team tasks, transportation of personnel and supplies, or off-site storage of critical records and materials. Information contained on checklists, telephone lists, inventories, vendor lists and other support resources can be verified for accuracy.
Component tests should be accomplished frequently since components are constantly changing. Although component testing is conducted in the first phase of a testing program, it continues once successful integrated testing is achieved.
Test Components
Some of the components of a recovery plan that can be tested individually are:
The Ability to Establish a Recovery Control Center:
Are the addresses of the Recovery Control Centers correct? Are the contact numbers and the names of the individuals who are to be contacted correct?
Are required resources available? Or- are procedures in place for acquiring the resources?
Is the Recovery Control Center equipped with appropriate furnishings? Telecommunications?
Recovery Plan Activation Notification Procedures:
Have the management recovery team leaders and functional team leaders make telephone contacts as specified by the recovery plans.
Have callers state that the call is part of a test, and verify business and home telephone numbers, facsimile numbers, pager and mobile telephone numbers, and E-mail and Internet addresses.
Establish time parameters for the test.
Off-site Storage Requirements:
Using the RecoveryPAC off-site storage checklist report, confirm that the list of required resources that are supposed to be stored off-site is correct.
If possible, go to the off-site storage locations and verify that the required items are actually stored off-site, and are in good condition. If this isn't possible due to the time and cost associated with travel to the off-site storage facility, for example, a commercial off-site storage site in a remote location, arrange with staff at the off-site storage location to verify the accuracy of the items stored.
The Availability of Supplies, Equipment and Services from Internal and External Sources:
Contact internal support sources to validate telephone numbers, contact information, and supplies that are required.
Contact vendors to validate contract information, and supplies and services offered by the vendors
If letters of intent or contracts have been agreed with vendors, review the commitments for accuracy and feasibility. If there are no letters of intent or contracts in place, request confirmation of support from the vendors in the event that they are called upon for assistance. Document the replies. (Obviously, if a vendor can no longer meet a specific requirement, alternative plans will have to be made.)
The Ability of the Support Teams and Support Coordinators to Meet Expected Levels of Support:
Note: Support departments typically provide support for business functions. Human Resources, Finance, Information Systems and Facility Management departments are examples of support functions. For smaller organizations, it may only be possible and necessary to have support coordinators rather than a full recovery team. Review the following items with the recovery teams and the support teams or coordinators:
Total quantity of critical supplies available in inventory, or available as required through reorder processes
Availability of funds required during a recovery
Preapproved procedures for acquiring assets during a recovery
Ability to acquire replacement staff if necessary
Alternate Site Requirements:
Verify contact information: names, telephone and facsimile numbers, email addresses and names of alternative contact people. Make sure that the agreements allowing use of alternate sites are documented. Review the site to make sure that it is still appropriate for recovery operations. Make sure that the time allowed for occupying the site and the agreed provision of resources (i.e., equipment and supplies) are sufficient to meet emergency requirements.
Tests should be performed if specific production, office, or computer equipment is involved. This ensures compatibility with backup equipment, and with equipment with which the backup equipment must interface. For example, Information Systems could perform the following component tests:
Start up the alternate site CPU and load the operating system and utility software.
Load critical application systems; test batch applications.
Test the telecommunication systems; process on-line applications.
The Acquisition of Alternate Services and Resources:
Verify the accuracy of contact information - names, telephone numbers, facsimile numbers, E-mail and Internet addresses. Agreements that describe services to be provided by vendors should be reviewed. The cost of the service and any other conditions or specifications related to the delivery of services should be verified. Verify that an authorized vendor representative has signed vendor agreements and contracts, or letters of intent.
Can the services or resources be provided in the time required during an emergency?
Has an alternative vendor been identified, or is this a sole source situation? If a vital resource can only be sole sourced, it may be appropriate to store the resource off-site.
For resources that are generally easily obtained without formal agreements or letters of intent, are the sources documented?
Integrated Testing
The second phase of the testing program integrates a number of related components in the order that they would occur during actual recovery operations. Integrated testing builds on the experience employees have gained during component testing. While integrated testing can begin before completing all component tests, follow the guidelines in this section when beginning an integrated testing program.
The increased complexity of integrated testing, involvement of team members and support personnel, as well as budget considerations, may limit the frequency of integrated testing. However, the benefits of well-planned integrated tests are great. At a minimum, integrated testing should be conducted annually.
Follow these guidelines when planning an integrated test:
Keep integrated testing simple at first, and then gradually increase the complexity with each test. We recommend this approach because it gradually familiarizes the recovery organization with the entire plan.
Involve auditing in integrated testing. Auditing may assist in establishing objectives and determining what should be tested from an organizational perspective. By encouraging auditing to issue an endorsement of the test objectives as well as a post-test report of testing procedures and results, enterprise-wide awareness of the plan is enhanced.
Conduct pre-test planning meetings with recovery teams. This increases the chances of test success, especially when test schedules are being established. Personnel representing departments that may be impacted by the plan components that are included in the test should also be present. Schedule integrated testing at a time that will minimize the impact on operations if possible.
Note: It may also be advisable to schedule a test during times when the workload is heaviest. Obviously this sort of testing has to be very carefully planned, and should not be done until non-peak load testing has been accomplished, and problems that have been identified are corrected.
Determine test objectives well in advance of a test. Objectives must be concise and realistic. Include projected test times, including breaks, with the objectives.
Remind participants that the status reports and progress logs must be completed during integrated tests. During testing, encourage participants to note in status reports and progress logs the portions of the plan that can be improved.
You could develop scenarios that focus on all of the types of natural disasters that might possibly occur in your organizations geographical area, as well as disasters scenarios for fire, terrorist attacks, bombings, equipment breakdown - etc. However this isn't really necessary. The focus should be on how to recover from a disaster, not on the event that caused the disaster. One exception to this rule is a test that includes recovery teams that are responsible for restoring a damaged facility to operational status. In this case it may be appropriate to develop scenarios, or portions of scenarios, that will test the availability of specific types of service vendors - such as vendors that restore or recover critical records damaged by water, or vendors that specialize in cleaning smoke and chemical residue from a facility following a fire.
Create realistic recovery scenarios for distribution to the business recovery teams and support staff during testing activities. Present two kinds of scenarios:
1. A general scenario that briefly describes the disruption. Have the teams respond to the scenario and determine what course of action to take.
2. A specific scenario for the business recovery teams that are participating in the test. You may require that the teams make contacts with external sources to gather information needed in order to complete the test. Using effective, realistic scenarios can generate discussion on aspects of the plan that might be overlooked if very simple or unrealistic scenarios are used.
Since a number of sequential components may be combined for an integrated test, it is not possible to list all of the possible combinations. The following examples utilize several components:
Walk through Integrated test: The first test conducted after plan development is a walk through of selected components. Follow the guidelines listed in the previous topics to create a test.
Select portions of the plan to test.
Establishing a Recovery Control Center
Notifying test participants, using the call lists included in the business recovery plan
Arranging transportation to the alternate sites
Completing the progress logs
Selecting a recovery strategy
Having team members contact vendors
Completing status reports
Arrange for the location of one or multiple Recovery Control Centers; ensure that telephones are available.
Establish test objectives and review the objectives with the auditors, if appropriate.
Create disaster scenarios that are relevant to the organization.
Provide copies of status logs, checklists, scenarios, and objectives to be distributed during the test.
Determine number and names of test personnel, usually all recovery team members or their alternates.
If the test involves the participation of the entire recovery organization, try to schedule the test for a maximum of four hours to minimize impact of the test on the organization.
On test day, distribute scenarios and allow team members to interact to address the scenarios.
Make as many telephone calls to vendors, service providers, travel agents, alternate sites, etc., as possible.
Emphasize that the purpose of the exercise is to allow recovery personnel to practice using the plan to solve a problem, and to identify areas of the plan that require improvement. No one is being graded, it is not a test of the participants - it is an exercise of the plans.
Information Systems Alternate Site Tests: The test required by a centralized computer center, especially mainframe operations, must be comprehensive to ensure effective use of the alternate site. Due to the complexity of these tests, they must be accomplished step-by-step, building on component tests. Select portions of the plan to test.
Procedures to obtain computer media from off-site storage
Procedures for establishing the Recovery Control Center
Procedures to gain access to the alternate site
Procedures to load systems software
Procedures to load and process application systems
Procedures to establish telecommunications services
Procedures to distribute printed output, or to print the reports at a remote location
Establish test objectives based on the portions of the plan being tested; review the objectives with audit, if appropriate.
Arrange for test time at the alternate site.
Conduct pre-test training meetings, emphasize that the test is also intended to familiarize recovery teams with the alternate site.
Select a few critical application systems to be tested. Repeat tests can also be expanded to include some of the less critical applications. It may also be possible to include full on-line testing.
Select team members who will travel to the alternate site.
Inform test participants of test objectives.
Assure that logs, checklists, and scenarios are available for distribution during the test.
Provide transportation for recovery team members and supplies.
Conduct and monitor the test at the alternate site.
Review reports generated during the test.
Expanded Information Systems Alternate Site Test: Based on the success of the tests that were designed to familiarize participants with the plan and the alternate site, you can now perform expanded alternate site testing. Observe the following guidelines:
Plan an initial test involving all team members, combining elements of prior tests.
Expand the number of application systems to be processed at the alternate site.
Include establishing the on-line network in the test.
Create scenarios that involve processing at the alternate site. Select personnel going to the alternate site during the test.
Establish objectives, schedule the test, and arrange transportation for personnel and supplies.
Conduct the test by testing all of the components that can be tested at the Recovery Control Center, and then proceed with testing of components at the alternate site.
Review reports generated during the test.
Alternate Team Members Test Participation: During recovery following a serious disruption, one or more of the recovery organization members may not be able to fulfill their responsibilities for one reason or another. It is also possible that recovery at an alternate site may continue for a long period of time, perhaps 60 days. In this case it will be necessary to relieve the primary team members periodically so that they can return home. The alternate team members are the logical choice to relieve the primary team members. For these reasons it is advisable to involve the alternate team members in recovery plan exercises to the extent possible. If it is not possible to include both the primary team members and the alternate team members at the same time, the exercises should be repeated with some or all of the alternate team members replacing the primary team members.
Recovery plan exercises expose a large number of people to the recovery process. The plan is reviewed, enhanced and revised over time, contributing to its accuracy and completeness. Component and integrated testing should be done periodically so that new and changed business processes and functions can be tested.
Recovery Drill Testing
This last phase of the testing program should only be attempted after you have completed extensive component and integrated testing. Recovery drill testing tests the ability of participants to use the plan without prior warning. It tests the skills that participants have refined during component and integrated testing. The following points should be addressed prior to the unannounced test:
Assure that only a few people, such as senior managers and auditing personnel, have prior knowledge of the test.
Using the guidelines from the previous topic, establish measurable objectives.
Using the guidelines from the previous topic, create extensive, realistic scenarios to support the test objectives.
Select a test date that minimizes the impact on normal operations, unless testing during peak times is a test objective.
Assure that funding to accomplish the objectives is available prior to test activities.
Budgeting for Testing
At a minimum, consider the following costs when planning integrated tests that require travel to alternate sites:
Cost of transporting recovery team members and materials to the alternate site
Costs for shipping materials, including insurance to cover loss or damage
Cost of lodging, meals and local transportation while at the alternate site
Cost of supplies
Testing fees charged by the alternate site vendor
Overtime wages for recovery team members, if applicable
All of these cost items should be addressed prior to the actual test.
Recovery Plan Maintenance Overview
Once developed, business recovery plans must be maintained. Changes that are internal to the organization, as well as changes that are external to the organization, may affect the business recovery plans. In order to be sure that the plans will work when they are needed, it is necessary to develop an effective maintenance program. If possible the business units should perform the maintenance tasks and departments that are included in the scope of the business recovery plan. This distributes the workload of maintaining the plans, and generally ensures that the plan will remain current. When development of an enterprise-wide business recovery plan extends over several months, it is possible that some elements of the plan will require testing and maintenance before other portions of the plan have been completed. Some changes to the plan are the result of omissions, errors, or inconsistencies found when the plans are tested. Other changes are required by changes within the organization. And still other changes result from forces that are external to the organization, such as changing technology and government regulations.
Two types of Recovery Plan maintenance programs should be developed:
1. Scheduled Maintenance
2. Unscheduled Maintenance
The BRC must develop a program for scheduled maintenance, as well as a methodology for performing the maintenance. The maintenance schedule should include the dates when changed information is due, which information is due at each scheduled date, and who is responsible for ensuring that the changes are submitted when due. Responsibility for making changes should be assigned at the point of origin to distribute the maintenance workload. As much as possible the maintenance tasks should be performed using RecoveryPAC. If it is necessary to use paper forms to collect changed information, the BRC can print the RecoveryPAC data collection forms and send the forms to the people who must provide the information. Another approach is to use RecoveryPAC's Import Utility. In this case, the BRC or the System Administrator could forward the standard picklist file information and resource data files in an electronic format (e.g., Microsoft Excel, CSV or ASCII text format) to remote locations. When information has been entered into the data files and returned to the BRC, it can be imported into RecoveryPAC.
Each time the recovery plans are updated, the revised plans, or pages of the plan, should be distributed to the manual holders. A copy of the Revised Master Recovery Plan Manual should also be stored in the off-site storage location, and the out-of-date Master should be destroyed. The BRC should develop a Recovery Plan Manual Distribution list and keep it updated to ensure that the printed plans are distributed properly.
Scheduled Maintenance
The frequency with which the different elements of the business recovery plan will require maintenance will vary. Maintenance can be streamlined by using the RecoveryPAC business continuity planning software product. For example, some of the information contained in RecoveryPAC's dynamic database information files -- personnel information, critical documents, resources -- will usually require frequent change. Other information contained in the database files -- Workgroups, Glossary, Recovery Time Objectives, Facilities, Locations in Facilities -- may change less frequently. The recovery team activity schedules tasks and procedures may change even less frequently. (Many of the changes required for recovery team activity schedules will occur as a result of the testing programs.) Dynamic information should be scheduled for frequent maintenance, while the more static elements of the plans can be scheduled for less frequent maintenance.
The BRC must develop a program for scheduled maintenance, as well as a methodology for performing the maintenance. The maintenance schedule should include the dates when changed information is due, which information is due at each scheduled date, and who is responsible for ensuring that the changes are submitted when due. Responsibility for making changes should be assigned at the point of origin to distribute the maintenance workload. As much as possible the maintenance tasks should be performed using RecoveryPAC. If it is necessary to use paper forms to collect change information, the BRC can print the RecoveryPAC data collection forms and send to the people who must provide the information. Another approach is to use RecoveryPAC's Import Utility. In this case, the BRC or the System Administrator could forward the standard picklist file information and resource data files in an electronic format (e.g., Microsoft Excel, CSV or ASCII text format) to remote locations. When information has been entered into the data files and returned to the BRC, it can be imported into RecoveryPAC.
Unscheduled Maintenance
The maintenance program should provide for unpredictable (unscheduled) maintenance. In general, unscheduled maintenance is required only for changes that are so critical to a successful recovery that delaying the changes until a scheduled maintenance date would impact the organization's ability to recovery from a disruptive incident. Unscheduled maintenance often also implies major changes to the business recovery plans. In general, since unscheduled maintenance obviously cannot be planned in advance, the BRC must develop a maintenance program as soon as learning about the requirement for changes. If possible the BRC should be directly involved in the organization's change control process, standards, methods and procedures development, and training programs so that business continuity will be a consideration when changes are made. If the BRC cannot be directly involved, business continuity planning must be identified as a vital element in the organization's change control process.
Examples of situations and events that probably require revision to the Business Recovery Plan Manuals:
Major changes to the technology infrastructure;
Changes to the structure of the organization;
Mergers with and acquisitions of other organizations;
Personnel changes:
Personal information changes: name, demographics, telephone and other contact information
Business changes: title, department or workgroup; resignations, promotions, terminations;
Changes in the business processes that are included in the plans;
New business processes that must be added to the planning process.
New applications that are required to support a business process and thus needed to be restored in order to recover the business process.
Recovery Plan Manual Distribution
RecoveryPAC provides the ability to have as many printed Recovery Plan Manuals as required for your organization. Both Master and Unit Recovery Plan Manuals can be designed for the same database. Distribution of Master Business Recovery Plan Manuals (the total plan) and all Unit Business Recovery Plan Manuals (recovery team, support teams or support coordinator plans) should be done immediately after the plans are updated. Every person who is authorized to have either a Master Business Recovery Plan Manual or Unit Recovery Plan Manual should have two copies - one for their office and one to be kept at their home. Recovery team members should have copies of their team's section of the total plan (unit plans), support departments or coordinators should have copies of their sections of the plans; the BRC, management, and auditors should have copies of the Master Business Recovery Plan Manual. A printed copy of the Master Business Recovery Plan Manual should also be kept in an off-site storage facility. If the organization can provide notebook PCs to some or all of the recovery organization staff, current copies of Master and Unit Business Recovery Plan Manuals can also be maintained on the PC. This is especially useful at the time of a disaster that affects only parts of the facility. Information about which functions have been affected, and which recovery teams have to be activated, can be printed in real time. However, a Master Business Recovery Plan Manual should always be printed since reliance upon notebook PCs may cause problems.
Suggested Distribution:
The President or Chief Operating Officer (COO)
The Chief Information Officer (CIO), Vice President, Director - or the person who is responsible for all Information Systems functions.
The Business Recovery Coordinator and/or the System Administrator
The Auditor, or the Audit Department
The Vice President, Director, or Manager of Human Resources - or the person who is responsible for all Human Resources functions
The Recovery Team Leaders and Members, and Alternate Leaders and Members
When an employee is no longer involved in the recovery planning process, or the recovery process, their copy of the Business Recovery Plan Manual should be returned to the Human Resources Department during the exit interview. Subsequently the manual should be returned to the BRC.