2. A Large-Scale Complex Project
in a Healthcare Organization

 

This book describes how to do a project to create an automated patient medical record in an HMO, a very large-scale complex project. This chapter describes in detail how to do any large-scale complex project in a healthcare organization, thus providing the structure for the remainder of this book.

 

If the reader is less concerned with the details of how to do a large complex project such as automation of the patient medical record, and more concerned with the possible results of such a project, the reader may wish to skip to section 2.10, which presents a summary of this chapter.

 

The method of doing a project presented here is based on my observations over my thirty year career. I was a major player—from requirements analysis to system design to programming—in a number of successful large-scale projects, including one to develop a patient appointment and resource scheduling system, a software system currently used by 35,000 users, serving 2 ½ million healthcare organization members. I have also participated in the requirements analysis for an automated patient medical record system.

2.1     A Successful Project

A successful project within any organization has the following characteristics:

·        Organizational needs: The project fulfills a logical set of important needs of the organization.

·        Integration: The project together with other projects, existing systems and employee workflows, support the totality of needs of the organization in the most efficient way.

·        Adaptability: When the organization changes, the project or resulting systems and employee workflows continue to fulfill the needs of the organization or can be easily changed to do so.

2.1.1    Fulfillment of Organizational Needs

Another term for “organizational needs” is “organizational objectives”.

For an organization to be successful, upper management of the organization must identify and work toward organizational objectives. For example, for an HMO these organizational objectives might be the following: (1) provide excellent patient care, (2) generate sufficient money to run the HMO well and expand the HMO, (3) have sufficient information to manage the HMO well, and (4) fulfill all the requirements of government and regulatory organizations.

Projects are selected which, together, fulfill these organizational objectives with the greatest benefit to the organization. The project objectives [1] of each of the projects must thus be compatible with, and support, the organizational objectives.

For example, the broad objectives of a project to automate the patient medical record might be (1) to provide better patient care, (2) to save money for the healthcare organization and (3) to provide better information to manage the healthcare organization. These project objectives all support the organizational objectives. To be useful in the evaluation of the success of a project, these project objectives must be measurable, so the organization can determine whether these project objectives have been achieved once the project is complete.

To measure a project objective, the organization can set targets to be met that can be measured and that lead to the final objectives. Such targets are called goals [1]. A goal can be set after a particular phase of the project or at the end of the project to measure whether or not an objective is being achieved. For example, “goals” for a five year project might be to have an immediately available automated patient record for 40% of the HMO patient visits after three years and 95% of the HMO patient visits at the end of the project. The project’s final success can be determined by how closely the project meets the objectives for it as measured by the final goals for the project.

Strategies [1] that lead more quickly to the objectives and fulfillment of the goals also need to be established. One “strategy” might be to implement the automated patient medical record system first in the west coast region of the HMO, where 50% of the healthcare organization’s membership is located, before being implemented in other areas of the United States where the healthcare organization functions.

2.1.2    Integration and Adaptability

A project must enhance organizational objectives: the needs of the organization.  Other positive attributes of a project are “integration” and “adaptability”.

“Integrated” in the context of an organization means that all the employee workflows, automated systems, and other parts of the organization function well together, meeting the totality of the objectives of the organization. “Adaptable” means that an organization can be changed to adapt to new business needs, functioning at least as well after the changes as before the changes.

Projects must produce products (changes in employee work flows, automated systems, etc.) that are integrated with the organization. The products should be adaptable so they can easily be changed to produce changed products that meet the changing business needs of the organization in the future.

Frederick Brooks in his classic book, The Mythical Man-Month, [2] wrote about “integration” and “adaptability” with respect to automated systems.  He wrote about integration of automated systems with other automated systems, and about adaptability of automated systems.

“Integration” of an automated system with other automated systems in an organization means that there is real-time sharing of common data with other automated systems via networks or databases so all systems can be kept in synchronization. For our automated system, a patient medical record system, sharing of data between systems is especially important: inconsistent data in an automated patient medical record could be disastrous as it could result in incorrect medical decisions.

“Adaptability” for an automated system means that the system can be easily maintained and updated, including

1.      The system is scalable (i.e., it can increase in size with more users or more data usage).

2.      The components of the system (whether software or hardware) incorporate strategies that allow the components to later be more easily replaced by improved components (e.g., use of industry standards).

3.      The system is well documented and put under a change control process, where only agreed upon changes are allowed and where changes can be backed out if necessary.

4.      The reasons for the current design of the system are recorded and well understood so that future changes to the system can be made which are compatible with the design.

The book The Mythical Man-Month refers to an integrated system as a “programming system”, to an adaptable automated system as a “programming product”, and a system that is both as a “programming system product”.  The author of The Mythical Man-Month, Frederick Brooks, contends that, for an organization, a “programming system product” is many times more useful than a comparable software product that is not, but may take up to 9 times as much money to develop.

It should be noted that “adaptability” whether for the entire organization or for a single automated system, requires anticipating future business needs of the organization so that the organization or system can be made adaptable. This anticipation of the future requires management ”vision”. In anticipating such future business needs, management assumes a significant probability of being wrong, as anticipating the future is often extremely hard to do.

2.2     Basic Project Model

Figure 2.1 presents a project model applicable to a large complex project. This project plan promotes the three requirements for a successful project listed in the last section: (1) that the project fulfills organizational objectives, (2) that the project is compatible and integrated with other projects and with all aspects of the organization, and (3) that the project is adaptable, and can be easily changed to meet the changing requirements of the organization.

Large-scale complex projects such as ours need to be broken into smaller sub-projects, or phases. An overall design of the entire project is done at the very beginning that includes breaking the project into phases. At the beginning of each phase there is also a determination of whether the phase will significantly change the previously agreed-upon design of the project. If so, the overall design of the project may be re-done at the beginning of the phase.

For example, our project begins with a complete overall design of the automated patient medical record. The project is then broken up into phases to implement the automated patient medical record. An early phase in our project, for example, might be to rewire hospitals and medical offices and implement the networks for the future automated system.

This project model has a number of advantages over others, foremost of which is always knowing what direction you are going on the project. Doing a complete initial overall design allows the final products of the project to be predicted at the beginning, with each phase then developing products or immediate products that can be adapted into producing the next product leading to the final products of the project. Possibly redoing the overall design after an analysis later in the project during the start of a phase allows redirection of the project, with the project personnel again knowing what products to develop. Thus, the people doing and controlling the project always know where they are heading!

Other advantages of this approach are the following: (1) Going from one consistent design to another is much easier than trying to fix an inconsistent design. (2) The final products of a large-scale complex project often cannot be completely defined at the beginning; this project model allows redefinition of the overall products as more is learned about the project over the life of the project. (3) Breaking the project into phases allows the results of the project to be given to the organization at bit at a time, early on, with early payback to the organization--phases can be ordered such that those producing immediate results are done first. (4) The project can be evaluated early on and continuously to determine if it is meeting the project and organizational objectives and goals, with this evaluation possibly resulting in early revision of the project or termination of the project before too much money is spent. (5) This approach minimizes the cost of expensive re-working of the project, as re-working is done as soon as possible, before it becomes too costly. (6) Phases of the project may be done concurrently, effectively using available resources.

Products of a project could include automated systems, changes in the way employees function in doing their work, rewired buildings with installed workstations, etc., or different combinations, or “sets”, of these products that function together.  Each product set during the lifetime of the project should be adaptable, allowing change of the products to produce the next product set.

The project model in figure 2.1 is similar to project models referred to as the “incremental model” [3], the “iterative model” [3], the “evolutionary delivery method” [4], the “staged model” [5], and the “spiral model” [6].

Doing the overall design up front, and potentially re-doing it again before each phase, insures both that the project meets the needs of the organization and that the final product is integrated and adaptable.

Revisiting the overall design of the project before each phase allows upper management to verify that goals are being met and strategies are being followed to work toward the final objectives of the project. Costs can be reviewed after each phase and kept under control. Return on investment can be evaluated. It thus allows upper management to have control over the project.

It will insure adaptability by allowing early redesign of the project if necessary.  After each phase, the phase and the overall project can be reviewed, revised, improved or restructured. New or changed requirements can be incorporated.

It will insure that other automated systems using the same information are integrated by allowing redesign of the project as necessary so that all automated systems are properly integrated.

In order to make the model in figure 2.1 somewhat more flexible, we allow the project model to have the following additional characteristics:

·        Depending upon the complexity of the upcoming phase, re-visiting the overall design may involve only reviewing a part, rather than the whole of the overall design. Prior to simple phases and phases where much was known ahead of time, the overall design phase may not need to be reviewed at all.

·        Phases may be added or changed as a result of revisiting the overall project design.

·        Maintenance of a phase occurs immediately after a phase is complete and could result in such significant changes that a new phase is required, with possible revisiting of the overall design.

·        Phases may be added or deleted from the project plan for other reasons, for example, because of changes in organizational strategies or cost considerations.

·        It may be appropriate to re-visit the overall design to account for multiple upcoming phases, rather than just one.

·        Unlike what is shown in figure 2.1, phases can partially or completely overlap one another in time.

This project model is necessary because a large project takes a long time to complete and thus must be broken down into phases, but all phases of the project must, at the end of the project, fit together seamlessly into a whole.

Use of this project model is not an excuse to do bad design. In fact, a large complex project is only feasible if the total of the design at each step fits together impeccably. Any significant error in the design may be embedded into the project and require significant re-work to get rid of it. Fixing multiple errors in a large project later on in the project may be cost-prohibitive or impossible. The earlier in a project a problem is detected, the lower the cost to fix it. For example, it has been estimated that fixing a problem early on during the requirements phase of an automated system could be 1% as costly as making the fix after the automated system has been coded [7].

2.3     Participants in a Project

Many people, of varying roles, and aptitudes and interests, must be participants in a large project for it to be successful. These many diverse groups must work together as equals with the professionalism of each group respected by the other. The reason this cooperation is required is that a large-scale complex project is a single organism rather than the sum of its parts, with any one decision in one part of the project (e.g., a database design decision) potentially effecting any other part of the project (e.g., the user interface, with a possible consequential effect on patient care).

Accordingly, this book, unlike many others, does not look at a project from the single point of view of a project manager, but from the points of view of the many participants in a large-scale project.

The most successful projects involve meetings involving a diversity of different categories of people who meet together and learn from each other, creating a group dynamics to exchange information, thus creating a unified product that meets the many diverse needs of the organization (e.g., provide excellent medical care, have good system response time, supplies information needed by government and regulatory agencies).

There are four important categories of people who should be involved in a large-scale complex project: domain experts, content facilitators, upper management, and process facilitators. Domain experts are experts in the project subject area (e.g., patient care and medicine). Content facilitators gather information from the domain experts to design and produce the product. Upper management pays the bills and determines the organizational objectives for the project. Process facilitators work in group meetings to insure that meetings are productive, that group dynamics are established and that participants learn to run their own meetings.

Domain experts include workers in the organization (including upper management), who know how the organization functions, customers who know how the organization provides services, and experts from the outside world who know what happens outside the organization.

In our project developing an automated patient medical record system, the organization will be a particular type of healthcare organization, a Health Maintenance Organization (HMO). An HMO is a corporate entity that provides comprehensive health care for each member of the HMO for a fixed periodic payment paid in advance by the member or his employer; such a payment system is referred to as “capitation”. Our HMO has its own hospitals and medical offices spread throughout the United States. This is a fictional HMO, but is representative of a very large HMO that may exist in the United States.

Upper management in our example are management physicians who control the operations of the HMO. Being management physicians, they care very much about quality patient care as well as about the financial health of the HMO.

The workers in the organization include everyone in the HMO, but especially “caregivers”, those who provide medical care to patients: physicians, nurses, medical assistants, hospital unit assistants, etc. The customers are the members and patients of the HMO. The outside world includes suppliers, the government, regulatory agencies, and software and hardware suppliers, and other healthcare organizations.

Content facilitators for a project include business analysts, systems analysts, database analysts, domain analysts, members of the technical staff, and project managers. Business analysts gather information from upper management, workers, customers and outside experts to identify what an improved organization would look like and to identify business requirements that would accomplish these improvements. Systems analysts take the business requirements and a description of the improved organization to create requirements for automated systems. Part of systems analysis is determining new and changed databases--databases are places to save information; database analysts work with system analysts and domain experts or domain analysts to define databases for a project or for an organization, or change existing databases. A domain analyst is an expert in how a particular type of business functions and works with domain experts to facilitate solutions; for example, a domain analyst might be an analyst who is also an expert on provision of medical care and work with domain experts (e.g., the caregivers) in an HMO to reengineer medical care, changing work flows. Technical staff create and implement automated systems. Project managers schedule activities making up the project and guide personnel in their execution.

For a large-scale complex project, meetings between project members are likely to occur over an extended period of time. For any long-lasting meetings of project group members, there should be a process facilitator, or simply “facilitator”, until the participants in the group or sub-group learn how to work together (in which case the whole group assumes the responsibilities of the process facilitator). Along with the participants, the process facilitator develops processes to be followed during meetings to enable the group to work efficiently and make decisions (e.g., “It seems that our group has decided upon the following process: Any agreement this week will be written down. During the following week’s meeting we will either finalize the agreement or retract the agreement. This allows us to think over our agreements between meetings and discuss the agreements with our various groups.”). The process facilitator also teaches the group about interventions to be used if the group bogs down; for example, when multiple people are talking at once, the process facilitator might say, “Just a moment, one person at a time. Joe you were first, and then Carol”. An excellent book on facilitation is [8].

When there are automated systems, members of the technical staff are involved in the procurement, development and implementation of these automated systems. This staff may include the following: system architects, people who define how computers and other devices are linked logically and physically, how data is distributed between databases on the computers, and how software is distributed; capacity planners, people who get information from business planners on anticipated number of users and customers, and who determine and anticipate the transaction loads of systems and networks, thus anticipating overcapacity problems; software developers, who design software; and other DP personnel such as programmers, programmer/analysts and testers who program and test the software. Additionally, vendors may supply off-the-shelf software in place of organizationally developed systems.

The next section describes the different steps (or activities) making up a project and which participants are involved in each step.

2.4     Steps in a Project

The overall project analysis and each phase of the project can be broken down into the following steps:

·        Business Analysis step: Determine the changes to the organization from the project or from the phase from a business point of view.

·        Evaluation step: For the project as a whole during the overall project design, or for a phase, evaluate the projected value or actual success of the project or phase. Determine whether to continue, change course, or terminate the project or phase; included in this process for the overall analysis is whether to re-do previous phases and re-do plans for future phases.

·        Business Reengineering step: Determine how employees will function differently due to the project or phase. Included in this process is determination of user interfaces for automated systems.

·        System Analysis step: Determine technical requirements for new automated systems and changes to existing automated systems as a result of the project or phase.

·        Project Plan step: For the project, break the project into sub-projects, or phases; for a phase, break the phase into tasks in order to develop or implement the phase.

·        Development step: Develop or make changes to automated systems based upon information from the System Analysis step. (This is done only as part of a phase.)

·        Implementation step: Change the way employees function in the organization, or implement or change automated systems in the organization. (This is done only as part of a phase.)

 

Figure 2.2 shows the project plan in figure 2.1 expanded to include the above steps.

Although figure 2.2 shows the Evaluation step for a phase as occurring at the end of the phase, an Evaluation step for a phase may occur at any time, or not occur at all, within the phase. The Evaluation step within a phase determines whether to continue, change or terminate the phase, other phases, or the entire project.

2.4.1    Business Analysis Step

Figure 2.3 shows the Business Analysis step, in which business changes to the organization as a result of the project or phase are identified. This is the first step in the initial overall project analysis or in a phase.

Looking at the needs of the organization and the current environment and systems, upper management identifies potential projects. By analyzing the current environment and systems, management identifies problems in the organization. Based upon these problems and the needs of the organization, upper management identifies potential projects to further the needs and fix the problems.

Each phase within a project constitutes smaller project within the project.

For a project or phase, upper management identifies objectives, strategies, and goals for the project [1] or phase. Objectives are the future positions of the organization expected from the changes resulting from the project or phase. Strategies are ways to accomplish these objectives. Goals are targets to be met and measured at specific points in time that can be used to measure the progress or lack of progress of the project in achieving the project objectives.

For example, for the automated patient medical record project, some project objectives might be to save money for the HMO, to have the patient medical record instantaneously available, and to have the system calculate all charges to patients and insurance companies automatically. Strategies might include to create a common hardware infrastructure throughout the organization so the automated medical record system can be easily implemented; to implement the automated patient medical record first in the west coast region of the HMO to implement it the quickest, before rolling it out to other regions; and to immediately work with insurance companies for a data format to transmit healthcare charges to them. One goal related to the objective of saving the company money would be to have a positive return on investment by the third year (where “return on investment” is a comparison of the total costs of the project with the increased revenue accruing from the project.).

These (project) objectives, strategies and goals would later be used to evaluate the progress and success of the project in a later Evaluation step. They could also be used to compare this project against other projects, whose project objectives might also be important for the organization, so money can be allocated between the projects according to their value to the organization.

Project objectives, strategies and goals are derived from objectives, strategies and goals for the organization. This is done by identifying aspects of the project that further these organizational objectives, strategies or goals. These become the project objectives, strategies and goals. For example, if an organizational objective is patient satisfaction and some aspect of doing the project strongly promotes patient satisfaction, then this would become a project objective. 

During the Business Analysis step, upper management identifies constraints for the project or phase. The most important constraint is the amount of money allocated to the project or phase.

Also, during the Business Analysis step, a vision for the project should be developed. This vision takes ideas from many different sources to predict or define the future of the organization and outside world with respect to the project, in our case with respect to the automated patient medical record. Ideally, by meeting the vision, the product would have an extended life span in that the resulting product would fit into the organization as it changes and, where applicable, the resulting product would later be compatible with, and perhaps could interface with, systems in other organizations. In the case of our automated patient medical record system, this vision might include integration of the automated patient medical record with a possible universal patient health record that would be available to many healthcare organizations both inside and outside the HMO.

Vision, anticipating the future, also applies to determining how this project effects other projects, in the interim and once completed, and how other projects effect this project. This provides upper management with important information to select, evaluate and coordinate projects within the organization.

Vision entails significant risk that the prediction of the future is wrong. Thus risk management, planning alternative measures if the prediction turns out to be wrong, is important.

After the development of objectives, strategies and goals, the workers and domain experts would then identify the current environment and systems as a baseline for evaluating any changes. From detailed observations and analyses of the current environment and systems, essential business practices that must be preserved are identified. The workers could then identify improvements they would like to see in the current environment or automated systems. From upper management’s ideas for change, objectives, strategies, and goals, from various people’s vision of the future, from the worker’s list of improvements, and from an analysis done by the workers, upper management, domain experts and business analysts, a proposed future environment and automated systems is described.

Finally, the business analysts and domain analysts together with workers and upper management create a list of business requirements (changes to the organization together with things to keep the same) to transform the current environment to the future one. Business requirements include both requirements for changing the organization and for creating and changing automated systems.

As a method to clarify ideas, models of systems, workflows, etc. may be created, presenting conceptual views, or less abstract, more concrete views, of the systems and working environment. These conceptual views are vehicles for communication and for critique, especially for people who find that working with objectives and business requirements alone is too abstract for them.

When a project or phase involves an automated system, then storage of information on databases is required. Because developing databases is so terribly difficult, a start should be made at the start of the project or phase to items making up the database: business objects (e.g., a physician order), database objects (e.g., order status, order type of a physician order) and relationships between objects (e.g., each caregiver order has an order status such as “in process”, “completed”, “results returned”), and requirements for new information that is not on any existing database. Existing corporate databases—databases that are currently being used throughout the organization—that could be used by the project should be identified; use of corporate databases is one way of integrating the different parts of the organization.

Finally, expected products of the project, including potential new and changed automated systems, and changed employee workflows, are identified.

2.4.2    Evaluation Step

Figure 2.4 shows the Evaluation step, in which the project, so far, or as projected to be in the future, is evaluated.

The business analyst and systems analyst present an evaluation of the project or phase to upper management. The business analyst explains the business side of the project and related issues. The systems analyst presents the technical and automated system side of the project and related issues.

From this evaluation, upper management determines if they are, or will be, getting what they are paying for. This evaluation compares the changed, or projected future, work environment to the initial work environment and determines if the objectives, goals and vision of the organization are indeed being fulfilled by this change. Evaluation also involves identifying obstacles to the project that must be considered, both for determining if an obstacle makes the project too difficult to do, and also for inclusion into the project in the form of additional business requirements that handle an obstacle or in the form of a contingency plan, a plan to be followed later on if the obstacle does occur. The results of the Evaluation step may be termination of the project, changes to the project, or continuing the project as planned.

The Evaluation step during the initial overall analysis is there to ask the question, “Should we continue with the project?”. In our project model the Evaluation step for the overall analysis occurs directly after the Business Analysis step, as enough information has been gathered to make management decisions but an inordinate amount of time, money and effort has not yet been expended.

On the other hand, in our model, the Evaluation step in a phase can be scheduled to occur anywhere in the phase.  These Evaluation steps are put in less to ask the question “Should we continue with the project?” but more to ask  “Where are we? Are we meeting our goals? And if not, what changes to the project do we need to make?”

The first evaluation point in a project, as part of the initial overall analysis, is actually done before any changes are implemented; thus, the evaluation is entirely based upon projected changes to the organization due to the project rather than actual changes. Such an evaluation study in the initial overall design before any project changes are implemented falls under the category of a feasibility study.

A feasibility study [9] evaluates whether or not the proposed system is feasible and whether or not it should be developed given existing budgetary constraints and obstacles. In other words, based upon business requirements, it determines “Is there at least one solution that is currently feasible?” and if there is, “Is there at least one solution that the organization will be able to afford that benefits the organization?”

The results of an evaluation are reported to higher management. Obstacles to the success of the project are also reported to management; these are important to include here because project objectives generally only look at the benefits of projects. Suggested improvements in the project from workers are reported to upper management.

Higher level management has to make decisions on whether to continue the project, change the project, or terminate the project.  They might make changes in the objectives, strategies, goals, or vision for the project, or on constraints such as the amount of money allocated for the project.

Changes in the objectives, strategies, goals, vision or constraints for the project could result in changes in projections of the future working environment, and new or changed business requirements.  These changes are identified by business analysts working in coordination with workers, domain analysts, customers and the outside world. 

An obstacle is any actual or potential hindrance to the successful completion of a project. For identified obstacles, additional business requirements might be added to the project. For an unpredictable obstacle, a contingency plan may be created, a plan to follow if the obstacle does occur.

Technical obstacles for a large automated system often involve future performance and adaptability of the system (e.g., an automated system may be so slow that users would not want to use it). A performance and scalability plan should be started in the Evaluation step that anticipates growth of the automated systems, growth in number of customers and changes in technology. This plan should be used during the duration of the project and also afterwards to plan upgrades to hardware and system software systems and record measured bottlenecks in the system so they can be resolved before there are problems. In the Evaluation step, the performance and scalability plan would be used to determine whether it is or will be technologically feasible to do the project within the required budget for the project.

The performance and scalability plan is developed by three sets of participants: system architects, business planners and capacity planners. System architects determine possible system architectures, which describe the computer systems running the automated systems, the networks connecting the computers, the distribution of data on databases, and the flow of transactions occurring within the system. Business planners make an estimate of future growth and changes in the organization, including estimates of increases in HMO membership, in number of patients, and in nurses and physicians who work in the HMO. Capacity planners evaluate whether or not the proposed automated systems and system architectures will be able to handle the future growth and changes, taking into account projected future technology and monetary constraints on the project. Possible problems could include slow response time, exceeding the capacity of databases or network connections, or other slow-downs or failures of the automated systems.

2.4.3    Business Reengineering Step: Workflows and User Interfaces

Figure 2.5 shows the Business Reengineering step, in which changes to employee workflows are identified, including those for automated systems. For automated systems, user interfaces are also identified.

The Business Reengineering step applies both for the overall project design and for any phase in the project. For each category of employee in the organization, the future employee workflow is identified.

Workflows must take into account management-determined business policies for the company, which may have to be revised to account for changes in the organization resulting from the project. For example, one business policy of the HMO might be to have each HMO member pick a primary care physician to be her or his primary caregiver. For each business policy, the following should be documented: (1) employee workflows related to the business policy, (2) user interfaces (i.e., information input by users into automated systems related to the business policy), (3) databases (all the information that needs to be collected and stored to implement the business policy), (4) algorithms to execute the business policy, implemented in code possibly on many different computers, and (5) interfaces between the various computer systems. The business policy to associate each healthcare organization member with a primary care physician, for example, might require changes in all five areas to implement in the automated systems: workflows, user interfaces, databases, code and interfaces while other business policies might be implemented purely operationally effecting only employee workflows. Business policies often cross projects and automated systems—Documenting business policies in this ways insures integration in that it looks beyond single projects or automated systems and considers the organization as a whole.

New and changed automated systems for the project are identified earlier in the Business Analysis step. Here in the Business Reengineering step, the user interface for each of these new or changed automated systems, and each category of worker using the system, is determined.

In order to develop user interfaces that best correspond to the business (in this case, patient care), it is best to define “business objects” that a typical user is familiar with in his work. For example, for a person providing patient care, these objects may include “patients”, “physician schedules”, and “the patient medical record”. Section 12.2.2 identifies many of the business objects for our project and describes in detail the benefits of identifying business objects.

Since, all data in the user interfaces comes from and will be stored on databases, user interfaces and databases must be kept in synchronization. Some of these databases may already exist in the organization while others might have to be created. Interfaces between automated systems (network communication) might also be identified, which pass data between computer systems.

The primary purpose of the Business Reengineering step is to redefine how the organization functions. Thus, this, the Business Reengineering step, is very important in a project, as the primary purpose of any project is to change the organization to fulfill organizational objectives (e.g., improve patient care, save the HMO money).

2.4.4    System Analysis Step     

When there are automated systems in the project, the requirements for the automated systems, including software, databases, hardware, interfaces and infrastructure, are determined in the System Analysis step (see figure 2.6). These requirements for automated systems are determined from the results of previous steps: Business requirements for the systems come from the Business Analysis step, while user interfaces for the systems and associated employee workflows and data requirements (e.g., for databases) come from the Business Reengineering step. System analysts facilitate and record the results of the System Analysis step, working with the technical staff, business analysts, domain experts, and vendors.

System requirements for automated systems can be put into the following categories: software, databases, hardware, interfaces between automated systems, and system software including operating systems, database management systems and network middleware. An interface is data transferred between automated systems, with the data either being transferred via network connections, or multi-system reading and writing information to and from databases or other data stores.

Interfaces would be designed with domain experts understanding the data and technical experts understanding the automated systems. Software and hardware vendors developing an automated system could be involved. Interfaces could be with systems outside of the initial scope of the project, including systems outside the organization. I propose that interfaces be described in an interface plan.

In the Business Reengineering step, organizational business policies associated with the project are defined, with specifications for code and associated tables, interfaces between systems, databases and changes to user interfaces. In this, the System Analysis step, the code and tables, interfaces between systems, and databases would be defined in more detail. In this book, an agent is a way of categorizing and separating out the set of items eventually implementing the organizational business policies: the operational rules, code and tables, interfaces between systems, databases, and user interfaces (possibly spread across a number of different software systems) implementing a business policy, so that the business policy could be implemented and changed by the people responsible for the business policy instead of relying on technical staff to do so.

Databases need to support all automated systems and business policies in the organization, including existing automated systems, and new or changed automated systems that are part of the project. For the System Analysis step, database information can most usefully be expressed in terms of entity-relationship (ER) diagrams (see section 13.7) and data dictionaries (see section 13.13), as well as by text, describing the required data and relationships between the data. Data dictionaries exist as documentation tools to describe a database, and also can be used for operational control of the database within a database management system when the database is actually used.

If there need to be changes in the infrastructure, such as building rewiring or remodeling, these changes are planned during the System Analysis step. The planned infrastructure would be implemented during the Implementation step. The choice of hardware for the automated systems could be influenced by space considerations within facilities and other infrastructure factors.

Hardware must also be compatible with the way workers actually do their job. Thus, the choice of hardware in this step may be influenced by workflows identified in the previous Business Reengineering step. For example, respiratory therapists who already carry around lots of equipment from room to room may not be able to carry any additional equipment, such as portable computers, as the total weight of these items may then be too much.

Finally, hardware must be compatible with the performance and scalability requirements identified in the Performance and Scalability Plan. The Performance and Scalability Plan is a document that should be kept up-to-date during and after the project, projecting future hardware capacity needed in the system. This capacity would be expressed in terms of number of users, transaction volumes and network traffic. When a project has automated systems that will grow in capacity over time, it is extremely important to identify projected long-term capacity and scalability requirements up front during the initial overall design so the systems can be developed and planned accordingly; otherwise, there is the high likelihood that a large-scale automated system could quickly become obsolete due to lack of capacity.

2.4.5    Project Plan: Breaking the Project into Phases or Tasks      

The Project Plan step occurs in the initial overall design and occurs in each phase. When this step occurs during the initial overall design, the project is broken up into phases. When it occurs as part of a phase, the phase is broken up into tasks. See figures 2.7.

Doing all aspects of a very large-scale complex project (such as automation of the patient medical record) all at once is risky, and undoubtedly not feasible. During the overall design, the project should be broken up into phases, smaller sub-projects. Both the overall project and each phase would have its own project plan and project manager.

In order for such an approach to work, phases (or tasks) that provide the infrastructure needed by later phases should be done first. For example, if creation of an automated patient medical record depends upon information coming from other automated systems, then the interfaces with these systems should be developed early in the project.

Ideally, high payback phases would be done first, so that the benefits from the project can be realized as early as possible. For example, if an automated patient medical record system provides the input for a billing system that would result in an increase in revenues to the HMO, then the phase to integrate the billing system could be done early on so as to create an earlier return on investment for the project. 

What is produced from the Project Plan steps is an overall project plan for the project scheduling the phases within the project, and a project plan for each phase, each scheduling tasks to accomplish the phase. The overall project plan would be used by the overall project manager, while the phase project plans would be used by each of the phase project leaders. Resources--people, equipment, rooms, etc.—would be determined and allocated for the initial overall project design, and for each task in each phase, estimating the allocations for each phase.

Once these project plans have been created, intermediate and end-of-project goals can be scheduled within the project plans. Contingency plans can be scheduled for risks (potential obstacles to the project) that can be anticipated ahead of time. Evaluation steps (also called gates) can be scheduled anywhere in the project plan of the project or phase to evaluate the status of the project at that particular time during the project.

In order to save money, fulfill organizational objectives more quickly, or other reasons, project strategies would be developed. Project strategies could change phases, tasks and the project plans. For example, one project strategy might be, early in the project, to minimize any changes to existing interfacing clinical systems, while in the long run to standardize clinical systems throughout the HMO.

2.4.6    Development Step: Automated Systems        

Figure 2.8 shows the Development step. This step only occurs within a phase and not as part of the initial design.

In the Development step, an automated system is created, or a vendor product is selected and customized for the organization. The Development step uses the prior Business Reengineering step in the phase to identify user interfaces and changed employee work flows. And it uses the phase System Analysis step to identify hardware and software systems, including databases and interfaces between systems. 

The Development step only occurs when the phase involves development of an automated system, purchase of an automated system from a vendor, or a change to an automated system. It only occurs as a part of a phase of the project, not in the overall design, as the overall design is for planning of the project rather than creation of products of the project.

User interface requirements for the automated systems together with associated changed employee workflows for the organization are defined in the Business Reengineering step. System requirements for the automated systems (hardware, application code, databases, interfaces with other automated systems, and other parts of the application) are defined in the System Analysis step. Specifications for implementation of business policies carried out within the automated systems are produced within the combination of the Business Reengineering step and the System Analysis step; these business policies are implemented through user interfaces, code and data or databases. Business requirements, how the organization is changed business-wise, is determined in a number of different steps, but primarily in the Business Requirements step.

If the automated system is developed within the company, then, in the Development step, the system, business, and user interface requirements and business policies are used by systems analysts and programmer analysts to create program specifications for the automated system. These specifications are reviewed by business, management and technical groups and then used to create the code.

If the automated system is to be procured from a vendor, rather than developed within the organization, this process will be somewhat different. The system, business, and user interface requirements and business policies would be used by the systems and business analysts to create a “request for proposal” (RFP) to send to vendors; this would make a request for proposals for systems meeting these requirements. The vendor would write a proposal, including identification of customizations to the system, and send it back to the organization for review.

Based on an evaluation of vendor proposals and the vendor system, analysts, management and other employees may select one of the vendor systems. The evaluation process would probably include management and analysts to evaluate whether the system met business requirements and business policies or could be customized to do so.  Potential users would evaluate user interfaces. And capacity planners would evaluate whether or not the vendor system could handle anticipated future network and transaction volumes and could be integrated with other automated systems.

After selection of a vendor system, a contract to use and change the vendor system would be written. The vendor, system analysts and programmer analysts would develop specifications for customization of the vendor product.

The hardware and system software would be selected (including operating systems, databases, and network middleware) and installed as part of an organizationally developed system. In the case of use of a vendor, the hardware and system software could have been automatically chosen because it came with the vendor system or, alternatively, the chosen hardware and system software could have been a requirement for the automated system and have been included in the RFP.

Organizationally developed application software would be programmed by programmers from the program specifications; vendor application software would be customized by the vendor according to agreements with the organization. In either case, the software would then be installed on a development system, in other words, an automated system used by the programmers, analysts and testers to develop, modify or test the system rather than one used by the organizational users of the system. The development process usually also requires utility programs to support the programming process and to manage software; examples are compilers and change management programs.

The development system may be on a single computer or a network of computers, and may, and usually does, include changes in code to other organizational automated systems used for development that are interfaced by network connections or databases. These interfaces allow all these development systems to share the same information (rather than creating multiple sets of the same information, which may cause data integrity problems by being out of synchronization with each other).

Additional development systems may be produced to test new ideas or to simulate interfaces with other organizational systems. Such systems may also be on a single computer or a network of computers.

Test plans would be created from the program specifications, vendor customization specifications, and performance and scalability specifications to test the whole or parts of the development system as it is programmed, installed and integrated. The development system would be tested, resulting in a tested system being created. The tested system would be implemented during the Implementation step and re-tested during implementation. Testing is also called validation.

If errors were found during testing, the program or vendor customization or business policies specifications might have to be updated or the development systems might have to be reprogrammed. In the latter case, the development system would then be re-tested again. This process would continue until the system passed all tests.

During development, the system would also be compared against system, business, user interface, and business policy requirements to determine if these requirements were being fulfilled. This process is called verification.

Once the development system passes all tests, it would be available for implementation in the Implementation step, where it would be transferred over to a nearly identical hardware and software systems used by the actual organizational users, called production systems. Bringing the code to production may involve moving over one or more of the development systems to production, with any remaining production systems remaining the same.

A conversion plan may be developed in the Development step to be executed in the Implementation step. The conversion plan tells how to convert data in existing and replaced production systems to produce the data for the new and changed production system databases.

During the development of an in-house system, capacity planners provide advice on how to create a system that meets performance and scalability requirements. During the procurement of a vendor system, capacity planners provide input to the RFP to identify performance and scalability requirements for the vendor system and to later evaluate vendor systems on how well they meet these requirements. The development system, whether an in-house system or vendor produced system, would be tested to insure that it could handle the requirement number of system users when it is implemented in production.

2.4.7    Implementation Step

Figure 2.9 shows the Implementation step, in which the changes in the organization are implemented, including the installation of any automated systems, moving them from development to production.

Implementation involves installing new or changed automated system(s) developed in the Development step, and changing the way workers currently do their jobs according to workflows identified in the Business Reengineering step. The Implementation step could occur without installation of new or changed automated systems, and thus might simply implement new workflows for employees in the organization.

The actual computer systems on which an automated system will eventually be installed must be put in as an early part of implementation, perhaps at the time of the Development step. This may involve extensive amounts of money on hardware, rewiring of buildings, or even remodeling of buildings. Any hardware selected must be able to handle future capacity and performance requirements.

Based upon the workflow requirements, the organizational working patterns would be changed. These changes could be minor ones, simply automating current manual functions, or could involve “drastic changes” to significantly transform and improve the organization. Drastic changes in the way an organization functions to significantly improve the organization has been termed “business process reengineering” [10].

Before using a new automated system, workers must be trained in its use and trained in any changes to way their work would be done. The automated system would be physically installed and the workers would use the new automated system in the changed work environment.

As part of installation of an automated system, data in other existing systems and in replaced systems may have to be installed in the automated system’s databases, with possible conversion of data from one format to another. This is controlled by the conversion plan created during the Development step.

If there is a new automated system replacing an old automated system, then there are several approaches to phasing out the old system and phasing in the new one. Of course, one approach is to just turn off the old one and start the new one. Other approaches would involve running both systems in parallel. See section 15.5.1 for additional conversion approaches.

2.5     A Project in More Detail

So we have identified the participants in a project. And we have identified the parts of a project. How do these parts fit together in a large-scale complex project?

Figure 2.2, introduced earlier, shows how the component parts of the project fit together, both in the overall analysis and in each phase of a project.

2.5.1    Initial Overall Analysis

The first part of a project is to plan the complete project. As shown by figure 2.2, the overall design includes an overall Business Analysis step to define the project, an Evaluation step to inform upper management of the project’s feasibility, a Business Reengineering step to identify changed organizational workflows, a System Analysis step to design automated systems, and a Project Plan step to break the project into phases.

The purposes of this overall analysis are the following:

·        to define the project and its objectives

·        to insure that all the component parts of the project, and all its phases, will fit together

·        to allow upper management to evaluate whether they will getting what they expect from the project, and allow them to make a decision of whether to terminate, change or go ahead as is with the project.

The overall analysis phase begins with the Business Analysis step, in which ideas for the project are introduced by upper management or sold to upper management. Upper management must provide strong support and agree to expend the required money. Upper management provides the initial objectives for the project (expected future results from the project), strategies for accomplishing the project, and intermediate goals for the project (e.g., in the third year of the project there will be actual cost savings to the organization as a result of the project). Objectives and strategies for the project are based upon objectives for the organization and strategies for reaching these organizational objectives, project objectives must support organizational objectives, and strategies for doing the project must be consistent with organizational strategies.

Workers and others describe the current working environment where the changes will occur and the current applicable automated systems. Through a process that includes incorporation of the information from management and workers’ ideas for improvements in the organization, a future projected organization as related to the project is described. Business requirements for producing the future environment from the current one are identified, where business requirements are future required characteristics of the organization that will result from the project.

Next, a very important Evaluation step is completed. The projected future environment is compared against the current environment, and benefits, risks, return on investment and other values are determined for the project recorded in a feasibility study. Upper management uses the feasibility study to determine whether the project is possible to do and, if so, to compare the project against other projects, evaluating how each project would satisfy organizational objectives relative to its costs, determining which projects should be done and which ones should have priority over other projects.

An additional determinant is how each project would change the organization. Upper management compares the changes against their vision of what they want the organization to look like in the future. If there is a significant difference between the two, upper management may change the project to be more in line with their vision, adding additional business requirements for the project and possibly resulting in a modification of the feasibility study.

If there is a go ahead from upper management to continue with the project, a Business Reengineering step is done to determine changes to organizational workflows and to determine user interfaces for any new or changed automated systems based upon business requirements. An overall System Analysis step is then done to determine automated system requirements from the business requirements and from the identified user interfaces. The Business Reengineering and System Analysis steps closely depend on each other and thus often overlap, with the user interfaces influencing the automated system requirements and the system requirements influencing the user interfaces.

Finally, and this is a very difficult step, the project is broken up into phases. (These phases later produce the actual products produced by the project.) The earlier phases should provide the infrastructure, if applicable, for further phases (e.g., the security subsystem and databases for an automated system should be developed early on as these are part of the infrastructure needed to produce the automated system; wiring within buildings and networks should also be created early on to support the automated systems). Ideally, although not necessarily, early phases should provide significant benefits to the organization (e.g., provide revenue for the organization), so that the benefits of the project materialize early on, even if it is later determined that the project should be terminated. Within the overall project plan and at the end of the project, measurable goals should be scheduled to evaluate the progress of the project and to compare the actual versus the expected results of the project.

The initial overall analysis, even if done without all the necessary information, is very important as it establishes a direction and consistency for the project. Projects where all its parts (i.e., all phases) are done in a consistent manner are much easier to latter change, than ones done inconsistently.

Reference [11] has shown how important this upfront initial overall analysis is for enterprise software projects: “Those users who spent more upfront time and effort analyzing their systems, business and process needs fared much better than those who didn’t. … Users who follow these procedures achieved positive outcomes 56 percent of the time vs. only 8 percent for those who failed to conduct such an analysis.”

2.5.2    A Phase of the Project

The actual creation of the products of the project occurs within the various phases of the project.  A project can usually be broken up into phases in many ways.,

For example, our project to improve patient care in an HMO through automation of the patient medical record might include the following phases: (1) a phase to standardize clinical systems within the HMO that interface with the automated medical record system, (2) a phase to automate the patient medical record within the emergency and outpatient clinics, (3) a phase to automated the patient medical record in the inpatient (i.e., hospital) setting, integrating it with the outpatient record, (4) a phase to create a single medical record for a patient across HMO medical centers and regions.

For each such phase of the project, there is, like in the initial overall analysis, a set of steps (see figure 2.2 again): (1) a Business Analysis step to compare the current working environment to the future one after the phase and to develop resultant business requirements describing the changes to the organization in a business sense; (2) a Business Reengineering step where changed workflows are developed for the phase and, if necessary, user interfaces for new or change automated systems are designed; (3) a System Analysis step to identify changes to automated systems or new automated systems for the phase, (4) a Project Plan step where a detailed project plan for the phase is developed in terms of tasks, including detailed estimated costs for each task and potential problems that must be managed during the phase; (5) a Development step where the automated systems are developed or procured from vendors and customized for the organization, and (6) an Implementation step where the new working methods are implemented, workers are retrained, and the automated systems are implemented. As part of the Business Analysis step of a phase is a determination of whether this phase unexpectedly affects the overall design of the system, requiring that the overall design of the system be re-done.

A phase may not involve automated systems (e.g., a phase to re-wire medical centers) and thus the System Analysis step and Development step (which design and produce automated systems respectively) may thus be skipped.

Periodically, the whole project must be reviewed, as part of the Evaluation step, to determine if goals are being met. Such points in the project are called gates. Although figure 2.2 shows the Evaluation step (a gate) occurring at the end of a phase, Evaluation steps (gates) could occur anywhere in the project. Gates should be pre-planned as part of the Project Plan step to periodically evaluate the project’s progress, allowing management to make decisions on changing, or even terminating the project.

2.5.3    Continuing or Terminating the Project

At an evaluation point it may be determined that the project should be terminated. Otherwise, the project is eventually finished.

People often think of projects as being done once all the products of the project are initially installed, e.g., once the automated patient medical record is installed. But I think that this is the wrong point of view. A project—such as the automated patient medical record system—should be viewed as always being a project until it is decommissioned, as it will likely be continually changing to meet the changing needs of the organization until that time. Changing a system after it is delivered to improve it or to correct errors, referred to as maintenance, is always required.

At the end of a phase, the phase goes into maintenance mode. Once all the products of the project are installed, the entire project goes into maintenance mode.

Maintenance, after an automated system is completed according to the schedule, could consume 70-80% of the costs during the lifetime of the automated system [12,13].

2.6     Flow of Information in the Project

Figure 2.10 identifies the flow of some of the information during the project.

Project objectives, what the organization hopes to gain from the project, that match organizational objectives, are identified; for example, an organizational objective might be to “improve medical care within the HMO” with one corresponding project objective being “to create a complete and always available patient medical record”. From these project objectives business requirements are derived, where these business requirements are characteristics of the organization after the project that fulfill the project objectives; for example, a corresponding business requirement to the example project objective might be “to store the patient medical record electronically, making it available to caregivers both inside the HMO and within other healthcare organizations”.

Goals for the projective objectives, ways to measure fulfillment of a project objective or progress toward the project objective, are identified. For example, a goal might be the following: “at the end of the project, an automated patient medical record will be immediately available for a patient during at least 98% of the patient visits of HMO members to HMO outpatient clinics as measured by observation of a representative sampling of outpatient visits in each HMO facility”. Goals can be set up for the end of the project, but also for immediate points in the project.

Additional business requirements are created by identifying essential business practices within the current environment and systems that should be preserved, and, at the same time, by identifying non-essential business practices that provide little value or that are no longer needed, and thus should be discarded. For example, one essential business practice might be for the HMO to produce required reports to the U.S. Government agency HCFA that identify Medicare patients who have been admitted to the hospital; generation of these reports could be preserved, but with the automated system generating these reports rather than the reports being produced manually. With the inclusion of an automated ordering and results reporting system within the automated patient medical record system, a now unneeded business practice might be the use of couriers to return back clinical test results from the clinical laboratory to the emergency department.

Business requirements are also derived from identified obstacles to achieving the changed organization and systems; for example, one obstacle to achieving the changed organization and systems might be “the government mandates certain security requirements for patient information”. One business requirement derived from this obstacle might be “to build these government security requirements in the automated patient medical record system”.

Organizational business policies are developed by the organization. These business policies may be implemented across many environments and systems. For example, there might be a company business policy to “to have each HMO member pick a primary care physician who would guide patient care for the patient with this primary care physician recorded for the patient in automated systems”. Such a policy might be supported by HMO automated systems, including the patient medical record system and many other HMO systems. Such business policies might thus result in additional business requirements for the project, but also changes outside the project, for example, a change to the HMO appointment system to identify members’ primary care physicians.

When enough business requirements have been developed, a projection of the future environment and systems is done. The future environment and systems could be reviewed by upper management to determine if it meets their ideas; as a result of this projection, additional project objectives could be developed, and additional business requirements derived.

Like for a project objective, a business requirement might also have an associated goal to measure progress towards its achievement; for example, for the business requirement to “to reengineer the care process based upon input from physicians” might be the goal of “surveying physicians to validate that 95% of them feel they are providing equal or better quality of care than previously”.

From the business requirements, workflows for operations within the organization are established, user interfaces for automated systems are determined, and ways the user interfaces mesh with the workflows are identified; a way to associate user interfaces with workflows are called use cases. Since user interfaces collect and display data, user interfaces can help determine the data that needs to be stored on databases. Additionally, common data within the organization collected within one automated system must be passed to other automated systems via database and network interfaces between these systems.

System requirements for the automated systems come from the user interface requirements, the business requirements, the logic of the various automated systems and from discretionary choices. New and changed automated systems produced by the project are identified. New databases and existing databases to be modified are defined, interfaces with other automated systems are finalized. Requirements for performance and scalability for the automated systems are identified.

From the system requirements, an automated system within the project is developed and implemented. Hardware, networks and operating systems are chosen.Functional and non-functional specifications  describe the automated systems from an external point of view; these specifications are broken up into functions, where a function is the smallest discrete, complete set of code of an automated system that can be initiated by the user and run to completion to produce a single purpose set of results. Program specifications are written to describe each of the programs, thus describing the automated systems from an internal point of view; often programs are written to be self-documenting as the program specifications could quickly become out of synch with programs, especially with programmers being notoriously lazy documenters. Test plans for each of the programs are created to define how to test each program and how to test the system for performance requirements and scalability.

The system is installed in a development system—the software and hardware implementing the system—to develop the system and test it using test plans. Later the automated system is installed in data centers and medical centers. User documentation is created to describe how the system, and user interface, functions. Manuals on how to train the system, training manuals, might also be created.

The various phases result in sets of requirements for the project. These requirements are listed in figure 2.11 and their relationship is shown in figure 2.12 [14].

 

Figure 2.11: Project Requirements

Requirement Type

Description

Information Source

Characteristics

Business Requirements

Changes to the functioning of the organization due to the project.

Upper management, employees, customers, domain experts

Must be consistent and complete; otherwise, missing business requirements will be determined by implication later on by people who should not determine them and/or resulting products will not fit together well.

Business Reengineering Requirements

Employee workflows in the domain of the project, including for any automated systems.

Upper management, employees

Must be consistent with the business requirements.

User Interface Requirements

User interfaces for automated systems, compatible with the workflows.

Employees (in particular automated system users), upper management

Must be useable and consistent with workflows.

Data Base Requirements

Data, data relationships and information needs.

Database analysts (from management, employees and domain experts)

Must both handle all user interfaces and all business requirements.

System Requirements

Hardware and software architecture and overall automated system design, including interfaces to other automated systems.

System architects, domain experts, upper management

Must be reliable, fast, adaptable and integrated, and be appropriate for the user interfaces and system interfaces.

System Non-functional Requirements

Automated system implementation of overall business policies, standards, performance guarantees, and regulations.

Business and system analysts (from upper management, domain experts, system architects, performance analysts); used by software developers.

Together with the functional requirements, must be consistent with all requirements and produce maintainable code. Must complement the functional requirements and all other changes in the organization.

System Functional Requirements

Requirements for each automated system function.

Created by business and system analysts in conjunction with automated system users; used by software developers.

Together with the non-functional requirements, must be consistent with all requirements and produce maintainable code. Must complement the other changes in the organization.

Program Requirements and the Programs Themselves

Requirements for each program.

Software developers.

Together with the functional and non-functional requirements, must be consistent with all requirements and produce maintainable code. Must complement the other changes in the organization.

 

 

Some organizations, especially U.S. government organizations, believe in tracing requirements from one set of documents to another, including tracing requirements to software code. Requirements tracing is discussed in detail in reference [15]. Requirements traceability is quite costly and requires significant discipline by those doing the documentation.  The author only believes in traceability of organizational business policies and associated documentation to workflows, database information, code, interfaces between systems, and user interfaces implementing them; this traceability facilitates participation of non-computer personnel in the later change of business policies.

Once an automated system is completed, it must be maintained as the system is used. Maintenance involves changing the system for new enhancements and corrections of errors in the system. Keeping and updating all the project documentation after the automated system is implemented is probably not feasible, but it is extremely useful for the maintenance phase to have the following documentation, which is kept up-to-date:

·        Functional specifications, describing the external view of the automated system on a function by function basis.

·        System design specifications (aka system requirements specifications) describing the overall internal design of the automated system.

·        Program specifications or internal program documentation, describing the internal design of each computer program.  Each program must implement the functions of the system and follow the overall system design of the system.

When a business requirements change necessitating a new enhancement, or when an error is corrected, the associated functional specifications are updated. If these changes result in a change to the overall design of the system then the system design specifications are updated. The programs are then changed to implement the changes in external design of the system and in the overall design of the system; a change in a program is either documented only in the program or both in the program and the corresponding program specification. In this way, the programmers and other maintainers of the system, produce a product that is consistent with the business requirements implemented by the system and with the overall design of the system.

2.7     Controlling Changes

Once major decisions have been made during the project, they should be recorded and put under change control [3]. The change control process is a formal process where changes are made to controlled documents with the agreement of members of a designated change control board.

During each step of the project, participants in the step are recording agreements. During this time, the final results of the step remain in flux and may change. But at the end of a project step, these final agreements need to be recorded comprehensively and understandably, with a sign off by all the participants so these results can be used in later steps. For example, if technical requirements for a new automated system are being determined during the Systems Analysis step based upon a set of business and user interface requirements in previous steps, then any later change in these business or user interface requirements would also effect the technical requirements. It is difficult for participants in a step to do their jobs if the results they depend upon change after the participants have already made decisions based upon these results.  Thus documentation of the final decisions in a project step should be recorded at the end of a step and placed under change control.

The change control board generally approves or disapproves a change in the project at a high, non-detailed, level. Once the change is approved, it is analyzed by project groups and defined in more detail, with determination of which controlled documents to update.

Within a project group a walkthrough may be held to review the change for inclusion in the controlled documents. A walkthrough is thus a meeting where controlled documents are reviewed in detail.

Controlled documents within a project may include one or more of the following documents:

·        organizational objectives, strategies, goals, and priorities of objectives

·        project objectives, strategies, goals, constraints, and priorities of objectives

·        business requirements and goals

·        user workflow requirements

·        user interface style guide

·        user interface requirements

·        system requirements

·        organizational business policies

·        data dictionary

·        interface plan

·        programming standards

·        programming specifications

·        program code

·        test plans

·        performance and scalability requirements

·        user documentation, including descriptions of user interfaces

·        training manuals.

Suggested changes to systems, both during the project and after project completion, can come from many sources, including from users and upper management.  Changes are often prioritized by the change control board or by user groups so the most important ones are done first.

A help desk may be set up for the automated systems in a project that consists of designated telephone numbers to which users can call to seek advice and to report errors in the systems.

Errors in systems differ from changes. Errors are identified by differences between the way the system works and the way the program specification, vendor customization specification, project business policies specification, or performance and scalability specification describes the system to work. Changes, on the other hand, are changes to any of these specifications and the corresponding changes to programs. Like for changes, errors are also usually prioritized and are fixed accordingly--but errors are usually given much higher priority than changes. The controlled documents thus have the important purpose of distinguishing errors from changes.

Another form of control of the project is only implementing changes and error corrections to automated systems on a pre-scheduled basis and keeping previous versions of the software and other controlled documents. This is referred to as a release process, with controlled documents being under release control. If an unexpected error occurs during a release, the release can be backed out, with return to the previous version.

2.8     Determining the Success of the Project

When a project reaches completion, it is deemed successful if all of the following are true [16]:

The project

·        achieves basically all the objectives originally set for it

·        is accepted and used by the audience for whom the project was intended

·        comes in on-schedule

·        comes in on-budget.

The latter two are related to two normal constraints on projects that are usually determined by upper management at the beginning of the project: time to do the project and the money to do the project. To determine whether the project was on-schedule and on-budget these two constraints are compared against the actual time to do the project and the actual cost of the project.

Whether or not project objectives have been achieved can be determined by goals set up at the start of the project to measure these project objectives at the end of the project. For example, for a project objective to increase member satisfaction with patient care, the goal might be that there be a 15 percent increase in the number of HMO patients who are highly satisfied with patient care after the project is complete as compare with those at the start of the project. Determining whether the project was responsible for the change may be harder to determine.

Surveying and observing users at the end of the project can determine whether or not the project is accepted and used by its intended audience.

2.9     What Makes for a Successful Project

Over a period of thirty-five years, the author has observed small to very large projects. Some of these have been highly successful while others have either failed or suffered from lack of quality. Although, the success of any project cannot be guaranteed, the failure or lack of quality of a project is almost certain if any of the following is true:

1.      The project is not based upon fulfilling organizational objectives: This is the whole basis of doing a project.

2.      Not enough time is spent on requirements: Requirements are what the project is supposed to do based upon the real needs of the organization. Many problems occur if these requirements are not identified from management, workers, and the total of automated system users at the beginning of the project, or if these requirements do not form the basis for the products of the project. 

3.      Project participant do not have the correct mix of talents: Project participants must collectively be strong representatives of each of their areas of expertise, coordinating and integrating their ideas, so the products of the project are coherent and work well.

4.      The wrong people are chosen to do the project: People with a variety of strong professional skills and with a variety of different ideas are needed to end up with the best results.

5.      People function outside their roles: Information technology personnel should not determine the non-technical requirements; people outside of information technology should not determine the technical requirements (e.g., software and database design).

6.      Not enough time is spent on infrastructure: Infrastructure, those things that are behind-the-scenes and not seen, are as important as those things that are seen.

2.9.1    Project Success Criteria #1: The Project Fulfills Important Organizational Objectives

The purpose of a project is to fulfill organizational objectives. For example, one of these objectives for automation of the patient medical record is to improve patient care. Achieving organizational objectives is the entire purpose of a project--The project cannot be considered successful unless it does so.

2.9.2    Project Success Criteria #2: Enough Time is Spent to Determine Requirements Up Front

Requirements are what a project is supposed to do based upon the real needs of the organization. It is important to spend sufficient time to determine the requirements from management, workers and potential users of systems before the products of the system (changes in workflows, automated systems and user interfaces) are designed or changed, as the later in the project a defect is found, the more costly it is to fix.

For example, reference [7] estimates the cost to fix a defect in an automated system after it is implemented as 100 times as much as identifying the error during the requirements phase. Reference [17] estimates this cost as 100 to 200 times as much. Even a small number of such defects could doom an already costly, large-scale project to failure. Certainly, other changes caused by a project, such as a change in an employee workflow or installation of wiring, would be much more costly to change once their were actually implemented.

One kind of defect is missing or inconsistent requirements. Often what happens in such a case is that new requirements are determined by analysts, designers or even programmers rather than by management or workers. Besides increasing the chance of creating defects, if requirements are not determined by management or workers, there is the strong possibility of getting products that no one—neither management, worker, or automated system user—wants.

2.9.3    Project Success Criteria #3: People With the Correct Mix of Talents are Picked

As stated earlier, a project should be considered to be a single organism, not just the sum of its parts. A decision in one area of the project may have an impact on other areas of the project; technical decisions may impact non-technical areas of the project and vice versa.

For example, for an automated patient medical record system to be successful, all of the following must be true: The users must want to use the system. Management and the organization must be able to get the information they need from the system. The automated system must be fast enough to handle the volume of transactions. The cost of developing the system must not be too much for the organization to bear.  The system must eventually make money for the organization or otherwise provide large benefits to the organization.  The system must be scalable for the future. And the system must comply with legal requirements.

All the above are interrelated. In order to accomplish all these, a variety of different kinds of professionals must be involved in creation of the automated patient medical record system, and they must communicate with each other and coordinate their decisions. Dictation of project decisions by any one group (e.g., physicians or administrators) could spell disaster for the project.

2.9.4    Project Success Criteria #4: The Right People are Picked for the Project

Who are the right people for a project? The right people are (1) people who are open-minded and willing to learn from others, and are willing to listen. The right people are (2) a variety of different people in key professional areas who are each very qualified in their profession (computer scientists, nurses, physicians, law professionals, database analysts, business analysts, patients). The right people are (3) people who can collectively look at the project from many different points of view. The right people are (4) a set of key people who are dedicated to continuing on and finishing the project. The right people are (5) people who are willing to keep discussions open when they perceive a bad decision was made, but who are willing to accept that the other person may be correct—they must be both analytical and open-minded. And the right people are (6) people dedicated to the success of the organization.

People should be chosen for a project based upon there being a diversity of relevant professional backgrounds within the project, and based upon a diversity of different viewpoints and ideas. People should not be chosen simply based upon amiability, friendships, charm or glibness. The people should be the most qualified professionals in their fields.

With this diversity of people and opinions, it is important to have a process facilitator—at least at the beginning—to teach the group how to work together and how to resolve differences. 

All the participants in the project must be committed to the success of the organization and making the project work. Employees of the company, rather than contractors, are more likely to fit into this category. In a healthcare organization this dedication to the organization is likely to be based on the participants ideals of helping people, providing best quality medical care—contractors are unlikely to have this dedication to healthcare.

2.9.5    Project Success Criteria #5: Project Participants Stay Within Roles

During a project to develop a software system, various sets of requirements are developed. Figure 2.11 identifies which groups are responsible for developing each of these sets of requirements. It is important that each group stay within its role.

Only employees in the project area and upper management in the organization, are in a position to determine business requirements, business reengineering requirements, and user interface requirements for a project and an organization—such requirements should not be determined by software and database designers. However, as bad as it is for software and database designers to determine business, business reengineering, and user requirements, it is equally bad for management and users to determine how a system should be designed or coded, or databases created, as doing so requires specialized capabilities. It is my experience that not following these roles results in non-structured, non-maintainable, or unusable software systems.

On the other hand, software and database designers, as well as system architects, should be facilitators in the identification of business, business reengineering, and user interface requirements, as they need to thoroughly understand these requirements and to insure that inconsistencies in requirements and missing requirements are resolved. They need this deep understanding of the requirements to be able to produce a logical and integrated software and database design that fulfills these requirements.

Viewing employees, upper management, and the organization as “customers” of the “technical community”—of the software and database designers and system architects, who develop software and hardware systems—the IEEE [18] developed a diagram to explain the interaction between these two groups in the development of requirements for an automated system. See figure 2.13 that was adapted from the IEEE diagram.

Management and employees identify business, reengineering and user interface requirements. The technical group identifies the technical representation of these requirements, determines if the requirements can be accomplished or not, and determines if there are incomplete, missing or inconsistent requirements. This is an iterative process, where well-formed sets of requirements result from these analysis, negotiation and discussions on each side, the customer and technical. The “environment”—management and industry constraints, and reality—is another factor in this process. Developing requirements is a highly interactive and iterative process.

What should be developed are well-formed requirements, “a statement of system functionality (a capability) that can be validated, that must be met or possessed by a system to solve a customer problem or to achieve a customer objective, and that is qualified by measurable conditions and bounded by constraints [18].” In general, well-formed requirements have the following desirable characteristics: simplicity, clarity, completeness, and consistency.

2.9.6    Project Success Critera #6: Develop the Infrastructure Before Implementation

It is a human tendency to want quick results. A project should be considered as building for the future, not building for quick results. To build for the future—to build something that lasts—it is important to build the infrastructure first. This takes time, and it takes management realization that something very useful is happening when the infrastructure is being built, despite their appearing to be no useful results out of the project.

Infrastructure is everything in the project that is hidden. There are two types of infrastructure associated with a project: (1) infrastructure for the project itself and (2) infrastructure for the products of the project.

Examples of infrastructure within a project are (1) the hiring or selection of the right people to do the project, (2) the meetings and hard work to design and implement the products of the project, (3) the necessary documentation and the control of this documentation, (4) the computer systems on which systems are developed (4) the training of personnel, (5) change management of software and databases, etc. If money is not spent on project infrastructure, then the quality of resulting products is likely to be much less.

Examples of the infrastructure for a product of the project (e.g., the automated patient medical record system) are (1) the people and facilities to support the product such as help desks and technical personnel, (2) the changes to buildings, including changes to wiring, (3) the training of employees in new workflows or in the use of new automated systems; etc. If money is not spent on the infrastructure for an automated system, then, when the automated system goes down, there are likely to be long periods where no work gets done or where data has a chance of being lost or corrupted. Putting in the infrastructure after the product is implemented almost always costs significantly more money.

Sometimes the infrastructure created for the project can become part of the infrastructure for the products of the project (e.g., controlled documentation and development systems are still needed once the product is implemented; change management of software and databases is still needed).

2.9.7    Problems of Developing Projects Within Healthcare Organizations

Silicon Valley companies are known as places for doing innovative projects, especially software projects. One of the reasons is the culture of such organizations. Silicon Valley companies pick people not based upon outward appearances or how well they speak English, but on how well they can do the job. Further, because Silicon Valley companies exist for a shorter period of time than others, they don’t develop the hierarchy found in other companies, and their employees share more or less an equal status. Employees are often given stock options, in other words, ownership in the company; as a result, they develop a strong commitment to the success of the company.

Virtually every other organization works differently than these Silicon Valley companies, especially healthcare organizations. In companies outside the Silicon Valley, my boss tells me what to do, and my boss’ boss told him or her what to do and so on up the ladder.

In a medical situation, a physician tells a nurse or nurse practitioner what to do, and she or he (most of the time anyway) does exactly that. This hierarchial approach works well in medical situations in that it enables decisions to be made quickly with the most experienced person making the decision and that person taking the responsibility if something should go wrong. This approach does not work well for innovative projects where a large variety of different skills, viewpoints and ideas are required.

This is not to say that an innovative project cannot be done in a healthcare organization, but to do such a project successfully, the normal cultural rules of the organization must be suspended for participants in the project. The next section describes an approach to doing a project that worked.

2.9.8    An Example of One Successful Approach to Doing a Project

This section describes one successful approach to doing a large-scale project in a healthcare organization. This project resulted in the design, creation and implementation of a very widely used automated patient care system, a patient appointment and provider scheduling system, and of databases for the automated system and many other automated systems within the healthcare organization.

This successful approach was as follows:

·        There was a main meeting centered around the future users of the automated system. The users were the key to whether the automated system would be used or not, and thus whether the system would be successful or not. The results of the main meeting were a detailed description of the automated system from the users point of view and from every other necessary point of view. The information technology group along with process facilitators led these groups and presented and recorded ideas. Decisions were agreed upon and recorded, and presented in meeting minutes.

·        There were also subgroups meeting. Management (management physicians) met with the information technology group to identify business policies to be included in the automated system. The data base group also met with information technology; this meeting used information from the main meeting and subgroups to create databases both for the corporation and for the automated system.

The database group met during the entire time of the main meeting; the management physician group met for only a part of the time of the user meeting. Information technology took the ideas of each of these subgroups to the main meeting to be presented and then incorporated into the automated system.

·        Users and management were knowledgeable about the previous automated patient care system whose ideas were used, but the previous system was completely replaced by the new system, with an entirely different, and improved, design.

·        There were key members of the main group who attended almost all of the main meetings, resulting in there being a continuity of ideas during the lifetime of the main group. A group dynamic was established where every member became an active participant in the group. New members were quickly brought up to speed. The group dynamic was partially a part of the group working together, but also a result of the process facilitator.

·        Through the process facilitator, processes for making decisions and reaching agreement within the main group were established based upon decisions of the group.

·        Hierarchies (such as between physicians and nurses) were dispensed with in the main meetings. Main meeting members were sometimes reminded of the equality arrangement, as sometimes when physicians attended, the other members tended to not participate and these members let the physicians ideas go unchallenged.

·        The business requirements for the system were completed and signed off in the main meeting before the final user interface requirements for the system were worked on. Some proposed user interfaces were discussed during the business requirements discussions, mainly as a tool to clarify business requirements.

·        Management and users determined business and user requirements for the project, never information technology or other facilitators. Facilitators sometimes determined potentially missing or contradictory business and user requirements, but they never determined these requirements themselves.

·        The requirements for the project were based upon logic, simplicity, and business requirements, not determined by politics or power. Simpler ways to accomplish the same thing were given priority. Difficult to understand or use capabilities were allowed, but were implemented in a way not to make the system harder to use for users who do not directly use the capability. All members of the main group, both users and facilitators made suggestions on possible easier approaches.

·        Models or prototypes were built to conceptualize what was decided upon in meetings.

·        During all meetings there were with clear, agreed-upon decisions all being written down in meeting minutes, with the minutes distributed quickly so the participants could review them with their local groups. These agreements were written on a pad or a board during the meeting and were thus seen by all participants so it was clear what was agreed upon.

·        Content facilitators, mainly the information technology group, prepared for every meeting, incorporating agreed-upon ideas into the prototype or model. The users consulted with people they worked with between meetings to determine if the correct agreements and decisions had been made; agreements and decisions were sometimes reconsidered or retracted at the following meeting.

·        All participants understood their area of expertise in full detail, and understood other areas of expertise to the extent that they could explain the reasons for resulting design decisions based upon this expertise to their local groups.

·        Different groups used different terminology sometimes (necessarily) with different terms for the same thing. Users used different technology from physicians from the database group from the data processing group.  Ambiguities and inconsistencies in terminology were resolved. Ideas from one group were conveyed to the other, most often using the terminology of the latter group.

·        In total, combining everything that happened inside and outside the meetings, every decision made in a meeting was analyzed in detail, determining the total implications of the decision. Analysis was done of every potential and agreed-upon decision to determine its total effects on the project, with the view that a decision in one area (e.g., requiring the caregiver to enter information into an automated system in a non-standard way) could negatively affect any other area (e.g., could harm patient care)—Such analysis was done for every decision or potential decision during the entirety of the project. During the meetings, such implications of decisions were used to identify new issues or missing parts of the system or project, and were future discussion items in the main or sub-group meetings; these implications sometimes resulted in reconsideration of the decision at a future meeting.

·        The success of the project was largely because of the dedication of the participants. Participants were professional, knowledgeable, dedicated, open-minded and enthusiastic. Many participants attended almost all meetings; this provided a continuity across different steps of the project. Some participants had the ability to convey ideas between groups.

·        The systems analysis work, producing the requirements for the automated system, was done by information technology. This step overlapped in time with the latter part of the main meetings, using the business requirements, business policy decisions and user interface requirements to create these system requirements, although much of this systems analysis work was done outside the main meeting. Programming was started after most the systems analysis work and database design was complete, starting well after the meetings ended.

·        Keys to the success of the project were (1) that the users got the product (e.g., an automated system) that they wanted to use, (2) that organizational business policies were properly incorporated into the system, (3) that business requirements were mostly completed and agreed upon before the user interface requirements were determined, (4) that database design was done concurrently with the main meetings, and incorporated business policies, and business and user interface requirements into these databases, (5) that the systems analysis work, defining the automated system, overlapped with, but mostly occurred after determination of the business requirements and user interface requirements, and (6) programming occurred after systems analysis.

·        Other groups that met after the above meetings included the following: (1) Information technology continued to meet with the database group to finalize the database design. (2) Technical groups met to discuss system architecture. (3) Designers of the automated system met with managers representing other interfacing automated systems to design and implement these interfaces. (4) Various groups met with medical center personnel to discuss infrastructure, including medical center redesign, training, security requirements, and workstation, wiring and network requirements.

2.10   Summary

The basic parts of the project approach presented in this chapter are as follows:

·        Identify a potential project: Identify a potential project and establish a project mission statement.

·        Describe the organization prior to the project: Describe the current environment and systems with respect to the project.

·        Identify how the organization will benefit from the project: Determine project objectives based upon organizational objectives.

·        What do you want the project NOT to change?: Identify current business practices to be preserved.

·        How would the organization function after the project?: Describe the changed (i.e., the future) environment and systems.

·        What must be done to make these changes to the organization?: Identify business requirements for the project to achieve the future environment and systems and satisfy the project objectives.

·        What could stop the project from succeeding?: Identify obstacles and how to overcome them (which may add additional business requirements).

·        Is the project worth doing?: Evaluate the project in comparison to other projects in the organization; determine which ones to do (if any) and which ones to terminate (if any).

·        How do you measure whether the project is successful?: Identify goals used to measure whether project objectives and business requirements are being or have been achieved.

·        How will employees be working differently after the project?: Determine workflows for the future environment and determine user interfaces or requirements for user interfaces for the new and changed automated systems.

·        Describe any new or changed automated systems: Determine the design of the new and changed automated systems.

·        Develop a project plan for the project and any sub-projects (phases) within the project: Develop a project plan, including estimates of costs. For a large complex project, break the project into phases (subprojects) such that implementation of the phases will accomplish the total project, with consideration that there may have to be, at any point, possible re-design of the project or of previous phases or possible addition of phases to accomplish the overall project

·        Make the changes to the organization and implement the products of the project, such as automated systems: Develop and implement the changed environment and the new and changed automated systems.

·        Monitor and maintain the changes.

Projects are most effectively done by people of different professional skills working together, sharing and coordinating their differing ideas to produce the products of the project. The group dynamics of an effective project development group is vastly different from the group dynamics of a typical healthcare team: the former has to be democratic to be effective; while healthcare teams are usually controlled from the top. For a project to be done well, the project must encourage this free flow of ideas. “We find comfort among those who agree with us; we find growth among those who don’t” [19]—encourage this growth so the best possible agreements can be made.

Not only must the products of the products be produced, but the infrastructure to make these projects function properly must also be set up. Infrastructure is all the hidden, behind-the-scenes, things that allow the products to function properly. Infrastructure includes support staffs, change management of software and databases, training of users, building changes to handle workflows and to handle new wiring, etc.

It is important for management to realize several things about infrastructure:

1.      Setting up infrastructure costs lots of money without any apparent results.

2.      Without properly setting up the infrastructure for a product of the project, the product will likely eventually function poorly, or even fail.


References

[1]        David I. Cleland, William R. King, editors, Project Management Handbook, Van Nostrand Reinhold, 1988, article by William R. King, “The Role of Projects in the Implementation of Business Strategy”.

[2]        Frederick P. Brooks, Jr., The Mythical Man-Month: Essays on Software Engineering, Addison-Wesley Publishing Company, 1978.     

[3]        Neal Whitten, Managing Software Development Projects, John Wiley & Sons, Inc., 1995, pp. 20-22, pp. 268-273.

[4]        Tom Gilb, Principles of Software Engineering Management, Addison-Wesley Publishing Company, 1988, pp. 83-114.

[5]        Vaclav T. Rajlich, Keith H. Bennett, A Staged Model for the Software Life Cycle, Computer, July 2000, Vol. 33, No. 7, pp 66-71.

[6]        Barry Boehm, “A Spiral Model of Software Development and Enhancement”, IEEE Computer Vol. 21, Number 15, May 1988, pp. 61-72.

[7]        Dr. Phillip Laplante, editor, Keys to Successful Software Development, IEEE, 1999, article by Shari Lawrence Pfleeger, “Measuring Software Reliability”, reprinted from IEEE Spectrum, August 1992, pp. 55-60.

[8]        Sam Kaner, Lenny Lind, Catherine Toldi, Sarah Fisk, Duane Berger, Facilitator’s Guide to Participatory Decision-Making, New Society Publishers, 1996.

[9]        Ian Sommerville, Software Engineering, Addison-Wesley Publishers, Ltd., 1996, p. 67.

[10]      James Champy and Michael Hammer, Reengineering the Corporation: A Manifesto for Business Revolution, New York City: Harper Collins, 1993.

[11]      Ellis Booker, “Enterprise Software Projects Rarely Satisfy”, InternetWeek, March 28, 2000, URL: http://www.internetwk.com/story/INW20000328S0004.

[12]      Roger J. Martin, Wilma M. Osborne, Guidance On Software Maintenance, National Bureau of Standards Publication 500-106.

[13]      R. S. Pressman, Software Engineering: A Practitioner’s Approach, 2d ed.. McGraw-Hill, New York, NY, 1992.

[14]      Karl E. Wiegers, Software Requirements, Microsoft Press, 1999, p. 8.          

[15]      Matthias Jarke, Guest Editor, “Requirements Tracing”, and associated articles by others, Communications of the ACM, December 1998, Vol. 41, Number 12.  

[16]      David I. Cleland, William R. King, editors, Project Management Handbook, Van Nostrand Reinhold, 1988, article by Jeffrey K. Pinto and Dennis P. Slevin, “Critical Success Factors in Effective Project Implementation”, pp. 481-482.

[17]      Dean Leffingwell, Don Widrig, Managing Software Requirements: A Unified Approach, Addison-Wesley, 2000.

[18]      Software Engineering Standards Committee of the IEEE Computer Society, IEEE Guide for Developing Systems Requirements Specifications, IEEE Std 1233, 1998 Edition.

[19]      Marty Nemko, Oakland career coach with talk show on radio station KALW in San Francisco and column, “Under the Radar” in the Sunday San Francisco Examiner.

 

NEXT
TABLE OF CONTENTS

Copyright © 2000-2001 Michael R. McGuire

Duplication not permitted without express written permission

 

Comments? mailto:Michael.McGuire@abac.com