Difference between revisions of "Operations and Support"

Latest revision as of 02:03, 23 December 2017

Welcome to the initial version of the EITBOK wiki. Like all wikis, it is a work in progress and may contain errors. We welcome feedback, edits, and real-world examples. Click here for instructions about how to send us feedback.

1 Introduction

In Enterprise information technology (EIT), operations and support coordinate and carry out the activities and processes required to deliver and manage services provided by technology assets to business users and customers, at agreed levels. A commonly used framework for operations activities is ITIL®, which is proprietary. ISO/IEC and IEEE standard 20000 is similar in many aspects, and may be used by any organization to develop and manage EIT operations. Many EIT service management terms have been redefined in ISO/IEC 20000, so it is important to be consistent with the source your organization prefers.

This knowledge area is about maintaining an operation normal state in EIT environments, meaning that systems and processes execute according to expectations, with no surprises to users. Operation normal states are disrupted in two ways:

• Through proper channels during the transition of an asset. Operation normal is suspended during a change, and then when the change is implemented, there is a new operation normal state. Changes can come when an asset is "retired, or when construction, acquisition, or sustainment activities are transitioned into operations.
• Through incidents (ITIL) that interrupt the normal operational services. Incidents can be resolved with or without creating a problem (ITIL) to be handled through sustainment or strategy and governance (see Strategy and Governance). Incident resolution relies heavily on disaster preparedness (see Disaster Preparedness).

There are three main elements involved in this function: assets (what), users (who), and services (how). Each of these has an inventory, catalog, or user list. Each of these also needs an overall function to manage and monitor the performance, regardless of asset, service, or user details. This chapter addresses how that overall function works.

Figure 1. Assets, Services, and Users

Overall management of assets, services, and users is composed of two main parts: operations (assets and services) and support (users).

The operations function manages the assets used to deliver services used by business processes via technology including:

• Technical administration—managing the assets and their connections within the organization
• Operations monitoring—monitoring the assets as they function and provide services to the organization
• Access management—providing access to users upon request or via linking to another system that manages access
• Application certification—testing applications to ensure they operate successfully in the environment
• Incident management—providing leadership and resolution when operational systems halt
• Problem management—providing assistance when operational systems are not running optimally due to some defect or workaround due to resolving an incident

The support function manages the users, and may include technology that provides the following functions:

• User management—providing the ability to manage user access to services and applications
• Help desk—providing a service where users can talk to a person who can help them with their request, troubleshoot issues, and create trouble tickets, which after analysis, are determined to be defect reports or change requests
• Self-service support—providing a service where users perform selected support functions on their own

Although these processes and functions are associated with operations, most processes and functions have activities that take place across multiple stages of the service lifecycle (see Service Operation).

This chapter covers topics in the Service Operation (SO) column, as shown in the ITIL Core Topics table below.

2 Goals and Principles

2.1 Goals

The main goals of operations and support are to maximize the use of resources and minimize the negative impact of changes to the environment. All organizations have an EIT environment, a sort of ecosystem where things do tasks for people. Here, this translates to assets provide services for users. The table below illustrates the translation.

To know whether these goals are being achieved, attributes of each should be measured quantitatively and objectively.

2.2 Guiding Principles

DO ...

• Have standards for technology and data that minimize confusion and rework, and reduce the opportunity for technology- or task-specific positions or services.
• Share resources where possible and practical.
• Make sure that backups and restores work by testing periodically.
• Make sure that all changes are properly approved, and scheduled to be implemented to cause the least disruption to business processes.
• Have procedures documented for when incidents occur that will guide the efforts to resolve the incident and restore services as quickly as possible.

DO NOT ...

• Panic when incidents occur.
• Allow emergency changes that have not been tested to enter your environment.
• Allow changes to systems (patches) that do not use the normal application interface without very high level clearance. If necessary, make a backup of the system before running any patch.

3 Context Diagram

Figure 2. Context Diagram for Operations and Support

4 Asset Management

What cannot be measured cannot be managed. The first requirement for any system is knowledge of the system itself. It is critical that a centralized or federated authority exists that is responsible for asset management within the organization, ideally with a separate budget to handle on-going operational asset needs outside of projects. Manage assets to enable business processes to occur predictably.

Many organizations have suffered from project-only EIT organizations, letting hardware and software age out of support due to lack of funding. Operations and support organizations have a stake in ensuring that assets are up to date, and that maintenance assistance from system suppliers (either vendors or in-house developers) is available for emergencies. See the Sustainment chapter for more on this important topic.

Some organizations may outsource support; even then, the organization has heavily invested in the assets and depends on the services, so it follows that knowing the asset and service inventory is critical, regardless of what organization provides the support.

4.1 Asset and Facility Administration

There are four main categories of assets. Support teams to manage those assets vary depending on the asset type and the organization's needs.

4.1.1 Types of Assets

There are four main types of assets to manage: facilities, hardware, software, and data. The first three types exist in order to create and use the third. Thus, the ultimate goal of EIT operations is to provide the correct data to the business processes and people who need the data.

• Facilities contain and protect the hardware, and include buildings, security systems, environment conditioning, and uninterruptible power supplies. These types of assets can break down over time, and need to be repaired or replaced. Facilities can also become insufficient for ongoing needs and require expansion, either in place, or with the addition of other facilities, which then requires network connections to join the facilities. Or, company needs might change and facility contents might need to be moved to new facilities.
• Hardware is the physical machinery installed in a facility, which enables software to run and data to be stored and moved. This includes (but is not limited to) desktop/laptop workstations, servers (individual and farms), disks, disk arrays, solid state storage, modems, routers, network cards and appliances, backup media, and wiring. These types of assets can break down over time, and need to be repaired or replaced.
• Software is the collection of digital assets that are loaded onto the hardware to provide a service. These include (but are not limited to) operating systems, utilities (backup, restore, compression, search, etc.), daemons, drivers, compilers and kernels, email and calendar applications, office productivity applications, hosted third-party applications, software development environments, database management systems, database access systems, network connection software, user interfaces, browsers, browser interfaces, client/server applications, reporting and analysis applications, and asset-management applications. These types of assets do not break down as they are used. However, they are patched or upgraded based on new hardware capabilities or programming changes. Such changes, although intended to improve the software, can cause the software to change its behavior for the worse, or to completely fail. Software does not always install and execute perfectly with no input. Most software requires some input or change to operate as expected. There are three main types of adjustments done to software: configuration, customization, and maintenance.
  
  • Configuration is the process of selecting or entering values for parameters that the software needs to be installed and execute in the environment. Configuration values can be entered manually, or stored in files for the application to use. No programming in the software is changed in configuration.
  • Customization is the process of changing a vendor's source code to change the functionality of software. Customization can be performed by the vendor that owns the software, or by the user if they have a software development toolkit (SDK) or the actual source code. One risk with customization is that implemented changes can be overwritten and lost when performing maintenance using vendor-supplied files.
  • Maintenance is the process of applying changes to applications via files provided by the suppliers.
    

    Feature	Configuration	Customization	Maintenance
    Frequency	Installation Based on analysis or requirement changes	Based on development team release schedules	Based on maintenance schedules Vendor release schedule
    Changes to supplier-delivered files	No change to source code or executable files	Changes to source code and executable files Note: Customization can make installing patches or upgrades from the supplier difficult or impossible to do without losing the customization.	Changes to source code and executable files by overwriting them with new files created and provided by the supplier
    Limits	Changes execution in pre-determined ways	No change limits	Changes execution in pre-determined ways
    Implementation	Application data entry screens (wizards) Local file with defined structure (config files)	Software development toolkits (SDK) Source code modification and re-compilation	Follow supplier-supplied instructions Follow supplier-supplied update application
    Content	Parameters (data) Selected options	Code (instructions)	Executable files Scripts
    Skills required	Ability to answer questions or follow directions	Familiarity with software used to create the application	Ability to answer questions or follow directions

• Data is an asset that is typically overlooked. Data is the content that all the applications use. Data is loaded and stored on hardware, and moved over networks. Data is read, written, copied, manipulated, displayed, added, deleted, backed up, restored, archived, and sent via many protocols between systems. By itself, this type of asset does not break down over time, although it can become corrupted. Instead, the detail becomes less relevant as time passes. This asset is managed through policies determining what kind of data needs to be available in specified timeframes, and when data should be archived (put on slower/older/less expensive access media) and purged (erased from media permanently). Never patch data through the database management system alone! All data changes should occur through an application interface to reduce the chance of corruption.

4.1.2 Types of Support Teams

There are four main types of support team organizations: in-house local, embedded, in-house remote, and offshore. Large organizations may have sub-teams of each type, depending on needs and costs. Larger organizations may have combinations of these types of support teams, depending on assets and business needs.

• In-house local—In-house local support has the support personnel co-located with the assets, usually in a datacenter. This support staff is part of an EIT organization. Business users may or may not be in the same location, depending on the organization. Support teams are local to the assets, so incidents involving hardware can be resolved sooner by eliminating travel time for remote staff to get to the site.
• Embedded support—Embedded support means EIT support staff is co-located with the business unit (BU) directly, not with the asset location. Having the support team embedded in business unit makes support convenient from the business side; however, from the EIT side, the main drawback is the possibility of resistance to EIT standards. If there are enterprise standards, co-location with the business unit can bring extra pressure to ignore standards in favor of business needs. Shadow EIT is a common term for BU-owned technical support and development. These organizational structures grow for a variety of reasons. The primary drawbacks are:
  
  • Asset lifecycle maintenance—Routine maintenance activities may not be performed consistently, if at all, resulting in assets going out of support over time, even though license and maintenance costs are still being incurred.
  • Future support—If in the future, support needs to come from EIT rather than the BU, non-standard assets may be more difficult or expensive to support, or must be converted to use standard assets, which may be costly.
  • Skills—Technical support staff that is part of a business unit may not receive the same technical training and cross training that an EIT support organization would provide. It may be that someone in the BU learned on the job instead of having formal technical training. When an incident occurs (something unexpected happens or an action performed without technical review has unintended consequences), they may not be prepared to resolve it without formal EIT assistance (see Support Incidents and Recovery).
• In-house remote—EIT organizations provide in-house remote support outside of the corporate headquarters or business offices. Users contact the support center to report issues and receive support services. Some travel between the remote location, the datacenter, and the user locations occurs when necessary. This support model can be useful if there is little need for personal contact with the assets (such as when using a vendor-owned cloud implementation), so the support employees can work from home, or at a separate support center location.
• Offshore—Offshore support occurs in another country. Many companies specialize in remote technical support. Additionally, offshore support can occur during work hours in that location, which could be overnight in the datacenter.

4.1.3 Understanding Asset Support Requirements

Each organization is different, and the support needs for all assets in the environment are different as well. Therefore, an analysis of needs must be carried out before looking for solutions (technical or otherwise).

1. Assess priority—The importance of an asset varies for each organization depending on the industry and the maturity of the organization. There may be several installations of the same asset in an environment (such as in a server farm); each instance is treated as a separate asset. This includes production environments, and any failover, testing, sandbox, or development environments that EIT is responsible for. The following questions can help guide asset support requirements analysis:
   
   • Which assets are mission critical?
   • Which assets are critical for teams to be effective?
   • Which assets can be non-functional in an emergency?
   • During an incident, how long can each asset be non-functional without significantly affecting normal business processes?
2. Determine asset type—Each organization has different types of assets that need support. Below are the two ends of the asset management spectrum.
   
   • If the organization uses cloud computing, they may only need monitoring of applications and data, as the supplier handles all the rest of the monitoring needs according to contractual agreements.
   • In a traditional EIT organization with an internal datacenter, asset types include machines, networks, and building features.
3. Review documentation needs—Each organization requires a different level of detail in documentation to ensure adequate support of the assets. In mature organizations, part of the turnover exercise is a review of all relevant documentation with the support team. Below are the two ends of the documentation spectrum.
   
   • LOW—If the organization has an agreement for support with an off-site vendor, the level of detail may be low—for example, a chart of the escalation process with support phone numbers for the vendor and minimal instructions for issue troubleshooting.
   • HIGH—If the organization has an internal support staff with great expertise, the systems are not supplier supported (that is, they are home grown or highly customized), the level of detail required may be high enough to include information that allows staff to make significant changes to the internal workings of the asset—for example, software development toolkits (SDK) or parts lists.

4.2 Asset Support Processes

Support requirements include determining the processes needed to provide adequate support for the enterprise's technology assets and users. There are three main types of support processes for assets: monitoring, auditing, and scheduling. The process of defining support process requirements needs to be finished before looking for mechanisms (technical or otherwise) to support or automate part of the processes, and in fact, provides information that can be used as requirements in tool selection later.

1. Monitoring—The kinds of processes needed to view the execution of the services operating on or using the assets vary by organization. Below are the two ends of the monitoring spectrum.
   
   • Simple monitoring and alerting—The systems are simple and only trigger alarms when certain thresholds are met or exceeded.
   • Complex monitoring and real-time management—The systems are complex and interdependent, so support team members may need to actually see messages or monitoring screens updating in order to keep the system operating within acceptable parameters.
2. Auditing—The level and frequency of auditing or review corresponds to the maturity of the EIT organization and the industry requirements. This may be particularly important for certain organizations dealing with healthcare or finances, or parts of organizations dealing with human resources information.
   
   • Simple annual review—The review of processes occurs annually as a checkpoint for owner or shareholder reports. There are no regulatory or legal requirements for reviews.
   • Complex periodic auditing—This includes:
     
     • Schedules for each type of audit to occur
     • Published certifications of compliance with laws, regulations, and internal process rules
     • Acknowledgements from employees that all required processes are executed correctly
     • Evidence of acknowledgement from customers that they understand their rights
3. Scheduling—Support schedules vary by each organization's needs. There are two main support schedules: monitoring and maintenance. An optional schedule can exist for system audits (see item 2, above).
   
   • Monitoring—Different types of monitoring can be required depending on the time of day, day of the week, or the day of the month or year. All monitoring should have instructions for what situations to look for, and what to do in those situations, such as who to inform if a situation arises, or a list of expected anomalies and how to handle them. Also, monitoring can be automated to a point of only sending alerts when expected thresholds are crossed.
   • Maintenance windows—This includes standard and emergency maintenance preferences (such as normal nightly, and during lunchtime in an emergency to reduce work impact when possible). Planning each maintenance session should have a minimum and maximum task limit to prevent unnecessary shutdowns for tasks that can be completed at a later time (not enough work to justify). Each instance should have a list of each process occurring and which asset needs attention during these windows (such as monthly cleanup of temporary storage areas). How many can be scheduled? How much of the system needs to be offline during this window? If one part is offline, what happens to systems downstream from or linked to that part? When are the preferred times for emergency maintenance if there is a choice?

4.2.1 Asset Support Levels

Define requirements for how to structure the support staff to handle the varying levels of support the assets require, especially depending on the support schedule. The monitoring staff would call for support when a system is not performing as expected or within defined thresholds.

• Support hierarchy—Most organizations have multiple levels of support, with increasing technical expertise. Larger organizations can provide support for the majority of issues with cheaper staff in the lower levels.
  

  Level	Description
  Self service (lowest)	This level of support consists of documentation websites or (interactive voice response (IVR) systems with instructions for resolving common situations.
  Service representatives	This level of support has a person with access to scripts or instruction sets for common problems, and possibly some familiarity with the systems, who can talk through basic troubleshooting steps with a caller. This level of support is sometimes called Tier 1.
  Technical support	This level of support has familiarity with the systems and user support tasks. They can work with the caller using their expertise and access to additional functions tools. In larger organizations, there may be more than one technical support layer, up to and including the development organization. This level of support is sometimes called Tier 2.
  On-site technician	This level of support is the highest level, and sends a technician to the site when all lower levels have failed to resolve the caller's issue. This level of support is sometimes called Tier 3.

• Escalation—In organizations with multiple support levels, a process for moving an issue to a higher level is necessary to minimize time spent effectively resolving the issue.

  When to escalate:

  • All options at the current support level have been exhausted without being able to successfully resolve the issue.
  • The issue is on a list of known issues that require a higher level of support expertise.
  • Upon request from the caller.
  • Upon request from the support staff.

  How to escalate:

  • Call in the next higher level directly using a defined process, such as using an on-call pager or phone number.
  • Reassign the issue in a ticket management system to the next level in the hierarchy for that system.
• Specialization—In larger organizations, support organizations can be specialized and dedicated to certain asset types (hardware vs. software vs. network) or systems/applications. The customer base can support the cost of dedicated staff maintaining a single or small number of systems.
• Generalization—Specialization is often not financially feasible in a smaller shop where the EIT staff covers all business systems and often has infrastructure support duties as well. In this case, it is important to assign at least one primary support person to each system as the go to person for support. A secondary EIT staff person can be the primary's backup for vacation and sick days, and cross-training staff where the primary for one system is the backup for another system can spread the responsibility and work effectively.

4.2.2 Asset Support Tools

Support tools come in two categories: asset management (what exists) and asset monitoring (what occurs). Some toolsets may cover both categories. As the business is at least tangentially funding operations and support, there should be a business view of the assets and services provided. Simply communicating with the business in technical terms may not be helpful or understandable when notifying business users of outages or the need for upgrades.

Figure 3. Business View of Operations

• Asset management tools—Many software packages manage asset information. The hard work lies in populating the package's database with the organization's asset catalog, and maintaining the catalog over time. No matter how comprehensive a tool is, it is useless until it has the organization's data loaded.
  Each tool may have one or more sweet spots in the following areas: depth of monitoring (system detail), breadth of monitoring (system and network scans), or user interface (information display and reporting/graphical output).
  
  • Asset detail cataloging—Some packages include options for managing data for each asset (hardware, software, services, and data) such as configuration settings, license and compliance information, contract and contact information, cost and depreciation tracking, requisition and procurement information, and support agreement information. Some of this data must be entered or uploaded as it does not exist on the asset itself.
    Some packages also allow for business views of the services offered, so assets can be tied to business functions. Each EIT service uses one or more assets (or components), and should map to one or more business processes, which then feeds into importance rankings and service level agreement parameters for each EIT service.
  • Asset scanning capabilities—Some packages focus on gathering information from the systems themselves, although there is usually some sort of processing cost while they run. This can be a way to quickly get accurate asset data into the tool's database.
    While system scans can replace some data entry effort up front, there is a lot of work to do making sure that the tool has the proper access to all the systems in order to run the scans, and each tool may not work with every system component or network connection in the organization. In those cases, some other tool can be used to dump data into a format that the asset management tool can import, but of course, that is a manual process with risks of errors or omissions.
    When the assets have been entered into the tool's database, review the data to ensure that it meets the expectations and has read the system data correctly. Some configuration is required to set up the frequency of updates and re-scan functions, as well as notification and alert thresholds and contacts.
  • User interface and reporting/analysis—Some packages focus on the ability to track and manage support issues for asset items, with analysis for trends in incidents or problems with certain assets. These tools commonly support tracking change requests and enhancement requests. Users may also report issues with assets that need to be resolved either via an enhancement change request or through sustainment. There should be a process for notifying the appropriate teams of relevant issues. See the Change initiatives and Sustainment chapters for more on this topic.
    User interfaces for lookup and reporting can either be installed clients or web-based. Client-installed versions can enable data entry and other auditable activities for authorized users, and may have more intuitive user interfaces for interacting with the tool in order to enter or modify asset data. However, web-based interfaces are much easier to deploy and manage for large numbers of users.
    Some packages include options for impact and dependency analysis to help determine if a change would have adverse or unintended effects, as well as alarms and notifications for license expiration or version updates from vendors.
• Asset monitoring tools
  
  • Monitoring capabilities—Some packages focus on being able to monitor components to very great detail. However, there may be a limited list of systems in which the tool can function to that level of detail.
  • User interface—Some packages focus on the ability to display monitoring data in useful formats, in both reports and in real-time on scrolling or continually updating monitors. These may also have more intuitive user interfaces for interacting with the tool in order to enter or modify system data or schedules.

4.3 Asset Support Incidents and Recovery

Support organizations exist not only to keep the systems operating normally, but to handle exceptions to normal operations—direct handling for small known exceptions, and calling another team within the organization, or calling in outside support from vendors for more serious incidents or unknown exception situations.

4.3.1 Incidents versus Problems

Incidents (as defined in ITIL) are interruptions in normal processing on production systems. If a scheduled process does not start when expected, unexpectedly stops while executing, or executes longer than expected and somehow prevents something else from executing, it is an incident. Incidents causing incidents in other processes are considered related. ITIL also includes as incidents the failure of an asset that has not yet caused an interruption.

It is possible to resolve an incident with a workaround, but the underlying cause needs to be recorded as a problem.

Problems are situations where processes are executing, but not as normally expected, and the difference is temporarily tolerable. Problems become either defect reports or enhancement requests submitted to the change request queue. Identification as a defect report means that the change request alerts those responsible for defect correction and patch release planning. Identification as a change request means that the change request alerts those responsible for planning changes for future releases of the product.

Incidents can result in problems, but problems are not incidents. One way to understand the difference is to consider a traffic accident.

1. A jack-knifed semi-truck lies across all lanes of traffic (an incident). Traffic is stopped.
2. Incident management (police, firemen, tow truck) is called.
3. Traffic is routed to the shoulder (workaround), so the incident is resolved. Traffic movement is restored, but not as much or as fast as normally expected.

There are now two remaining issues:

• The first problem is that some lanes are still blocked. Getting the truck moved to the side helps, and removing the truck entirely restores traffic to the pre-incident state.
• The second problem is to understand and address the reason why the truck jack-knifed in the first place—the weather, the road, the traffic, the truck, the driver, or some combination.
  
  • If caused by the weather (bad conditions), the incident management team can resolve the issue by communicating that road conditions are bad, while salting the road (preventing future incidents).
  • If caused by an issue with the road, the team can patch the bad area and schedule a more permanent repair.
  • If the level of traffic or the traffic pattern is the reason for the incident, the team can examine how the traffic deteriorated to cause the accident, and recommend changes to the traffic pattern, such as adding turn signals or turn lanes.
  • If the truck caused the issue, the owner can fix the truck.
  • If the driver caused the issue, the driver be retrained.

Incident management is the process of restoring normal operations as quickly as possible with minimal business impact. This is an emergency, where getting processes working now is more important than making the system permanently stable. A problem occurred, causing an incident.

Problem management is the process of removing causes of incidents that have occurred, or preventing potential incidents from occurring in the first place. Some problems may be so rare that a permanent remedy could be more costly than the incident causes, so problem management also includes prioritization. Problems turn into change requests submitted to a change request queue for evaluation (see Change Request Queue Management).

4.3.2 Incident Notification or Detection

Incidents are detected automatically by monitoring processes, or are reported by users. Support teams may have standard email addresses or phone numbers for support calls. A survey taken by Everbridge [1] includes the following findings.

• Organizations averaged 150 EIT incidents a year, each taking an average of 2.25 hours to resolve.
• Reporting outages by phone alone is insufficient.
• One third of respondents reported difficulty quickly connecting to the right on-call support person to handle the outage.
• Almost unanimously, respondents reported incidents of late or no response from the assigned support team.
• Of the five most reported incident types (hardware failure, application outage, datacenter network outage or performances issues, or connectivity issues between sites and offices), network connectivity of some kind seems to be the most common issue. It follows that the survey also reported that network administrators were the most common type of support staff pulled into incident resolution efforts.
• Two thirds said that incidents caused personal stress or increased workloads for staff members.

Mature organizations have reporting processes, and automated responses for every step in the process of resolving the incident. These reporting mechanisms can be as simple as an email sent to a team mailbox, or as sophisticated as using a standard ticket assignment tool. Both of these can provide logs of the issue's history and tracking for future reference.

It is important to keep the person reporting the incident updated, as well as anyone affected by the outage. Lack of information leads to a bad impression of the support team, no matter how well they resolve the problem.

4.3.3 Incident Resolution

By nature, incidents are unpredictable, and therefore uncomfortable for all involved. Mature organizations plan for scheduling a good mix of skilled resources to be available at all times, and funding for staff enhancement when needed during incidents so that support staff members are not overwhelmed during a major incident.

Link known incidents to standard remedies to aid support staff in efficient resolution of certain expected situations. Below are two ends of the ranges of incidents:

• Minor incident—Process does not execute as expected. For example, an expected file is late in arriving from a supplier, preventing a process from executing. The problem is a file arrival issue.
  
  • If the cause is that there was no data to send, then the incident is resolved by removing the requirement for the file for this one instance. The problem is that there may be no data for multiple execution instances, requiring a resolution of some kind that can automatically handle the situation without causing an incident, such as the supplier sends an empty file or the application wait times out.
  • If the cause is that there was a delay in processing on their side, then the incident is resolved only when the file arrives. The problem is then that the SLA between the supplier and the application owner is not being met, requiring a resolution of some kind (the SLA to be revisited and the schedule adjusted, the dependency on that file arriving on time is removed, or some other solution).
  • If this incident occurs frequently, the problem importance can be elevated to a higher priority for resolution.
• Major incident—A bug fix patch, upgrade, or enhancement implementation experienced an unexpected issue, causing an unexpected system restart. All processing stops without notice.
  • Automatic restart—If the system processing restarts automatically, there could be problems with the data, depending on where in the processing the halt occurred, and how robust the processing is for handling exceptions like unexpected restarts. It could be that incidents or problems occur for processes and data unrelated to the upgrade or enhancement due to co-existence on the same system.
  • Unsuccessful restart—If the system processing does not successfully restart, or the system does not restart at all, the incident continues until either a restore occurs from a backup taken prior to the change, or the system restarts successfully after some other action.
• Major incident—Power outage occurs in the datacenter (see the Disaster Recovery chapter for more on this topic).

Ideally, each asset has documentation on normal operations expectations, remedial steps for known exception occurrences, and a guide for troubleshooting when unexpected exceptions to normal operations occur.

4.3.4 Resolution Notification

In most organizations, whenever an incident occurs, all users affected by the incident receive a notice that includes what happened, what actions are occurring to resolve the incident, and an expected time of return to normal operations.

It is important to communicate with users at least every hour during an incident, even if the communication is only that work is continuing to determine how to restore operations as quickly as possible. Assign one person to handle user notifications during the incident to prevent confusing or conflicting messages to the users. Some service level agreements can specify the frequency of communications during an incident (see Service Level Agreements).

When the incident is resolved, communicate the status to the users as well as the next steps for determining the cause and resolution of any problems detected.

4.3.5 Closed-loop Analysis

Closed-loop processes provide the data needed for process improvement. Monitoring of assets is not sufficient by itself; monitoring of support instances and resolutions is necessary to identify trends, and then develop improved remediation actions for frequently occurring issues. To improve resolution processes, it is essential to track the disposition of each incident and problem. For example, there must be a repository and a process that records whether each incident or problem resulted in a change (stating why and where the change was made), whether it was deferred and marked for future action, or was dismissed.

Asset support teams can track the frequency of issues, and reduce time spent resolving issues in several ways:

• Reduce the time spent for successful resolution:
  • Develop and deploy better self-service instructions to reduce Tier 1 calls.
  • Modify Tier 1 scripts to identify if the issue needs escalation immediately, rather than trying other solutions first.
  • Modify Tier 1 scripts to use the optimal solution, based on analysis of prior instances.
• Reduce the frequency of the issue:
  • Create a problem ticket or enhancement request requiring a system change that prevents the problem from occurring.
  • Create a problem ticket or enhancement request that handles the issue automatically, avoiding support calls.

5 User Management

Users use assets. Asset management enables business processes to occur predictably. Therefore, EIT must proactively manage users to enable controlled access and prevent chaos. Depending on the organization, funding for user support may come from EIT as a provided service; otherwise, funding comes from the departments on a usage basis.

Always communicate with the business users in business terms to ensure clear understanding, as most business users do not know technical terms or understand technical details.

5.1 User Access and Security Administration

Users require access to systems and applications to do their work. Systems and applications require information to authenticate users and enable access to authorized activity. Both sides need to agree. The most common method is to have profiles for users, and security authentication on the system side.

5.1.1 User Profiles and User Logins

A user profile is a collection of data about (a digital profile of) an individual or identity. Users within a single database typically share similar or identical qualities/characteristics. A profile often has information about the user's job or department, usage patterns, level of experience, location, and an ID. Often, user credentials are associated with a user profile and that is how the user is given access to applications, systems, or networks.

When a user enters an organization, the service department creates a user profile and associated login credentials. When a user leaves the organization, the EIT team should promptly disable their profile to prevent unauthorized access and potential negative actions. Some users might require multiple user profiles (IDs) each covering a different situation, such as one for normal work and another for when performing maintenance or recovery operations.

Applications have access points where users connect to the application or system. There are three aspects to enabling a secure connection between a user and a system:

• Authentication—Each access request must verify that the user has access to the system. This can be a user ID and password, a certificate, a token, or a notification or pass-through from another security system like Active Directory or LDAP.
• Authority—Each user has authority to perform functions within the application. Functions can be enabled or disabled based on the user's assigned authority.
• Auditing or accounting—Every access request to the application is logged for review, regardless of success.

There are three common ways to assign access rights to users:

• User-based—Each user has an individual login/profile with assigned rights and privileges to various systems. Mass changes are difficult and time consuming, but this method has the most flexibility.
• Job-based—Each defined job has rights and privileges assigned to it. Users are linked to jobs so they can inherit the rights and privileges associated with that job. When job responsibilities change, all users linked to that job inherit the changes. Mass changes are easier, but some changes may not apply. This method requires a way to maintain the job definitions.
• Role-based—Each system has defined roles with access rights to certain parts of the system or application. Some systems can have identical roles, so one role can share access to multiple applications for the same rights. Roles attach to users, and if job functions or access needs change, roles are attached to or detached from the users. Mass changes are simply adding or removing users from roles.

5.1.2 User Support Requirements

Each organization is different, and the needs for all users in the organization are different as well. Needs analysis for user support is fundamental to hiring the right people and selecting the right tools. Two aspects should be considered when determining support requirements:

• Task complexity—Each organization varies in how complex their organizations are. Some user support tasks are simple and frequent, like password resets. Other tasks can require specialized access from certain teams to accomplish. These questions may help guide your requirements design:
  • Which support tasks are common?
  • Which support tasks are deliverable through self-service functions?
  • Which support tasks can be disabled during an emergency?
  • What is the expected task turnaround time for each task?
• User skill level—Each organization varies in the types of users needing support. Some users are familiar with the applications and can perform many support tasks themselves. At the other end of the spectrum, some users are uncomfortable with computers and need additional attention to make them comfortable and successful.

5.2 User Support Processes

There are three main types of support processes for users: help desk, self-service portals, and change request queues. Each of these support processes requires monitoring and auditing to ensure proper performance. Help desks are monitored through side-by-side call monitoring, or after-call reviews. Monitoring ensures standard policy compliance and the proper execution of standard activities. Self-service portals have logs that can be monitored by the support team and change requests are typically monitored by operations. See Change Request Queue Management.

User support auditing is also important. The level and frequency of auditing or review corresponds to the maturity of the EIT organization and the industry requirements. Auditing can be critical for organizations dealing with healthcare or finances, or organizations that deal with human resources information.

5.2.1 Types of User Support Teams

User support can be provided in a number of ways, including embedded support, in-house remote support, and offshore support.

Embedded support means that the EIT support staff is co-located directly with the business unit. Having the support team embedded in business unit makes support convenient from the business side; however, from the EIT side, the main drawback is the possibility of resistance to EIT standards, such as using a ticketing system consistently. If there are enterprise standards, co-location with the business unit can bring extra pressure to ignore standards in favor of business needs.

In-house remote support means that the EIT organization provides support in a separate location from the users. Users contact the support center to report issues and receive support services. Some travel between the remote location, the datacenter, and the user locations occurs when necessary.

Offshore support is provided by an organization in another country. Many companies specialize in remote user support.

5.2.2 Help Desk Support Levels

When setting up a support system, the operations team needs to structure the support staff in the best way to handle the varying levels of support the users require. The team also needs to understand the timing of those needs to set up a support schedule. The help desk staff needs to call for support when a system is not performing as expected or when the help needs are out of their area of expertise. A support team can be structured by hierarchy, escalation, and specialization.

• Hierarchy—Most organizations have multiple levels of support, each increasing in authority. Larger organizations can provide support for a majority of issues with cheaper staff in the lower levels.
  • Service representatives—This level of support has a person with access to scripts or instruction sets for common problems, and possibly some familiarity with the systems, who can talk through basic troubleshooting steps with a caller. This level of support is sometimes called Tier 1.
  • Technical support—This level of support has familiarity with the systems and user support tasks. They can work with the caller using their expertise and access to additional functions tools. In larger organizations, there may be more than one technical support layer, up to and including the development organization. This level of support is sometimes called Tier 2.
  • On-site technician—This level of support is the highest level, and sends a technician to the site when all lower levels have failed to resolve the caller's issue. This level of support is sometimes called Tier 3.
• Escalation—In organizations with multiple support levels, a process for moving an issue to a higher level is necessary to minimize the time spent resolving the issue.
  • When to escalate:
    • All options at a support level have been exhausted without being able to successfully resolve the issue.
    • The issue is on a list of known issues that require higher level of support expertise.
    • Upon request from the caller.
    • Upon request from the help desk staff.
  • How to escalate:
    • Call in the next higher level directly using a defined process, such as using an on-call pager or phone number.
    • Reassign the issue in a ticket management system to the next level in the hierarchy for that system.
• Specialization—In larger organizations, support organizations can be specialized to certain subject areas or applications.
  • Security access—Specialized teams evaluate requests for privileged access.
  • Application deployment—Specialized teams deploy applications to users upon request.

Some organizations may outsource user support; however, the organization is heavily invested in the users and depends on the services, so it follows that knowing the user profiles is critical, regardless of where the support is provided and by what organization.

5.2.3 User Self-Service Support

Examples of self-service functions are password resets or standardized software downloads and installations. Common self-service functions occur through IVR systems or websites where users request the support service without support staff involvement.

5.2.4 User Support Tools

Support tools come in two categories: user management (what exists) and help desk monitoring (what occurs). Some toolsets may cover both categories. As the business is at least tangentially funding operations and support, there should be a business view of the users and services consumed.

User Management Tools

Many software packages manage user information and profile assignments. The hard work lies in populating the package's database with the organization's users, and maintaining the catalog over time. No matter how comprehensive a tool is, it is useless until it has the organization's data loaded, and becomes useless over time if the data becomes stale.

Each tool may have one or more sweet spots in the following areas:

• Profile management—Some packages include options for managing data for each user and profile, such as permission and authorization settings, contact information, and support agreement information. While there may be some useful templates or default profiles included, an organization's user data must be entered or uploaded into the tool's repository.
• Remote connectivity—Being able to log directly in to the user's workstation enables support staff to see what the user sees to determine and resolve the user's issue.
• User interface and reporting/analysis—Some packages focus on the ability to track and manage user support requests, with analysis for trends in incidents or problems with certain assets. These tools commonly support tracking change requests and enhancement requests. Users may also report issues with assets that need to be resolved through either an enhancement via a change request or as a sustainment issue. See the Change initiatives and Sustainment chapters for more on this topic.

Help Desk Monitoring Tools

Other tools are designed to help the help desk support team:

• Monitoring capabilities—Many tools handle monitoring help desk or call center activity. This type of monitoring is not specific to EIT operations and support help desks.
• User interface—Some packages focus on the ability to display monitoring data in useful formats, in both reports and in real-time on scrolling or continually updating monitors.

6 Service and Operational Level Agreements

Service level agreements (SLAs) and operational level agreements (OLAs) are contracts between teams that set expectations for how the teams cooperate with each other, and what each team is responsible for providing to the other team. SLAs contain expectations between a service provider and users consuming the services. An example of an SLA is an agreement between a business unit and EIT regarding desktop services. OLAs contain expectations between service providers who support systems providing services. An example of an OLA is an agreement between network operations, security, and desktop services to manage virtual private network (VPN) software and access for workstations. The figure below illustrates the difference.

Some service providers may not always have direct interfaces with users, and therefore no SLAs.

Figure 4. SLAs and OLAs

6.1 Service Level Agreements (SLAs)

There are two main types of SLAs: generic and negotiated.

• Generic SLAs are created when there is no specific relationship between the provider and the users. Sometimes the users are not part of the organization (external) or are on large teams where negotiation would be impractical. In these cases, the service provider alone may create a generic SLA as a baseline. Users that find the general SLA unsatisfactory can negotiate with the service provider to create an SLA that is specific to their team's needs.
• Negotiated SLAs are between providers and users to set expectations for each team's deliverables to the other teams, what services are provided, the availability of those services, notifications of service interruptions, and if appropriate, rewards for compliance and penalties for non-compliance.

SLAs include multiple sections describing the agreement, the participants, and any required actions. Following are the most common sections included in SLAs.

6.2 Operational Level Agreements (OLAs)

OLAs are between multiple technical teams (participants), and therefore have different contents and cover different areas. OLAs are negotiated between participating teams, and set expectations for each team's deliverables to the other teams, including services provided, the availability for those services, and notifications of service interruptions. Some service providers only have agreements with other service providers, and so do not participate in OLAs with users.

OLAs include multiple sections describing the agreement, the participants, and any required actions. Following are the most common sections included in OLAs.

7 Change Request Queue Management

Change request queues contain requests from multiple areas to correct defects, improve system functionality, or comply with changing external drivers. Most organizations use a ticketing system to record incidents, defect reports, and user requests. These systems can be configured to handle requests for compliance changes required by external drivers. Operations and support teams are logically placed to handle the intake of these requests.

There are three main sources for change requests:

• Defects drive change requests because they are degrading operations and need to be removed. Some defects may affect large systems or large parts of smaller systems. Care must be taken to evaluate the frequency of the defect occurring and the overall effect on the organization's operations. Some defects may not be significant enough to be assigned a high priority, and thus may be deferred for later evaluation, although in some cases, deferring a defect may mean a higher effect later, and a higher cost to remove.
• External drivers demand that changes occur to comply with laws, regulations, or contractual obligations, such as supplier support agreements. Other external drivers may be industry standards, such as for security processing, which are deemed mandatory due to fines or loss of business resulting from non-compliance.
• Strategic changes are those from the business side that improve how the organization does business, either reducing expenses or increasing revenue. Change requests go through a lifecycle, starting with the origination of the need for the change. The figure below shows the lifecycle of change requests.