There are ‘No Slam Dunks in IT.’ Many who have worked in technology operations agree, but this statement is often discounted in organizations.
IT professionals often get caught up in the urgency of the moment, short cut change management procedures, or fail to think about the downstream impact of what they see as a minor, isolated change. The mindset of “the easy change” or “the lay-up” ends up as an unexpected outage. That break away slam dunk clanks off the rim and bounces out of bounds. That easy two points turns into a turnover.
The need for change management in an organization’s technology ecosystem is widely acknowledged and accepted. It would be uncommon to walk into any technology department and not find numerous documented processes and procedures related to change management. Following strong change management processes can be a key contributor to operational stability, which leads to improved organizational performance.
Audit functions, both internal and external, emphasize IT change management as well. The Institute of Internal Auditors recognizes the importance of IT change management through the publishing of its Global Technology Audit Guide: IT Change Management: Critical for Organizational Success.
The guide provides information to auditors about how to distinguish effective change management processes from ineffective ones, recognize red flags and indicators that IT environments are having control issues related to change management, and understand that effective change management hinges on implementing appropriate preventive, detective, and corrective controls to ensure adequate management supervision.
The AICPA also recognizes the importance of change management. The 2017 SOC 2 Trust Services Criteria includes an entire section devoted to it. Common Criteria 8.1 states, “The entity authorizes, designs, develops or acquires, configures, documents, tests, approves, and implements changes to infrastructure, data, software, and procedures to meet its objectives.”
The volume of guidance and principles is large, but the fundamentals of change management boil down to these five steps:
Plan. Make sure each change action or project is well thought out, with steps documented, risks assessed, rollback steps known and plans in place to respond to any post-deployment issues. If disruption in service is expected, plan to limit the impact of the disruption.
Communicate Before the Change. Before executing the change, communicate each change action or project to the parties that will be potentially impacted. A Business Impact Analysis report can be a great source for understanding potential departments and users impacted by changes, especially those that are application- specific. The communication should provide information on the nature of the change, the timing, any known negative impacts that could arise, and any actions required by the users related to the change.
Execute. Execute the change according to the documented plan. If issues arise during execution, adhere to the established plans around when to roll back the change. If the change is scheduled to happen during a defined maintenance window, keep a close watch on the clock so that changes and testing are completed during the defined window.
Test. Test to make sure the change achieved the expected results and there are no unintended consequences from the change. Testing should be conducted by technical resources and by end users, when applicable. If issues are found during testing, have a clear process for providing the results to the group executing the change and have a clearly defined decision path for determining when to roll back a change.
Communicate After the Change. Once the change has been completed and tested, communicate to the potentially impacted parties. Include specific instructions for how to report any issues that occur post change. IT Help Desk staff should also be aware of the changes and receive clear instructions on how to escalate issues resulting from recent changes.
Following these basic steps can improve integrity and reliability of technology, regardless of whether it is managed and hosted internally or provided by a third-party.
Managing and validating adherence to effective change management is more straight-forward for internally hosted and managed solutions, but there are ways to work with external third-party providers to ensure awareness of change management. With the increasing reliance on external partners to provide mission critical IT services, having an approach to govern and monitor the changes in externally provided and managed solutions is more important than ever.
Some key governance and monitoring actions include:
- Understand the service level agreement commitments and the available recourses when service levels are not maintained.
- Know where to monitor the solution’s health and availability. Many providers have health dashboards that report on known issues, provide information on service restoration, and show overall availability metrics.
- When system maintenance or change notices are received from a service provider, read the notice, assess the potential impact to the organization, communicate the upcoming change to the relevant areas of the organization and consider logging the change in the organization’s internal change management tracking system. If the announced change is optional or provides for a selection of a date/time for the execution of the change, IT functions should collaborate with potentially impacted user groups to determine when to move forward with the change.
- Know how to report availability issues to service providers, how efforts to restore services are communicated, and what kinds of root cause analysis is done and reported by the service provider.
- When outages of a third-party provided IT service occur, communicate internally to the organization in the same manner as an internally hosted, managed solution.
- When selecting a new service provider, request a copy of its Service Organization Control (SOC) report or a similar report to use as part of the evaluation process. For existing service providers, request and review SOC reports on an annual basis.
Organizations should regularly assess the design and effectiveness of IT change management processes. These assessments can be performed within the IT function, by other internal functions like Internal Audit, or by an external party with the skills and experience to review and assess these critical IT management processes.
By consistently following sound IT change management processes and avoiding disruptive self-inflicted IT outages, the IT function can become the organizational all-star that can reliably make the game winning shot.
For information about IT change management for your organization, contact us. We are here to help.
In the spring of 2020, as organizations made a rapid shift to a remote workforce, their IT teams had to implement, update…