Your corporate decisions are either manual, report-enabled, semi-automated, or fully automated....
Review timing (during or after?)
In-Flight vs. Post-Implementation reviews: model build and refinement
Data-driven (or data-enabled) decision-making hinges on reliability and accuracy.
Assurance activities (a.k.a. reviews, audits) can help ensure trust and confidence in the inputs to those decisions.
For simplicity in this article, we will call those inputs "models".
In practice, they could include data, rules, models, flows, transformations, and a range of other items.
There are two main timing approaches for assurance work:
- In-flight reviews happen during the project or build
- Post-implementation reviews happen after deployment.
In this article, we detail the benefits of both approaches for model build and refinement.
In-Flight Reviews
Auditors conduct these during the active development and deployment of models.
Sometimes auditors call these "active audits". Others use the term "hot reviews".
They are real-time, identifying and addressing issues while the project is in progress.
The key benefit of in-flight reviews is real-time mitigation.
In the dynamic world of model build, issues can surface at any stage of development.
In-flight reviews allow teams to detect and address these issues as they emerge. This real-time problem-solving can prevent small hiccups from snowballing into major roadblocks.
Addressing issues in-flight reduces the risk of costly failures or undesirable outcomes.
The reviews act as a safety net, enabling you to identify and mitigate risks early.
Model development projects must map to business objectives. In-flight reviews help ensure the project continues to remain goal-aligned. This helps guard against misalignment that can derail projects or deliver poor results.
Post-Implementation
Auditors conduct these after your team has deployed your model.
This is the typical approach - retrospective evaluations.
Some auditors call these "cold reviews".
While they don't provide real-time adjustments, they offer other advantages:
-
Maturity. Data quality, and processes, can improve over time. Post-implementation reviews work with a more mature dataset and refined process. This may be more accurate and comprehensive than what was available during development.
-
Resource capacity. Conducting audits during projects can put strain on project resources during development. This becomes easier (in theory) post deployment.
A Hybrid Approach
A hybrid approach combines the advantages of both.
But doing both in the traditional way can be too resource intensive. So you need to have half of each. How can you do that?
-
Emphasise In-Flight. A heavier emphasis on in-flight reviews during the active development and deployment phases.
-
Lightweight Post-Implementation. Focus on the most critical items, rather than exhaustive reviews. Include items that have shifted since implementation. For example
-
Adjust and Refine. Apply insights gained from post-implementation assessments to refine in-flight processes for future projects.
Summing up
The choice between in-flight and post-implementation is not an "either-or" scenario.
It's about finding the right balance that aligns with your assurance needs.
In-flight reviews help uncover issues early.
But they can put a strain on project resources.
Post-implementation reviews help plan for future builds, correction and refinement.
But they could be too late.
The hybrid approach offers the best of both worlds.
But the key is not in choosing one over the other.
Rather, it is about combining the best of each approach.
