FICO Model Central is the latest addition to FICO’s Decision Management platform and is focused on lifecycle management of predictive analytic models in the context of an overall Decision Management implementation. The original driver was a growing regulatory requirement in the banking industry to have a regular cycle of model validation and monitoring of predictive analytic models for performance and stability.
FICO Model Central is the latest addition to FICO’s Decision Management platform and is focused on lifecycle management of predictive analytic models in the context of an overall Decision Management implementation. The original driver was a growing regulatory requirement in the banking industry to have a regular cycle of model validation and monitoring of predictive analytic models for performance and stability. Both domestic US and global regulations have made it essential for banks and other credit issuers to show how their models work and to open up their regular monitoring and updating processes for review. With traditional processes this can result in almost the whole modeling department working on this rather than new models. While most pressing in retail banking, this need for a more visible and manageable analytic process is growing across all industries as organizations adopt predictive analytics.
In this context there are a number of model management challenges including an increasingly large and complex inventory of predictive analytic models, regulatory and internal audit pressures, the time it takes to deploy models and difficulties in reporting, tracking and validating deployed predictive analytic models. These challenges are most intense in financial services but most organizations using predictive analytic models have them. Organizations asked about these issues identified a set of features that they needed:
- A web-based dashboard for managing potentially large numbers of models
- Automated model validation
- Reports and rules-based alerts when models drift out of alignment, stability metrics drift from historical baselines or accuracy decreases
- Rapid deployment of models to operational systems
- Decision Simulation and optimization to see how the models influence decision outcomes
Model Central is FICO’s response and has three product tiers. A foundation tier provides a web-based dashboard for tracking and monitoring, automated model validation, alerts for model degradation and management reporting. The Professional Development version adds the Model Builder environment to develop new models, variable library management, model diagnostics, a common repository and model refresh. The Advanced Decisioning version adds simulation software, an impact analysis framework and the ability to optimize a portfolio of decisions using FICO Decision Optimizer.
Core to all tiers, the main dashboard lists models by name, portfolio, type, status and more. Portfolios can be added and removed freely and models are allocated to these portfolios when they are registered. The modeling technology can also be specified using a set of user-defined types. Models can be singletons and can also be composite models (such as one with a model for each of several customer segments). The date of last validation is tracked as is the result of any alerts defined for the model (the dashboard shows how many are currently triggered). After reviewing validation results, Model Central users update the status of the model (OK, rebuild, watch list or whatever the company wants) as appropriate. All these properties can be used to filter the list to just those with a specific status or portfolio for instance so you can manage the overall portfolio of models.
The Professional Development tier provides full versioning of the models and rules through the Blaze Advisor repository while the Foundation tier allows new versions of a model to be registered within the existing model. Summary metrics and drill down reports can easily be compared across multiple versions of the model.
Each model has a home page with the history of the model’s performance over time (both standard metrics defined as part of the product and potentially custom ones generated using groovy scripting), any alerts triggered, comments, graphs of performance over time etc. More detailed reports can be viewed from this environment. Model documentation can be attached such as reports generated during development, underlying methodology documentation etc.
Model validation is handled by a runtime that allows model reports to be run on a scheduled or ad-hoc basis. This uses data defined in Model Central (you can manage files, database connections and data import scripts for this) and brings in data required to re-test models. Essentially queries are defined for testing and validation that pull the data used by the model and the scores that were calculated. These can be simple, focusing only on scores and business outcomes, or can include predictive variables so that additional reports on changes in the underlying data can be run. This external interface approach means that models can be “black boxes” as far as Model Central is concerned and that it uses performance data from production systems.
Model workflow like follow-up emails for validation review or approval etc can also be managed and the rules for alerts are specified using a standard web-based rule interface. Models are registered through importing PMML, published from FICO Model Builder or defined explicitly (because the environment does not need to know HOW it works, only how to see how well it is working). If a model definition is available from Model Builder then this drives the data interface required to monitor the model. For explicit models this must be defined manually, though it can be reused. For the more advanced simulation and testing capabilities the platform needs to know how to score a model but for the core reporting features it does not even need to know that – it can treat the model as completely opaque.
Model Central packages up a number of functions currently available in Model Builder and Blaze Advisor to deploy models faster. Model Central Foundation packages capabilities from Model Builder (calculate performance metrics and generate validation reports), and Blaze Advisor (provide repository metadata storage, define and scheduling validations jobs, define and manage validation rules and automated alerts) while Model Central Professional Development includes a full Model Builder license and a limited license to Blaze Advisor for Model Deployment purposes. This enables teams to deploy predictive models quickly, without recoding, directly into the decision services they were built to enhance. Models are expressed and managed as business rules that enjoy complete lifecycle management and governance. Models built from other platforms can be brought into this system and new models deployed this way can be seamlessly imported into Model Central to support ongoing monitoring.
