How to review an Account Fit model

Context 

In Rev.Up, you can use Account Fit (Custom Event) models to create fit ratings for your accounts. This will enable your sales and marketing teams to focus their spending and effort on the right accounts. Once an Account Fit model is built, it is important to review the model before activating it to ensure that the model will perform well for the use case for which it was trained. This guide describes best practices for evaluating model quality. 

Step 1: Check model warnings

Rev.Up shows warnings about attributes that are used by the model but may be problematic.

To check model warnings, open the Attribute List screen and click the Warnings link under the green summary banner. This will show the attributes with warnings.

There are four types of warnings:

• Prediction from later data (future information): This attribute looks like it was populated later in the business cycle and contains future information. This warning appears when available attribute values show a good lift (greater than 1.5x), but the attribute is not populated in at least 80% of the records, and these records have a low lift (below 0.6x).
• Prediction from missing data: The model uses this attribute to make predictions based on missing data. When unpopulated values of this attribute can be used for significant prediction (lift is less than 0.7 or greater than 1.2), later model scores are often inaccurate.
• Too many category values: This attribute has more than 200 category values. Such attributes cannot be used in modeling. Where possible, replace this free text field attribute with a pick list attribute. 
• Too many identical values: This attribute has the same value for 98% or more records. This
  can lead to inaccurate model scores.

For optimal model performance, attributes with warnings generally should not be used in a model.

If you see attributes with warnings, review the attributes from a business perspective. You may keep the attribute with a warning in the model only if there is a good  business justification do so. Otherwise, remove the attribute from the model.

Step 2: Review model health score

If the model health score is below 0.65, ensure that the data used to train the model is good quality data: 

• Use at least 10,000 accounts for training an Account Fit model. 
• Ensure that the training data sufficient number of conversions and an optimal conversion rate (see step 3).  
• The training data should have at least some field values with meaningful differences between successful and non successful events.
• Having mostly empty or unmatched values in the majority of attributes used by the model can result in inaccurate model predictions. This is especially true for records with successful events when the number of such events is small. 
• The data and the success criteria (event column) should be accurate. A model trained with inaccurate or contradictory data will likely produce inaccurate results. 

• An attribute in which values are populated only after the account reaches a particular marketing or sales stage is an attribute containing future information.
• Attributes for which the "Not Populated" value has a lift below 0.3x are ones that likely contain future information. 

Step 3: Review conversions and conversion rate

Providing an appropriate number of conversions and an appropriate conversion rate is essential to ensuring that data scarcity will not negatively impact the performance of the model.

To review the number of conversions and the conversion rate, view the "Total Conversions" box in the  summary banner at the top of the Attribute List screen.

• Total conversions is the number of success events (1s in the event column) in the data that you provided for model training.
• The displayed conversion rate is the percentage of such success events in the data provided for model training. 

For optimal model performance, aim for at least 500 conversions and the conversion rate between 1% and 5%. In general, the conversion rate in the training data should be similar to the conversion rate that you observe in the real world.

If too few conversions are provided or the conversion rate is too low, the model may be less accurate. If the conversion rate is too high, the maximum lift (which is relative to the average conversion rate) may be lower, hence it will be more difficult to compare the performance of this model with other models. 

If the total number of conversions is below 500 and/or the conversion rate is below 1%, try one of the following:

• Provide more success events in your training data, if available.
• Provide fewer non-success events in your training data. Only use this option if you can retain at least 10,000 records in the training data.
• Adjust your definition of a success event to make it less restrictive. For instance, if you are creating a model to predict which accounts are likely to convert but have a conversion rate lower than 1%, you could instead try using accounts with opportunities as your success event. The model will now predict which accounts are likely to have an opportunity with your business.

If the conversion rate is above 5%, and it is above the typical conversion rate that you observe in the real world, try one of the following:

• Provide more non-success events in the training data.
• Provide fewer success events in the training data. Only use this option if you can retain at least 500 success events in your training set.
• Adjust your definition of a success event to make it more restrictive. For instance, if you are creating a model to predict which accounts are likely to convert but have a conversion rate above 5%, you could instead try using converted accounts that have high spending as your success events. The model will now predict which accounts are likely to convert and have high spending with your business.

Step 4: Review the Lift chart

The Performance screen shows how the model is performing on a set of test accounts. A random 80% of the accounts that you provide are used to train the model. The remaining 20% of the accounts are held out to test and measure model performance. This is a machine learning best practice that helps ensure the model is performing correctly. 

When viewing the model, click the Performance tab and see the Lift bar chart that shows how the model scored the accounts in the test data.

The bars represent score deciles. The first bar represents test accounts with scores from 91 to 100; the second bar represents accounts with scores from 81 to 90; and so on.

The height of the bar represents the lift for this score decile relative to the average lift. The horizontal dotted line denotes the average 1.0x lift, which corresponds to the average conversion rate in your training data.

A decent model is one in which the first decile has a high lift, the chart has a clear segmentation between deciles and the decile bars generally get smaller toward the right side of the
chart.

If your lift chart does meet these guidelines, try one of the following:

• Check the Attribute List for attributes where the "Not Populated" value has a lift below 0.3x, as these attributes likely contain future information. More detail is provided in the Step 6: Check ‘Not Populated’ Lift section of this guide. Strongly consider removing these attributes from the model.
• Check the Attribute List for Firmographics attributes with more than 40% "Not Populated" values, and non-Firmographics attributes with more than 70% "Not Populated" values. More detail is provided in the Step 5: Check Attribute Population section of this guide. Strongly consider removing these attributes from the model.
• Review conversions and the conversion rate to make sure they are in the recommended range. Refer to the Step 3: Review Conversions And Conversion Rate section of this guide. 

Step 5: Check attributes with "Not Populated" values

A common problem is when the training data has many accounts that do not match to the D&B Data Cloud. The resulting model scores unmatched accounts either very high or very low. This situation should be avoided.

To check match rates and ensure that the attributes selected for the model have enough populated values, go to the Attribute List screen and click the Used by Model link under the summary banner. Locate the "Not Populated" value (if displayed) on each attribute and check its "% Accts" column. This is the percentage of accounts from the training data that do not have a value for this attribute, which may be because these accounts did not match to the D&B Data Cloud.

Healthy population of the attribute depends on the attribute category:

• Data in the Firmographics category is usually highly available. For optimal model performance, aim for no more than 40% "Not Populated" values for Firmographics attributes.
• In other categories (like Technology Profile, Online Presence and Website Keywords), data is often less available. For optimal model performance, aim for no more than 70% "Not Populated" on attributes in non-Firmographics categories.

If an attribute used by the model has too many "Not Populated" values, review the attribute from a business perspective. If there is business justification to keep an attribute with a high percentage of "Not Populated" values in the model, do so. Otherwise, remove it from the model.

Step 6: Check the lift of "Not Populated" values

It is not desirable when the absence of data drives model predictions. In a good model, predictions come from specific populated values of internal and D&B Data Cloud attributes.

To check the lift of "Not Populated" values for attributes selected for the model, go to the Attribute List screen and click the Used by Model link under the summary banner. View the Lift column for the "Not Populated" value on each attribute.

The Lift column represents how likely records with each attribute value are to convert compared to the average conversion rate. The 1.0x lift represents the average conversion rate.

An attribute is problematic if the "Not Populated" value has a lift smaller than 0.6x or larger than 1.2x. This shows that the lack of data is predictive but generally this should not be the case.

If an attribute used by the model has a "Not Populated" value with the lift smaller than 0.6x or larger than 1.2x, review the attribute from a business perspective. If there is business justification to keep such attribute in the model, do so. Otherwise, remove it from the model. 

Step 7: Review feature importance

As a final step, advanced users may want to review the feature importance file.

Each attribute that the model uses to make predictions is represented by a row in the feature importance CSV file. The Feature Importance column represents an attribute’s relative importance in predicting the outcome. An attribute with feature importance above 0.05 may be a potential issue because it may represent overfitting.

Note: the file may have fewer columns (depending on the model type) but the Feature Importance column will always be present.  

To download the feature importance file, click the Model Summary tab. From the Model Diagnostics section, download the RF Model CSV file and inspect the Feature Importance column.

If an attribute has a feature importance above 0.05, review the attribute from a business perspective:

• If you are certain the attribute does not contain future information (is not influenced in any way by your success event), or if there is business justification to keep the attribute, do so.
• If you determine that the attribute may contain future information, or if you do not need the model to make predictions based on this attribute, remove it from the model. Note that the reported model health score may slightly decrease after removing such attributes.