How to Use ROC

Receiver Operating Characteristics (ROC) supports what-if analysis. You can use ROC to experiment with modified model settings to observe the effect on the confusion matrix. For example, suppose that a business problem requires that the false-negative value be reduced as much as possible within the confines of a the requirement that the number of positive predictions be less than or equal to some fixed number. For example, you might offer an incentive to each customer predicted to be high-value, but you are constrained by budget to a maximum of 170 incentives. On the other hand, the false negatives represent missed opportunities, so you want to avoid such mistakes.

To use ROC, move the red line and observe the changes in the confusion matrix. As you change the confusion matrix, you are changing the probability that result in a positive prediction. Normally, the probability assigned to each case is examined and if the probability is 0.5 or above, a positive prediction is made. Changing the cost matrix changes the positive prediction threshold to some value other than 0.5, and the changed value is displayed in the first column of the table beneath the graph.