Our main objective during this project was to generate a responsive report that could give insights on fairness and interpretability measures
in a model-agnostic way. To accomplish this, we generated different models, implemented fairness methods and interpretability techniques. We describe the work below.
Models
Our project focus is determining accurate metrics for fairness and interpretability, so we naturally needed a
varied set of models to test our algorithms on. With that in mind, we decided to train 7 different models on the LendingClub
dataset, with varying parameters and values, so that we could later on test our interpretability metrics on them. It is important to remark
that accuracy was not our prime objective, but we were instead looking for varied models to refine our final deliverable: the dynamic report.
The main models analyzed were:
The training problem was set up as follows: we tried to predict if a particular client was going to default or repay their loan,
given 114 features that describe their situation and characteristics (data points obtained from the publicly available LendingClub data).
The data has a lot of categorical features and unusable predictors, so a round of data cleaning was needed.
We removed useless predictors such as client id and loan id, transformed dates into categoricals and categoricals into one-hot encoded variables,
removed columns with zero variance, and finally applied mean imputation to the cleaned dataset to fill out all the NaN values.
We ended up with a dataset containing 194 features. This was fed to our models and we observed an unreasonably high accuracy on most of them.
After this, we realized that some features clearly contained information posterior to the loan outcome. We removed most during the initial cleaning,
but some remained. Features such as debt_settlement_flag, a flag that indicates if a charged-off borrower is working with a debt settlement company,
were easy to detect as post-facto, but other features such as funded_amnt, the total amount committed by the loan,
were harder to catch: it turned out to be the total amount after a repay
After removing the post-facto features, we ended up with a new model with 147 features that had much more realistic results. We operated
hyperparameter search with a grid search approach on all of our models. We show below the models implemented with their respective metrics and parameters.
Logistic Regression
- Accuracy: 79.8%
- AUC: 0.68
- Max-depth of tree: None
Decision Tree
- Accuracy: 79.8%
- AUC: 0.68
- Max-depth of tree: None
AdaBoost
- Accuracy: 79.9%
- AUC: 0.72
- Number of estimators: 100
XGBoost
- Accuracy: 80.3%
- AUC: 0.73
- Max-depth: 8. Number of estimators: 100
LGBM
- Accuracy: 80.4%
- AUC: 0.73
- Number of leaves: 31
Multi-Layer Perceptron
- Accuracy: 79.1%
- AUC: 0.69
- Hidden layer sizes: 50x50x50
Random Forest
- Accuracy: 79.8%
- AUC: 0.68
- Number of estimators: 100. Min samples per leaf: 5
As this is a highly imbalanced problem, we also obtained ROC curves and confusion matrices for our models, to further present the
performance of our model. These figures are part of the final report that we generate later on. We present some of them below.
Interpretability
Driving understanding for the credit analyst
Local interpretation models are the core of our interpretation
functionality.
Our objective for local interpretations is to take a
black-box model and its training set and return a ranked set
of features that
drive the model outcome.
Back-end: Reasons For Decision
We worked with three primary methodologies to produce local explanations for a given observation:
LIME: First proposed by Ribeiro, Singh, and Guestrin (2016), LIME is an explanation technique that creates an ad hoc
linear model in the local region of the decision manifold. This is done by applying small input perturbations near the observation.
For EXIGO, we use a third-party LIME SDK and add several functionalities (see next section).
Input Gradients: Initially proposed by Hechtlinger (2016), this method approximates the distance to the decision
boundary by taking the gradient of the output with respect to the feature set. We implement this technique in a python module.
Influence Functions: We attempted to utilize the Influence Functions method proposed by Koh and Lang,
a loss-based method that calculates how different features affect the gradient of the in-sample loss.
We ultimately discarded this methodology, though, as it requires a well-defined loss
function and therefore breaks our requirement of working with black-box models.
We attempted to circumvent this issue by reproducing a black-box model with a feed-forward neural network -
this would in theory generate a loss function for a black-box model and hence allow us to utilize Influence Functions.
However, in practice this was unproductive; the neural net was an "approximation of an approximation"
and we had difficulty reproducing an arbitrary model with any practical accuracy.
Back-end: Confidence In Calculated Decision Reasons
Each of our selected feature estimators (LIME and Input Gradients) require a linearization on the
decision manifold in the local region around the observation. As such, there is inherent error in the explanations.
While we cannot eliminate the error, the next best thing is to provide a confidence in the output. We built three methods for evaluating the confidence in the reasons for any given observation:
LIME local perturbation: LIME builds a local linear approximation to the model's decision boundary at the point of interest based on random perturbations to the input vector. We capture the accuracy of LIME's approximation by running multiple instances of the random perturbation and quantifying how the LIME output changes.
LIME stochasticity testing: Further, we perform our own perturbations to the input into LIME and measure the divergence of explanations in the local region. We believe this will capture the linearity error plus errors inherent in the LIME method.
Hessian: The Hessian of the decision manifold is a measure of curvature. As a result, it is naturally suited to providing a measure of local non-linearity, which in turn informs the error of the local approximation.
Front-end: Local Plots used in Exigo
We used the various local explanation methods and their respective confidence measures to produce a number of visualizations for the end-user. These are:
LIME plot: The LIME plot is the standard output used by line, showing feature importances,
with our error bars from the LIME stochasticity testing overlaid above.
Its purpose is to provide the user with most important reasons for each individual decision. 
LICE plot: The LICE plot is an extension of the ICE plot, which computes partial
dependency quantile plots for given features and overlays a what-if analysis for the selected observation.
We run this for the 5 most important features in the dataset (as selected by LIME).
The purpose of this plot is twofold: first, it visualizes the overall shape of the data with respect to each feature,
and secondly, it highlights the observation's relative position in the dataset.
Confidence score (local accuracy): This is simply a one-number visualization of how many of the LIME-perturbed points are accurate to the base model under LIME's linear approximation model.
Local/Global Patterns
Having access to observation-level decision reasons opens up a world of analysis possibilities at the aggregate level. We are able to detect regional or global patterns in these reasons and visualize them to provide unique glimpses into the model's performance and the shape of its input data. The visualizations we performed are:
GRIME: GRIME is an aggregation of most important features, achieved by taking the mean of each local feature importance
across a significant subsample of observations. We note that certain models already offer techniques for
global-level feature importance (RandomForest, XGBoost, etc),
however our model-agnostic solution allows this to be performed on any binary classifier.
We quantify error in this method with the standard deviations of the individual feature importances for each observation.
DICE: The DICE plot is created by dividing the dataset into default-probability quantiles
and using local importance to find the most common positive- and negatively- correlated features
for each quantile. These are overlaid onto a PDF of the dataset, resulting in a visualization
that correlates feature importance with both loan risk as well as overall frequency of observation.
BILE: BILE is our unique method of plotting decision boundaries for important features.
First, KD-trees were selected as a fast ( O(log(n)) ) method for searching
nearest-neighboring observations. We compute these nearest-neighbors when
compressing the highly-dimensional decision boundary into two-dimensional space,
and then apply a contour plot to visualize default probabilities.
The resulting plot shows the interrelation between two important features and
the response variable. We recognize that our KD-Tree based approach represents
a maximum likelihood approach to sampling other non-displayed features in the
dataset. If we approached this from a bayesian perspective, we could get nice
error bounds, but we find the bayesian approach computationally burdensome, so
we stick with the maximum likelihood approach.
Discarded attempts:
PCA/K-Means clustering: We attempted to reduce the dimensionality
using PCA and clustering over the principal components to deduce interesting global-level
insights. In practice, we found that the first few principal components often tended
to be dominated by a single feature. Although this is useful proxy for feature importance,
we already have strong tools for this so we discarded this redundant functionality.
T-SNE: We also attempted to visualize the space with T-SNE.
First we used an initial dimensionality reduction technique to reduce to less than 50 dimensions
(as T-SNE often underperforms on manifolds over 50 dimensions). We attempted to
label clusters using various feature and response combinations. However, the resulting
clusters were occasionally insightful and T-SNE was inherently too unpredictable as a methodology, so we chose to discard it.
Fairness
Why Fairness?
As Artificial Intelligence becomes more powerful, algorithm-based decisions have become increasingly common for various problems for everything from automated lending to parole hearings. With this trend comes the issue of ensuring fairness along with other key aspects such as accountability or transparency. The EU General Data Protection Regulation (GDPR, enacted during the course of this project), regulates that anyone subject to an algorithmic decision has a right to an explanation. As such policies go into effect, explaining a black-box model such as neural network and controlling undesirable patterns in its decision-making become essential. As legal frameworks continue to be established, organizations that use algorithms to make important decisions could face legal risk. If their models were to exhibit statistically significant discrimination, they may face litigation. In particular, our cursory review (the authors are admittedly novices in the practice of law) shows that class action lawsuits are in particular subject to statistical examination due to their stochastic nature. As a first step to codifiyng such issues, we study different ways of quantifying the fairness.
Is My Black-box Model Fair?
We present the three, perhaps most universal, metrics that quantify the level of fairness. These are global fairness metrics because they look at how the model in question makes decisions overall for a protected group vs. the entire population. By protected group, we mean groups based on sensitive characteristics such as gender, religion, or race. When an algorithm is completely rule-based (where each rule is specified manually by a human) it is relatively easy to evaluate its fairness because one can review each individual rule. However, the powerful inference models we use today have complex inner-workings to inspect and hence, usually the best we can do is understand their statistical behaviors. Most of the best inference models we use today take an probabilistic approach and hence, we measure the fairness in terms of expectations. We expand on that approach with a battery of staistical tests for evaluating different fairness conditions:
Statistical Parity
shows an equal portion of protected class and general population is expected to receive loans. The bias can be defined as the absolute difference of the two terms below.
Conditional Parity
shows an equal portion of protected class and general population is expected to receive loans, conditioning on a set of pre-specified features. The bias can be defined in the same way.
Predictive Equality
shows an equal level of accuracy is expected for protected class and general population as measured by False Positive Rate (FPR) and False Negative Rate (FNR).
Implementation & Use Cases
We added a tool to EXIGO that dynamically generates the report that shows each of these evaluation methods, given a model and training data set. Since all these evaluation metrics are statistical tests, we obtain a probability value after running each of them, with which we must assign a "pass" or "fail" result. In all these tests, we are looking for small differences between the two terms of the equations to ensure fairness. We choose a 95% confidence level based on generally accepted statistical practices in industry. Although open to the interpretation of the user, we consider a fairness test to have "passed" if all of the atomic tests above pass.
The evaluations also all come with confidence intervals to show variability in each fairness measure. Why is this useful? Suppose you are a CEO of a financial company that lends money to individuals. Under regulations like Equal Credit Opportunity Act (ECOA), you are obliged to comply with the regulation and should you violate it, your business may face serious legal and financial issues. In addition, suppose you are a credit analyst. This section of the report is important for a credit analyst making a decision for a single loan, but also because it allows the credit analyst to verify generally whether the model treats people fairly on the basis of their membership in various protected classes. This report not only verifies that the assisting model is fair, but (subject to the agreement of legal professionals), it could also provide evidence of "best efforts" and "due diligence" from a legal perspective.
Future Work
There are many open research questions. So far, we assume we are given a fully-trained black-box model and that we cannot change the model. One important question is how can we bake in the notion of fairness to the training of a black-box model from the beginning? A logical next step would be to incorporate the fairness likelihood metrics into the loss function of a model to optimize the model decisions itself. This would also potentially allow for the training of models that are robust against adverarial attempts to "fool" an analyst that a model is fair with unbalanced data sets, etc.
Another interesting direction is to measure the level of fairness in the decisions made for an individual, regardless of the global statistical behavior of the model (local fairness).
Testing
Testing is an essential part of ensuring our dashboard actually performs the task we are building it to perform.
Since our goal for the report is to aid in the loan approval decision a credit analyst must make, it makes sense to
pursue human evaluation testing. Although human evaluation testing is the direction we would like to pursue to
evaluate our report, we sadly do not have access to the number of credit analysts necessary to perform a legitimate
human evaluation test.
As a result, we must leave the testing functionality to further work, but it is an interesting challenge to consider hypothetically. As our results are primarily interaction-driven, it is quite complex to design an unbiased, representative metric of "improved performance" for a credit analyst. The key functionaliy of our proposed test would be:
We propose to test our dashboard with a speed comparison between analysts with and without our dashboard. Ideally, we would run the evaluation for the same observations with multiple analysts, some with, and some without the dashboard.
The testing process will also control for accuracy to make sure we provide accurate, not just decisive, insights. In particular, to control for analyst performance, we would run a control experiment with the dashboard access switched - analysts who previously were ranked with the dashboard would be timed on new observations without dashboard access (and vice versa).
We would consider a human evaluation test like this will be able to validate if the presence of the dashboard makes a statistically significant difference in time per observation.
Further tests could also be performed - for example, designing tests to consider whether the dashboard improves the accuracy of analyst decisions. We would begin, though, with the speed test as accuracy testing is more subject to bias (for example, we would only need to consider "random audit" cases rather than marginal decisions, as applying the accuracy test only to inherently high loan risk cases may introduce unexpected bias.
