Exploring the accuracy-explainability trade-off on credit scoring classifiers

Abstract

Recent research has highlighted the significance of accuracy and explainability of classification models applied across various disciplines. A wide range of classification models and combinations of models have been extensively studied to determine those with superior performance. These studies demonstrate that models that tend to be more accurate are also difficult to understand; there appears to be a trade-off between accuracy and explainability. Consequently, this has led to an increased focus on explainable artificial intelligence, a field of research concerned with explaining model predictions. Although explainable artificial intelligence is an area of research with growing popularity in the science community, there are still limited case studies that explore its applications in credit default risk. Credit default risk refers to the potential financial loss or risk that is incurred by a credit provider when an obligor fails to meet their debt obligations. To quantify, mitigate and manage the risk associated with granting credit proactively, credit providers utilise scoring classifiers to assess the risk of credit applicants prior to granting credit. Furthermore, credit risk providers are legally required to explain predictions of scoring classifiers. Popular classifiers used in credit risk include logistic regression, discriminant analysis, decision trees, random forests, bootstrap aggregation, neural networks, support vector machines and gradient boosting algorithms. Logistic regression and discriminant analysis are widely adopted in the financial industry because they perform reasonably well and are inherently interpretable. However, these approaches are giving way to alternative approaches that offer improved accuracy in risk assessment, even though these alternatives lack interpretability; they are less comprehensible and are often regarded as black boxes. This lack of interpretability has resulted in a reluctance to adopt these alternative techniques in credit granting. The aim of this study is to remove the aforementioned barrier of using black box models by utilising explainable artificial intelligence methods, such as Shapley additive explanations and local interpretable model-agnostic explanations. The study also examines the accuracy-explainability trade-off of different classifiers by developing and evaluating eight classification models on two publicly available credit datasets. Eight classification models were constructed, including decision trees, logistic regression, linear discriminant analysis, support vector machines, artificial neural networks, bootstrap aggregation, random forest, and light gradient boosting classifier. Their performance and interpretability were assessed after training and tuning the hyperparameters for optimal comparison on training, testing and validation subsets of the data. Performance accuracy was measured using the area under the curve on 30 random subsets generated from the validation data. Furthermore, the Kruskal Wallis test and Dunn’s multi-comparison test were used to rank the predictive models by accuracy and to determine if the differences in mean accuracy are statistically significant. The interpretability of these classifiers was conducted for both transparent and black box models. To achieve these ends, key preprocessing steps were developed to reduce the complexities of local and global model interpretation. In addition, Shapley additive explanations and local interpretable model-agnostic explanations were utilised to analyse the relative importance of features and the impact on predictions. The experiments show that the artificial neural network, ensembles and other treebased algorithms significantly outperform logistic regression and linear discriminant analysis in the first case study. However, contradictory results are obtained for the second case study, as the performance of the classifiers are relatively comparable. This indicates that model performance depends on the data from which the models are constructed. These two case studies show that the perceived trade-off between accuracy and explainability does not always hold true. Furthermore, Shapley additive explanations yielded results that are consistent with the intrinsic interpretability results of the transparent methods. This post-hoc interpretability enables us to understand how the predictions are made and what factors contributed to the prediction. This is important to create a reliable and trustworthy framework that uses black box models for credit decisions. The research highlights the benefits of using alternative methods for credit risk scoring, showing that the performance can vary significantly. It also demonstrates the effectiveness of Shapley additive explanations and local interpretable modelagnostic explanations to explain predictions of black box classifiers. However, it identifies challenges in using the Shapley additive explanations. The mean absolute value may be sensitive to outliers, which could have an impact on feature importance. Therefore, further work is required to enhance the efficiency of calculating Shapley additive explanations’ values for linear classifiers and some ensembles.

Onlangse navorsing het die belangrikheid uitgelig van die akkuraatheid en verduidelikbaarheid van klassifikasiemodelle wat dwarsoor verskeie dissiplines toegepas word. ’n Wye reeks klassifikasiemodelle en modelkombinasies is omvattend bestudeer om daardie modelle met voortreflike prestasie te bepaal. Hierdie studies het gedemonstreer dat modelle wat neig om meer akkuraat te wees, ook moeilik is om te verstaan; dit kom voor of daar ’n kompromie is tussen akkuraatheid en verduidelikbaarheid. Dit het gevolglik aanleiding gegee tot ’n verhoogde fokus op verduidelikbare kunsmatige intelligensie, ’n navorsingsveld wat met die verduideliking van modelvoorspellings gemoeid is. Alhoewel verduidelikbare kunsmatige intelligensie ’n navorsingsgebied is wat besig is om in gewildheid toe te neem binne die wetenskapgemeenskap, is daar steeds beperkte gevallestudies wat die toepassing daarvan op kredietwanbetalingsrisiko ondersoek. Kredietwanbetalingsrisiko verwys na die potensi¨ele finansi¨ele verlies of risiko waaraan ’n kredietverskaffer blootgestel word wanneer ’n skuldenaar in gebreke bly om hul skuldverpligtinge na te kom. Ten einde die risiko wat met kredietverskaffing geassosieer word proaktief te kwantifiseer, versag en bestuur, moet kredietverskaffers kredietgraderingsklassifiseerders gebruik om die moontlike risiko te evalueer wat kredietaansoekers inhou, voordat krediet toegestaan word. Voorts is kredietrisikoverskaffers volgens wet verplig om die voorspellings van kredietgraderingsklasifiseerders te verduidelik. Gewilde klassifiseerders wat in kredietrisiko gebruik word, sluit logistieke regressie, diskriminantanalise, besluitnemingsbome, ewekansige woude, skoenlussamevoeging, neurale netwerke, ondersteuningsvektormasjiene en gradi¨entversterkingsalgoritmes in. Logistieke regressie en diskriminantanalise is algemeen deur die finansi¨ele bedryf aanvaar aangesien hulle redelik goed presteer en inherent verduidelikbaar is. Hierdie benaderings skep egter ruimte vir alternatiewe benaderings wat verbeterde akkuraatheid ten opsigte van risiko-assessering bied selfs al gaan hierdie alternatiewe benaderings mank aan interpreteerbaarheid; hulle is nie so verstaanbaar nie en word dikwels as swartkissies (black boxes) gesien. Hierdie gebrek aan interpreteerbaarheid het tot gevolg dat daar ’n traagheid is om hierdie alternatiewe kredietverleningstegnieke aan te neem. Hierdie studie het ten doel om die voorafgenoemde versperring tot die gebruik van swartkissiemodelle te verwyder deur verduidelikbare kunsmatige intelligensiemetodes soos Shapely se additiewe verduidelikings en plaaslike interpreteerbare model-agnostiese verklarings te gebruik. Die studie ondersoek ook die akkuraatheidverduidelikbaarheidskompromie van verskillende klassifiseerders deur agt klassifikasiemodelle vir twee openbaar beskikbare kredietdatastelle te ontwikkel en te evalueer. Agt klassifikasiemodelle is saamgestel, naamlik besluitnemingsbome, logistieke regressie, liniˆere diskriminantanalise, ondersteuningsvektormasjiene, kunsmatige neurale netwerke, skoenlussamevoeging, ewekansige woud en ligte gradi¨entversterkingsklassifiseerder. Hul prestasie en interpreteerbaarheid is geassesseer na opleiding en instelling van die hiperparameters vir optimale vergelyking van opleiding, toetsing en geldigverklaring van deelversamelings van die data. Prestasie-akkuraatheid is gemeet deur van die area onder die kurwe van 30 ewekansige deelversamelings wat uit die geldigverklaarde data gegenereer is, gebruik te maak. Voorts is daar van die Kruskal Wallis-toets en Dunn se multivergelykingstoets gebruik gemaak om die voorspellingsmodelle ten opsigte van akkuraatheid te klassifiseer en te bepaal of die verskille in gemidddelde akkuraatheid statisties beduidend is. Die interpreteerbaarheid van hierdie klassifiseerders is vir beide deursigtige en swartkassiemodelle uitgevoer. Om hierdie resultate te verkry, is belangrike voorverwerkingstappe ontwikkel om die kompleksiteite van plaaslike sowel as globale modelinterpretasie te verminder. Daarbenewens is Shapley se additiewe verduidelikings en plaaslike interpreteerbare model-agnostiese verduidelikings ook ingespan om die relatiewe belangrikheid van kenmerke en die impak op voorspellings te ontleed. Die eksperimente toon dat die kunsmatige neurale netwerk, ensembles en ander boomgebaseerde algoritmes in die eerste gevallestudie beduidend beter as die logistieke regressie en liniˆere diskriminantanalise presteer het. Die tweede gevallestudie het egter teenstrydige resultate opgelewer. In die tweede gevallestudie is die prestasie van die klassifiseerders relatief vergelykbaar. Dit is ’n aanduiding dat modelprestasie afhanklik is van die data waaruit die modelle saamgestel is. Hierdie twee gevallestudies toon dat die waargenome kompromie tussen akkuraatheid en verduidelikbaarheid nie altyd waar is nie. Boonop het die Shapley additiewe verduidelikings resultate opgelewer wat met die intrinsieke interpreteerbaarheidsresultate van die deursigtige metodes ooreenstem. Hierdie post-hoc interpreteerbaarheid help ons om te verstaan hoe die voorspellings gemaak word en watter faktore tot die voorspellings bygedra het. Laasgenoemde is belangrik ten einde ’n betroubare en geloofwaardige raamwerk te skep wat van swartkassiemodelle vir kredietbesluite gebruik maak. Die navorsing beklemtoon die voordele van die gebruik van alternatiewe metodes vir kredietrisikogradering; dit toon dat die prestasie aansienlik kan varieer. Dit demonstreer ook die doeltreffendheid van die Shapley additiewe verduidelikings en plaaslike interpreteerbare model-agnostiese verduidelikings in die verduideliking van voorspellings van swartkissieklassifiseerders. Dit is egter so dat dit uitdagings ten opsigte van die Shapley additiewe verduidelikings identifiseer. Die gemiddelde absolute waarde mag dalk sensitief wees vir uitskieters wat ’n impak op die belangrikheid van kenmerke kan hˆe. Daarom is verdere werk nodig om die doeltreffendheid van die berekening van Shapley se additiewe verduidelikings se waardes vir liniˆere klassifiseerders en sommige ensembles te versterk.

Diphuputso tsa morao tjena di totobaditse bohlokwa ba ho nepahala le ho hlaloswa ha mefuta ya dihlopha e sebediswang dikarolong tse fapaneng. Mefuta e mengata e fapaneng ya dihlopha le motswako wa mefuta e nnile ya ithutwa haholo ho fumana hore na ke efe e nang le tshebetso e phahameng. Diphuputso tsena di bontsha hore mehlala e atisang ho nepahala haholwanyane le yona e thata ho e utlwisisa; ho bonahala ho e na le kgwebo pakeng tsa ho nepahala le ho hlalosa. Ka lebaka leo, sena se lebisitse tlhokomelong e eketsehileng ho bohlale bo hlakileng ba maiketsetso, lefapha la dipatlisiso le amanang le ho hlalosa dikgakanyo tsa mohlala. Leha bohlale ba maiketsetso bo hlaloswang e le sebaka sa dipatlisiso se ntseng se hola setumo se ntseng se hola setjhabeng sa mahlale, ho ntse ho na le dithuto tse fokolang tse hlahlobang tshebediso ya yona kotsing ya ho se be teng ha mekitlane. Kotsi ya ho se be teng ha mokitlane e bolela tahlehelo ya ditjhelete e ka bang teng kapa kotsi e hlahiswang ke mofani wa mokoloto ha motho ya tlamang a hloleha ho fihlela mekoloto ya hae. Ho lekanya, ho fokotsa le ho laola kotsi e amanang le ho fana ka mokoloto ka potlako, bafani ba mekitlane ba sebedisa dihlopha tsa dintlha ho lekola kotsi ya bakopi ba mekitlane pele ba fana ka mokoloto. Ho feta moo, bafani ba kotsi ya mokoloto ba hlokwa ka molao ho hlalosa dikgakanyo tsa dihlopha tsa dintlha. Dihlopha tse tsebahalang tse sebediswang e le kotsi ya mokoloto di kenyelletsa ho theola maemo, hlahlobo ya kgethollo, difate tsa diqeto, meru e sa rerwang, pokello ya bootstrap, marangrang a neural, metjhini ya divector ya tshehetso le dialgorithms tse matlafatsang. Phokotso ya dintho le hlahlobo ya kgethollo di amohelwa haholo indastering ya ditjhelete hobane di sebetsa hantle ka mokgwa o utlwahalang mme ka tlhaho di ka tolokwa. Leha ho le jwalo, mekgwa ena e fana ka mokgwa wa mekgwa e meng e fanang ka ho nepahala ho ntlafetseng ha ho hlahlojwa kotsi, le hoja mekgwa ena e meng e se na tlhaloso; ha di utlwisisehe mme hangata di nkwa e le mabokose a matsho. Kgaello ena ya hlaloso e bakile ho qeaqea ho sebedisa mekgwa ena e meng ya ho fana ka mekoloto. Sepheo sa thuto ena ke ho tlosa mokwallo o boletsweng ka hodimo wa ho sebedisa mehlala ya diblackbox ka ho sebedisa mekgwa e hlakileng ya bohlale ba maiketsetso, jwalo ka dihlaloso tsa tlatsetso tsa Shapley le dihlaloso tsa sebaka sa habo bona tsa agnostic. Boithuto bona bo boetse bo hlahloba kgwebo e nepahetseng le hlaloso e nepahetseng ya dihlopha tse fapaneng ka ho theha le ho lekola mefuta e robedi ya dikarolo ho didatabase tse pedi tse fumanehang phatlalatso ya tsa mekoloto. Ho ile ha ahwa mefuta e robedi ya dikarolo, ho kenyeletswa lifate tsa liqeto, ho theoha ha thepa, hlahlobo ya kgethollo e tshwanang, metjhini ya divector tse tshehetsang, marangrang a maiketsetso a neural, aggregation ya bootstrap, moru o sa rerwang, le sehlopha se matlafatsang se bobebe. Tshebetso ya bona le hlaloso ya bona di ile tsa hlahlojwa ka mora ho kwetliswa le ho lokisa di-hyperparameters bakeng sa papiso e nepahetseng mabapi le kwetliso, diteko le ho netefatsa dikarolwana tsa data. Ho nepahala ha tshebetso ho ile ha lekanyetswa ho sebediswa sebaka se ka tlasa lekgalo ho disubsets tse 30 tse sa rerwang tse hlahisitsweng ho data ya netefatso. Ho feta moo, teko ya Kruskal Wallis le ya Dunn ya ho bapisa dintho tse ngata di ile tsa sebediswa ho beha maemo a ponelopele ka ho nepahala le ho fumana hore na diphapano tsa ho nepahala ha moelelo di bohlokwa ho latela dipalo. Hlaloso ya dihlopha tsena e ile ya etswa bakeng sa mehlala ya dibox tse bonaletsang le tse ntsho. Ho finyella diphello tsena, mehato ya bohlokwa ya ho lokisa esale pele e ile ya ntlafatswa ho fokotsa ho rarahana ha hlaloso ya mohlala ya lehae le ya lefatshe. Ntle le moo, dihlaloso tsa tlatsetso tsa Shapley le dihlaloso tsa sebaka sa sebaka sa motlolo wa agnostic di ile tsa sebediswa ho sekaseka bohlokwa bo lekanyeditsweng ba dikarolo le phello ya dikgakanyo. Diteko di bontsha hore marangrang a maiketsetso a methapo ya kutlo, di-ensembles le di-algorithms tse ding tse thehilweng sefateng di feta haholo ho theoha ha thepa le hlahlobo e fapaneng ya kgethollo thutong ya pele. Leha ho le jwalo, diphetho tse hanyetsanang di fumanwa bakeng sa thuto ya mohlala ya bobedi, kaha tshebetso ya dihlopha di batla di bapiswa. Sena se bontsha hore tshebetso ya mohlala e itshetlehile ka data eo mehlala e ahilweng ho yona. Dithuto tsena tse pedi tsa dinyewe di bontsha hore phapang pakeng tsa ho nepahala le ho hlalosa ha se kamehla e leng nnete. Ho feta moo, dihlaloso tsa tlatsetso tsa Shapley di hlahisitse ditholwana tse tsamaellanang le sephetho sa ho toloka ha mekgwa e pepeneneng. Hlaloso ena ya post-hoc e re thusa ho utlwisisa hore na dikgakanyo di etswa jwang le hore na ke dintlha dife tse tlatseditseng ho bolela esale pele. Sena ke sa bohlokwa ho theha moralo o ka tsheptjwang le o ka tsheptjwang o sebedisang mehlala ya lebokose le letsho bakeng sa diqeto tsa mokitlane. Patlisiso e totobatsa melemo ya ho sebedisa mekgwa e meng bakeng sa dintlha tsa kotsi ya mokoloto, e bontsha hore tshebetso e ka fapana haholo. E boetse e bontsa katleho ya dihlaloso tsa tlatsetso ya Shapley le dihlaloso tsa sebaka seo ho ka tolokwang tsa mohlala-agnostic ho hlalosa dikgakanyo tsa dihlopha tsa diblackbox. Leha ho le jwalo, e supa mathata a ho sebedisa dihlaloso tsa tlatsetso ya Shapley. Theko ya boleng bo felletseng e kanna ya ameha ho barekisi ba kantle, e ka amang bohlokwa ba karolo. Ka hona, mosebetsi o mong o a hlokahala ho ntlafatsa bokgoni ba ho bala boleng ba dihlaloso tsa tlatsetso tsa Shapley bakeng sa dihlopha tsa linear le diensembles tse ding.