ArticlePDF Available

Special Issue on AI Governance

February 2020

Authors:

Massachusetts Institute of Technology

This is a Special Issue on AI Governance with a Forword written by Paul Nemitz (EU Commission). It gathers 8 research papers on this topic.

Content uploaded by Eva Thelisson

Content may be subject to copyright.

IS SN 2626 -373 4

DELPHI – INTERDISCIPLINARY REVIEW

OF EMERGING TECHNOLOGIES

AIGovernance

PaulNemitz,EuropeanCommission

Technologyand(economicandpolitical)powerareenteringintoan

everclosersymbiosis.Atechnologythatknowsmoreaboutmanand

theworldthanmanknowsabouthimself,andthatisgivenevermore

decision-makingpowers,leadstoamassiveasymmetryofknowledge

andpowerintherelationshipbetweenmanandmachine.Classical

modelsofactionanddecision-makingindemocraticsocietiesare

challengedbythesedevelopments.

Thequestionoftechnicalpowerandthecontroloftechnicalpoweris

raisedinanewway .Whowilldecideinfuture?And,asShoshanaZuboff

asks,'Whodecides,whodecides?'

Whentechnologychangesthepowertoshapethingssoradically ,itis

notsurprisingthatthefundamentalintellectualandculturalconcepts

onwhichmodernsocietiesarebasedaresubjectedtoastresstest.

Delphiisapioneeringinterdisciplinaryreviewofemergingtechnologiesasseen

throughtheperspectivesofexpertsfromthefieldsofscienceandtechnology,

ethics,economics,businessandlaw.Inspiredbytheideatoencourageinclusive,

thoughtful–andsometimesunsettling–debatesonthemanyopportunitiesand

challengescreatedbytechnologicalprogress,theinternationalquarterlyreview

bringstogetherauthorswithdifferentprofessionalbackgroundsaswellas

opposingviews.ContributionstoDelphicomeincompactformatsandaccessible

languagetoguaranteealivelydialogueinvolvingboththinkersanddoers.

DELPHI – INTERDISCIPLINARY REVIEW

OF EMERGING TECHNOLOGIES

2019|VOLUME2|NUMBER4

AIGovernance

TowardsanIndexofDigital

Responsibility

EvaThelisson,Jean-HenryMorinand

JohanRochel

AIandOnlineIntermediationPlatforms

ConciliatingEconomicEfficiencyand

EthicalIssues

FrédéricMartyandThierryWarin

AISafety

ClassificationSchemasforArtificial

IntelligenceFailures

PeterJ.ScottandRomanV .Y ampolskiy

2 0 1 9

V O L U M E 2

N U M B E R 4 D E L P H I – I N T E R D I S C I P L I N A R Y R E V I E W O F E M E R G I N G T E C H N O L O G I E S

BIWEEKLY NEWS ON EU DATA PROTECTION

DATA PROTECTION INSIDER

Dara Hallinan and Diana Dimitrova,

this free newsletter provides you with

up-to-date information and insights on

Data Privacy, Data Protection and GDPR related issues.

Let’s talk competition law!

coreblog.lexxion.eu

Ocial blog of the

European Competition and Regulatory Law Review

Delphi4|2019IContents

Contents

Editorial155

EvaThelisson

F o r e w o r d

PowerinTimesofArtificialIntelligence158

PaulNemitz

A r t i c l e s

TheRighttoanExplanation161

AnInterpretationandDefense

MaëlPégny,EvaThelissonandIssamIbnouhsein

AIGovernance:DigitalResponsibilityasaBuildingBlock167

TowardsanIndexofDigitalResponsibility

EvaThelisson,Jean-HenryMorinandJohanRochel

AIEthicsforLawEnforcement179

AStudyintoRequirementsforResponsibleUseofAIattheDutchPolice

LexoZardiashvili,JordiBieger,FrancienDechesneandVirginiaDignum

ClassificationSchemasforArtificialIntelligenceFailures186

PeterJ.ScottandRomanV .Y ampolskiy

AnAGIwithTime-InconsistentPreferences200

JamesD.MillerandRomanY ampolskiy

AIforSustainableDevelopmentGoals207

NicolasMiailhe,CyrusHodes,ArohiJain,NikiIliadis,SachaAlanocaandJosephinePng

TheUseofAIbyOnlineIntermediationPlatforms217

ConciliatingEconomicEfficiencyandEthicalIssues

FrédéricMartyandThierryWarin

SustainableAISafety?226

Nadisha-MarieAliman,LeonKester,PeterWerkhovenandSoenkeZiesche

M i s c e l l A n e o u s

ImprintII

MastheadIII

Delphi4|2019 IIImprint

Publisher

LexxionVerlagsgesellschaftmbH

Güntzelstraße63·10717Berlin·Germany

Phone:+4930/814506-0·F ax:+4930/814506-22

www.lexxion.eu

Typeset

AutomatictypesettingbymetiTEC-software

me-tiGmbH,Berlin

Delphiannualsubscription*rates2019(4issues)

RegularPrice

printedversiononly417,00€

printedversion+onlineedition(incl.archive)**471,00€

onlineeditiononly(incl.archive)**410,00€

Startups

printedversiononly165,00€

printedversion+onlineedition(incl.archive)**198,00€

onlineeditiononly(incl.archive)**158,00€

Students

printedversion+onlineedition(incl.archive)***98,00€

onlineeditiononly(incl.archive)***72,00€

*PricesincludePostageandHandling.EUMemberStates:V ATwillbe

addedifapplicable.

**Multi-useronlineacessviaIPauthentication(IP-range).

***Singleuseronlineaccessviausernameandpassword.

Delphiissuppliedunderourtermsofsaleandsupply .

Copiesofourtermsandconditionsareavailableuponrequest.

LexxionVerlagsgesellschaftmbH.V ATReg.No.DE209981990.

Forfurtherinformationpleasecontact

info@lexxion.eu

Tel.:+4930/814506-0·Fax:+4930/814506-22

Contributions

arewelcomeandshouldbesubmittedaccordingtotheDelphi

authorguidelines.Anysuchcontributionisacceptedonthe

understandingthattheauthorisresponsiblefortheopinions

expressedinit.Moreinformationsavailableatdelphi.lexxion.eu.

Photocopying

Allrightsreserved.Nopartofthisjournalmaybereproducedin

anyformwhatsoever,e.g.byphotoprint,microfilm,orany

othermeans,withoutpriorpermissionofthepublisher.This

journalwascarefullyproducedinallitsparts.Nevertheless,

authors,editorsandpublisherdonotwarranttheinformation

containedthereintobefreeoferrors.Readersareadvisedto

keepinmindthatstatements,data,illustrations,procedural

detailsorotheritemsmayinadvertentlybeinaccurate.

OwnershipandshareholdingspursuanttoSection7lit.aNo.2

andSection2No.6oftheBerlinPressAct:Shareholderof

LexxionVerlagsgesellschaftmbHisDr.W olfgangAndreae,

Publisher,Berlin.

Thisjournalmaybecitedas[2019]Delphi.

ISSN(Print)2626-3734·ISSN(Online)2626-3742

e d i t o r - i n - c H i e F

CianoAydin

UniversityofTwente,theNetherlands

A s s o c i A t e e d i t o r s

FrancescaBosco

WorldEconomicForum,Switzerland

SarahFischer

DeutscheGesellschaftfürInternationale

ZusammenarbeitGmbH,Germany

FlorianKrausbeck

ambriteAG,Switzerland

AnnaLaesser

ImpactHubBerlin,Germany

MatthiasLamping

MaxPlanckInstituteforInnovationand

Competition,Germany

VinceMadai

CharitéBerlin,Germany

IdaRust

UniversityofTwente,theNetherlands

Yueh-HsuanWeng

TohokuUniversity,Japan

CeesZweistra

DelftUniversityofTechnology,theNetherlands

e d i t o r i A l B o A r d

SteffenAugsberg

Justus-Liebig-UniversitätGießen,GermanEthics

Council,Germany

WoodrowBarfield

ProfessorEmeritus,ChapelHill,USA

AubreydeGrey

SENSResearchFoundation,USA

WilliamEchikson

CentreforEuropeanPolicyStudies,Belgium

PaulNemitz

EuropeanCommission,Belgium

VuyiswaM’Cwabeni

SAP,Germany

NishantShah

ArtEZUniversityoftheArts,theNetherlands

StefanLorenzSorgner

JohnCabotUniversity,Italy

RobvandenHovenvanGenderen

VUUniversityAmsterdam,theNetherlands

StevenUmbrello

InstituteforEthicsandEmergingTechnologies,

Italy

AnnaZeiter

eBayInc.,Switzerland

e X e c u t i V e e d i t o r s

ClaraHausin

LexxionPublisher,Germany

hausin@lexxion.eu

JakobMcKernan

LexxionPublisher,Germany

mckernan@lexxion.eu

Delphi4|2019155 Editorial

Editorial:TheCentralRoleofStatesfor

BuildingaBalancedAIGovernance

Thedisruptivenatureofartificialintelligencetransformsalmostallhumanactivities

andrequiresacohesiveandsustainableAIgovernanceframeworkonaglobalscale.

Thisframeworkshouldaimatmanagingboththeopportunitiesandtherisksderived

fromthistechnologyinaproportionatemanner.Thedigitaleconomyhasincreased

theneedforatrustedecosystem,includingreinforcedregulationsandadditionalcon-

straintsforallactorsdealingwithartificialintelligenceatwhateverpartofthevalue

chain.Asaresult,publicactorshaveinitiatedaprocessthatpromotesabalancedap-

proachbeneficialforallinnovation,societyandindividuals.Itispartofaconcerted

internationalframeworkatEU,OECDandG20levelandalsoincludesisolatedprojects

likethe‘ModelAIGovernanceframework’fromSingapore.

ThegoodwillofStatesiskeyinensuringaneffectivegovernanceofartificialintelli-

gence.Thepeerreviewmechanismorreviewsbyindependentexpertscanplayacen-

tralroleintheeffectiveimplementationoftheseframeworks.DependingonhowAI

willbeused,itcanindeedeithercontributetoachievingtheUN’sSustainableDevel-

opmentGoals(SDGs)orleadtonegativesocietalexternalitieslikeharmtocitizens,

misuseofdata,themanipulationofpeople(deepfakemisuse)ormasssurveillance.

Withintheframeworkoftheirsovereigntyofpositiveresponsibilityandprotection,

Statesareresponsiblefortheimplementationofthesenon-bindingprinciplesorguide-

linesonartificialintelligenceatanationallevel.AIgovernancesafeguardsusers’in-

terestofdigitalservicesandproducts,aswellascitizens’interestsinpublicspaces.

OneofthemostrecentexamplesistheEUprojecttobanfacialrecognitiontechnolo-

giesforupto3to5years,followingtheClearviewscandal.Thisbanisfoundedon

theGeneralDataProtectionRegulationandtheright‘nottobesubjectofadecision

basedsolelyonautomatedprocessing,includingprofiling,whichproduceslegalef-

fects’.

WhiletheconstraintonStatestoimplementtheAIprinciplesandguidelinesasco-

ordinatedattheinternationallevelhasnotchangedinnature,thispressure–mainly

political–seemstohaveincreased,asisthecaseinnon-cooperativeterritoriesintax

matters.

AIGovernanceShouldConsidertheLongTermPerspectives

Inthewrestlingmatchbetweenexcessofindividualfreedomandthe‘commongood’,

thequestionastowhatconstitutes‘meaningfulgovernance’isapertinentone.TheAI

TransparencyInstituteholdstheopinionthatmeaningfulgovernanceviaabinding,di-

rectlyenforceableregulationisnecessarytoensurethesafetyofAI.Itshouldbepro-

portionate,basedonarisk-basedapproachandrespectdemocraticvaluesandprin-

DOI:10.21552/delphi/2019/4/3

Delphi4|2019 156Editorial

ciples.Integratinglongtermcriteriawithindecisionmakingprocesseswouldalsocon-

tributetomitigatingriskandensuringirreversiblestepsareavoided.

Inparticular,ex-anteandex-postmechanismsshouldbebuiltforasustainableAI

governance,mainlybasedonaccountability,transparency,gooddesign,safetyandli-

ability.Regularriskassessmentsforhighriskprojectspriortomarketdeploymentof

digitalproductsandservicesandlegallybindinginstrumentsshouldbeputinplace

tosafeguardthedemocraticuseofartificialintelligence.Softlawlikequalitylabels

andcertificationmechanismscouldcompletethisframework.

Inaglobalmarketeconomy,notboundbyterritorialbordersandmainlydrivenby

interdependenciesandshorttermindicatorslikestockexchangevalueorpollingda-

ta,longterminterestoffuturegenerationsshouldbeakeycomponentoftheAIgov-

ernanceframework.

CallforMultilateralGovernance,IncludingallPrivateandPublicStakeholders

OurfocusinthisSpecialIssueisonthegovernanceofAIandonGovernanceviaAI.

WerecommendahybridgovernancemethodologybyState(hardlaw)andbythemar-

ket(softlaw),inspiredbytheGDPRandthe108+Convention,inparticularanetwork

ofindependentcontrolauthoritiesandeffectivelegalremedies(classactions).

Inthefirstcontribution,MaelPegnyhighlightstheneedtorecognisearightofex-

planation.Hisarticleoffersapleainfavourofthetransparencyofautomateddecision-

makingasarequirementforasustainabletrustinaquantifiedanddata-drivensoci-

ety.Asheputsit,algorithmsandtrainingdatascrutinyandauditabilityarecorner-

stonesoftrustworthyAI.

WithJohanRochelandJean-HenryMorinwealsopresentaDigitalResponsibility

Indextoquantifytheresponsibilityofeconomicactors.

Inhiscontribution,LexoZardiashviliinvestigateswhyandhowtodeveloparespon-

sibleuseofAIwithinpoliceservicesandbuildagroundworkforhardregulationin

thelawenforcementenvironmentoftheNetherlands.

PeterJ.Scottanalyseshistoricalfailuresofartificialintelligenceandproposesaclas-

sificationschemeforcategorisingfuturefailures,whileJamesD.Millershowshow

time-inconsistencyincreasesthechallengeofbuildinganAGIalignedwithhumani-

ty’svalues.

NicolasMiailheproposesananalysisofspecificusecases,toachieveSustainable

DevelopmentGoals(SDG)andformulatesproposalsformulti-stakeholdercollabora-

tionandnewkindsof‘public-private-people’partnershipswhichwillreconciletech-

nical,ethical,legal,commercial,andoperationalframeworks.Headvancesnewin-

ternationalinitiatives,suchastheGlobalDataAccessFrameworkandtheAI4SDG

CenterspearheadedaspartofawiderinternationalpartnershipcalledAICommons.

FredericMarty’scontributiondealswiththegovernanceofplatforms.Heshowsthat

anincreasinguseofAIcansubstantiallyimproveperformanceinseveralareasand

improvetheleveloftrustinplatformsandadvanceduserdissatisfactiondetectiontools.

Finally,Nadisha-MarieAlimanaddressesthecomplexityofAIgovernancewithsafe-

ty-relevant,ethicalandlegalimplicationsataninternationallevel.Shealsoprovides

novelconstructiverecommendationsforanSDGinformedAIgovernanceandanAI-

Delphi4|2019157 Editorial

assistedapproachtotheSDGendeavour.AIgovernancecouldaimatasustainable

transdisciplinaryscientificapproachinstantiatedwithinacorrectivesocio-technolog-

icalfeedback-loop.Sheemphasisestheneedofastrongeducationandappropriate

institutionsfortherealisationofthispotentiallyrobustAIgovernancestrategy.

AContinuouslyImprovingAIGovernanceLegalFramework

ThisSpecialIssueisafirstcontributiontothediscussionofameaningfulAIgover-

nancelegalframework.Itproposesanoverviewofsomeofthemultifacetedaspects

ofthistopicaswellassomeconcreteproposalstopolicy-makersasawayforwardto-

wardsaneffectivehuman-centredAIgovernance.Theframeworkthatwillbeshaped,

mainlybasedonbilateralandmultilateralagreementsbetweenStates,willrequire

continuousimprovement.

EvaThelisson

GuestEditor

AIT ransparencyInstitute

Delphi4|2019 158ForewordbyPaulNemitz

PowerinTimesofArtificialIntelligence1

ThisissueofDelphiisaboutpowerinconfusingtimes,intimesofartificialintelli-

gence.Itshowswhatthenewtechnologicalpowermeansforthefundamentalfree-

domsofushumansandourdemocracy.AwisestartingpointisthatAImustnotbe

consideredinisolation,butratherinthecontextoftheconcentrationofeconomic

poweranddigitaltechnologicalpowerasitexiststoday.ThisissobecauseAIisde-

velopedanddeployedtoalargeextentbythosemajordigitalplayerscolloquially

calledtheGAFAM(Google,Apple,Facebook,Amazon,Microsoft)whichalreadyhave

astronggriponshapingtheinternetanddigitaltechnologiesasweallusethem.

AIwillbeaddedtoexistingtechnologyandbusinessmodelsandincreasetheirgrip

evenfurther,ifwedonottaketheappropriatemeasuresofregulation.Theanalysisof

AIrequiresaholisticviewofbusinessmodelsofthesedigitaltechnologiesandofthe

powertheyalreadyexerttoday.

WeneedtounderstandnotonlytheoreticalpotentialsbenefitsofAI,whichwithout

doubtexist.Wemustalsoandforemostunderstandthepowerthatiscreatedbythe

combinationofthedifferentdigitaltechnologiesinthehandsofthecorporationsthat

dominatetheinternetandthestate,andwhich,duetotherapidpaceoftechnologi-

caldevelopment,unfoldsitsowndynamicthatchallengesdemocraticprocesses.

Tounderstandthispoweranditsconsequences,aholisticviewisneededwhich

goesbeyondmarketimpacts.Wemustaskwhatitmeansforgovernmentanddemoc-

racythatnearlyallsoftwareforthethinkingandcommunicatingstate,whetheronthe

leveloftheEUorEUMemberStates,isprocuredfromMicrosoftandthatnearlyall

informationisstoredoncloudsystems.90%ofthesesystemsareownedbyUSsup-

pliers,withAmazonaccountingforalmost30%.Wemustalsobeawarethatmore

than90%ofinternetsearchesarecarriedoutonGoogle,whichinturnknowsmore

abouteveryoneindividuallythanindividualsandtheirfamilymembersthemselves.

ThefactthatanevergrowingsectionofsocietyexclusivelygetsitsnewsfromFace-

bookandY ouTubemustalsobeaconcern.WhatwilltheimpactofAI,developedand

deployedbythethusalreadypowerfulcorporationsbeonindividuals,democracy,

governmentsandmarkets?

Technologyand(economicandpolitical)powerareenteringintoanevercloser

symbiosis.Atechnologythatknowsmoreaboutmanandtheworldthanmanknows

abouthimself,andthatisgivenevermoredecision-makingpowers,leadstoamas-

siveasymmetryofknowledgeandpowerintherelationshipbetweenmanandma-

chine.

Classicalmodelsofactionanddecision-makingindemocraticsocietiesarechal-

lengedbythesedevelopments.Thequestionoftechnicalpowerandthecontrolof

technicalpowerisraisedinanewway.

DOI:10.21552/delphi/2019/4/4

1InMarch,PaulNemitzandMatthiasPfefferarepublishingPrinzipMensch–Macht,FreiheitundDemokratieimZeitalterderKünstlichen

Intelligenz(DietzVerlag)<https://prinzipmenscheu.wordpress.com/>.AnEnglisheditionisforthcominglaterthisyear.

Theauthoriswritinghereinhispersonalcapacity,notnecessarilyrepresentingthepositionoftheEuropeanCommission.

Delphi4|2019159 ForewordbyPaulNemitz

Whowilldecideinfuture?And,asShoshanaZuboffasks,‘Whodecides,whode-

cides?’2

Whentechnologychangesthepowertoshapethingssoradically,itisnotsurpris-

ingthatthefundamentalintellectualandculturalconceptsonwhichmodernsocieties

arebasedaresubjectedtoastresstest.

Wearealreadyexperiencingthesecondstageofthedigital‘revolution’withthecur-

rentupheavalsofpopulism,fakenews,foreignpropagandaandthemanipulationof

companieslikeCambridgeAnalytica,basedonFacebookdata.Andnowthatwelook

forwardtoAIandquantumcomputers,itisworthtakingalookbackatthebeginning

ofthedigitalagetounderstandandlearnwhythegreathopesoffreedomandempow-

ermentofindividualsthatwereassociatedwithithavelargelynotbeenfulfilled.On

thecontrary,wenowliveinaworldnotonlyofunsustainableclimatechangeandpol-

lution,butalsoofanincreasinglyunsustainableconcentrationofpowerandundermin-

ingofdemocracyandindividualfreedom,includinginformationalself-determination.

Inthecurrentsecondphase,wecannolongeraffordthemistakesoftheearlydays

ofdigitaltechnologyandtheglobalInternet.T echnologyandknowledgearedevelop-

ingrapidly,seeminglyexploding(somespeakofanexponentialincrease),whichshould

leadtoatransitiontoawholenewqualityinthenearfuture.

Ontheotherhand,therearethedeliberatelysloweddownprocessesofdelibera-

tivedemocracies.Sloweddown,becauseitisanimportantexperienceofhumanrule,

thatreflectiveanddiscursiveprocessesarevitalbeforeopinion-forminganddecision-

makingprocessesindemocraciesarecompleted.Aconsequenceofthisinsightisal-

sotheseparationofpowersandthetraditionalguaranteesofthefreepress.

Iftechnologycreatesfactsanddevelopsfasterthandemocraciesdecide,doesthat

meanthatinthisgameofhareandhedgehog,technologywillwinforsure?Doestech-

nologyevenhaveitsowndevelopmentallogic,whichisprovingincreasinglyimmune

todemocraticcontrol?T oday,technologyiscreatingfactsatapacethatrisksanswer-

ingthequestionofpowerinitsfavourbythisspeedalone.

Thequestionofwhowillruleinthefutureandwhowillmakethedecisionsmust

beaskedtodayinlightofdevelopmentsinAIandQuantumcomputing.Weriskbe-

ingruledbyAInotonlythroughartificiallyintelligentsystemswhichself–develop,

asidentifiedbyStuartRusselandothers,3butalsothroughtheapplicationofthese

technologiesbypowerfulcorporationstodominateourdemocraciesandfreewill,

bothindividualandcollective.

Whoeverwantstoanswerthesequestionswithafirmcommitmenttodemocracy

mustnotonlybringtherepresentativesoftechnologyanddemocracyintoanewcon-

versation.Wealsoneedaclearcommitmenttosupportthegoodfunctioningofde-

mocraticprocessbythe‘T echnicalIntelligentsia’,aclearcommitmenttotheruleby

democracyandtheruleoflawratherthantheruleoftechnologicalpowerandspeed.

Thisalsomeans:Democracymustbewillingtouseitsmostnobletool,thelaw,tothe

settherulesinthisevermoretechnologicallycolonisedworld,includingforAI.

2SoshannaZuboff,TheAgeofSurveillanceCapitalism(ProfileBooks2019)

3StuartRussel,HumanCompatible,ArtificialIntelligenceandtheProblemofControl(Viking/Penguin2019)

Delphi4|2019 160ForewordbyPaulNemitz

InhisseminalStudyof1976,4EugenKogon,afrequentpanellistwithAdornoand

Horkheimer,theprotagonistsofthecriticalFrankfurtSchool,showedthatthepolitical

attitudesofengineersinGermanyarecharacterisedbyahighdegreeofresponsibili-

tyforthepoliticalandsocietalimpactsoftheirinventions.Itwasthetimeinwhich

‘ThePhysicists’bytheswissplayrightDürrenmatthadbeenreadinschoolbyallchil-

drenontheirpathtoanentryexamforuniversity.Itisthissenseofresponsibility,

whichatthetimewasspurredbythethreatofweaponsofmassdestructionandatom-

icpower,whichtodaymustbemobilisedforfendingoffthethreatstoindividualfree-

dom,fundamentalrights,democracyandsustainabilitythroughuncheckedtechno-

logicalpoweranditsconcentrationinthehandsoffewpowerfulcompanies,atthe

topofthestockexchange.

PaulNemitz

Directorate-GeneralforJusticeandConsumers

EuropeanCommission

4EugenKogon,DieStundederIngenieure(Düsseldorf1976)

Delphi4|2019161 TheRighttoAnExplanation

TheRighttoanExplanation

AnInterpretationandDefense

MaëlPégny,EvaThelissonandIssamIbnouhsein*

TheopacityofsomerecentMachineLearning(ML)techniqueshaveraisedfundamental

questionsontheirexplainability,andpromptedthecreationofaresearchsubdomain,Ex-

plainableArtificialIntelligence(XAI).Opacitywouldbeparticularlyproblematicifthose

methodswereusedinthecontextofadministrativedecision-making,sincemostdemocrat-

iccountriesgranttotheircitizensarighttoreceiveanexplanationofthedecisionsaffect-

ingthem.Ifthisdemandforexplanationwerenotsatisfied,theveryuseofAImethodsin

suchcontextsmightbecalledintoquestion.Inthispaper,wediscussanddefendtherele-

vanceofanidealrighttoanexplanation.Itisessentialbothfortheefficiencyandaccount-

abilityofdecisionprocedures,bothforpublicadministrationandprivateentitiescontrol-

lingtheaccesstoessentialsocialgoods.Weanswerseveralobjectionsagainstthisright,

whichpretendthatitwouldbeatbestinefficientinpracticeoratworstplaytheroleofa

legalsmokescreen.Ifthoseworst-casescenariosaredefinitelyintherealmofpossibilities,

theyarebynomeansanessentialviceoftherighttoanexplanation.Thisrightshouldnot

bedismissed,butdefendedandfurtherstudiedtoincreaseitspracticalrelevance.

I.Introduction

Thereisafundamentalambiguityinthecurrentuse

ofthetermexplainabilityintheAIcommunity.On

theonehand,explainabilityor(human)interpretabil-

itydenotesafundamentalscientificproblem,the

problemofunderstandingthebehaviourofcomplex

MLsystems,whichcanleadtothedevelopmentof

sophisticatedtechniques.Ontheotherhand,the

termexplainabilityisalsousedtodenoteapedagog-

icalproblem,theproblemofexplainingtoalayau-

dience,betheypolicy-makersorordinarycitizens,

thebehaviourandoutcomesofthosesystems.Those

twochallengesarenotcompletelyindependent:of

course,acomputerscientistneedstohaveafirmsci-

entificgrasponagivenissuebeforeshetriestogive

apedagogicalexplanationtoalayaudience.They

neverthelessneedtobedistinguishedifwearetoun-

derstandtheconsiderablepedagogicalchallenges

raisedbyMLprocedures.Inthispaper,thetermsex-

planation,explainableandexplainabilitywillhave

thepedagogicalmeaningbydefault.Wewilltalk

aboutdecisionexplainabilitywhentheexplanans

willbeanoutputthatcanbedescribedasadecision.

Theneedforexplainabilityismademoreurgent

bytheuseofopaqueMLtechniquesincontexts

wherethepublichasarighttodemandanexplana-

tionofthedecisionsaffectingthem.1Theuseofsome

ofthemostsophisticatedMLtechniquesasanaidto

decision-makingmightthusbecompromisedifthose

techniquesarenotexplainable.

However,someauthorshaverecentlymadelight

oftherighttoanexplanation,dismissingitasatooth-

lesslegaltoolatbest,orasmokescreengivingtheil-

lusionofarightatworst.Althoughthosearerealpos-

sibilitiesofperversionoftheright,theyarebyno

meansanessentialvice:legalandtechnicalstrate-

giescanbeenforcedtomakeitafruitfullegaltool.

Inordertomakethispoint,wewillfirstexplain

thepoliticalstakesoftherighttoanexplanation(Sec-

tionII).Wewillthenpresenttherecentobjections

tothisright,andshowitspromotersthatwecanas-

DOI:10.21552/delphi/2019/4/5

*MaëlPégny,P ostdoctoralFellow ,ArchivesHenriPoincaré,

UniversitédeLorraine(Nancy),MembreAssociéIHPST(Paris1);

EvaThelisson,UniversityofFribourg,MITConnexionScience,for

correspondence:eva.thelisson@unifr.com;

IssamIbnouhsein,Quantmetry,P aris,forcorrespondence:iib-

nouhsein@quantmetry.com

1Thepersonsaffectedbyanadministrativedecisionmightof

coursebemoralaswellasphysicalpersons.However,explana-

tionscanonlybeprocessedbyhumanbeings.

Delphi4|2019 162TheRighttoAnExplanation

similatethemtoimprovetheconceptionandenforce-

mentofthislegaltool.W ewillfirstexamineVeale

andEdwards'objectionthattherighttoanexplana-

tionmightonlyprovideanillusionofaright(Sec-

tionIII),andFloridietal'sobjectionagainsttherel-

evanceofcounterfactualexplanations(SectionIV).

II.ThePoliticalStakesoftheRighttoan

Explanation

1.TheRelevanceofaRighttoan

ExplanationforGovernment

Transparency

Althoughmostofthispaperdealswithgenericis-

suesofAI-assisteddecision-making,itisimportant

tostresstherelevanceofbureaucraticprocedures.

Bureaucraticprocedures,publicandprivate,areone

ofthemainsurfacesofcontactbetweenthepublic

andsystematizeddecisionprocedures,andassuch

theyareahugeorganisationalandpoliticalissue.

Whenitcomestobureaucraticproceduresingov-

ernment,thereisagenerallegalidealofgovernment

transparency,whichtranslatesintoa‘righttoanex-

planation’:thecitizenshavearighttobegivenanex-

planationoftheadministrativedecisionsaffecting

them.Therighttoanexplanationwementionhere

isagenericphilosophicalideal,notitsparticularand

perhapsflawedimplementationinanygivensystem

ofpositivelaw ,egtheEUGeneralDataProtection

Regulation(GDPR)ortheFrenchLoisurla

Républiquenumérique.However,ourdiscussionof

thisidealrightwillofcoursebeinformedbythepos-

itivelegalsystemsandthechallengesraisedbytheir

application.Thisrighttoanexplanationshouldim-

poseexplainabilityasapre-conditionfortheuseof

AIsystemsinbureaucraticprocedures.

Socialcontracttheoriesclarifywhygovernment

transparencyisapre-requisitetocitizens'trustin

suchprocedures.

Governmenttransparencyofferssomesafeguards

tocitizens:initsEssayConcerningtheTrueOriginal

ExtentandEndofCivilGovernment2,JohnLockear-

guesthattheLawofNaturecommandsthatwedo

notharmothers.Inthisconception,governmentis

basedonthevoluntaryagreementsbetweencitizens

andgovernmenttocareforeachother.Astandardof

duecareobligesthegovernmenttoprotectitsciti-

zens.Ifindividualsconsenttocreateapoliticalsoci-

etyandagovernment,theyreceiveincounterpart

laws,judgestoadjudicatelaws,andtheexecutive

powernecessarytoenforcetheselaws.Therecogni-

tionofarighttoanexplanationispartofasustain-

abilitypolicyforaState,actinginatransparentand

responsiblemanner,ieinreferencetoitsdutyofcare.

Therighttoanexplanationisthuspartofthecon-

ceptofgovernmentaccountability:itwillfacilitate

thedemonstrationbytheuserofabreachofthedu-

tyofcare.

Therighttoanexplanationalsoplaysmanyroles

intheconcreteinteractionsbetweengovernmentand

citizens.Itisofcoursethebasisforappealingfroma

givendecision.Italsoplaysadecisiveroleinraising

awarenessoftheirrightsandinterestsamongciti-

zens.Itisworthbeingremindedthatformanyciti-

zenstheirinteractionwithadministrativeofficialsis

theironlysourceofinformationonthelegalenviron-

mentaffectingthem.Theexplanationofaspecificde-

cisiongivesthemanopportunitytoimprovetheirun-

derstandingofthisenvironment,andconceiveastrat-

egytodefendtheirrightsandinterests.Onemightof

coursethinkofdetrimentaleffectsofthisadaptation

tothelegalenvironment,suchastheexploitationof

legalloopholesfortaxevasion.However,oneshould

notreducethisstrategicadaptationtothosenegative

examples.Thecitizens'abilitytoadapttotheirlegal

environmentisoftenadesirablething,whichisen-

couragedbygovernmentsthroughincentives.Forex-

ample,theUSproposesfinancialandtaxincentives

toencourageitssustainabledevelopmentpoliciesand

promotetheuseofenergy-savingtechnologies.3

Havingdecisionswithoutexplanationswould

thuscutoneofthemainchannelsofcommunication

betweengovernmentandcitizens.Thedebateonex-

plainabilityshouldnotfallpreytoacrudeopposi-

tionbetweenproceduralefficiencyandrespectof

rights.Firstly ,givingexplanationscreatesopportuni-

tytocorrectmanymistakes.Secondly ,procedures

withoutexplanationswouldlosesomeoftheirmain

functionalities,especiallytheirabilitytoincrease

awarenessofrightsandchannelincentives.Admin-

istrationwithoutexplanationwouldnotbesystem-

aticallymoreefficient:itwouldbemaimed.

2JohnLocke,TwoT reatisesofGovernment(PeterLaslett,ed,CUP

1983)

3USGovernment,DatabaseofStateIncentivesforRenewablesand

Efficiency(2019)http://www.dsireusa.org/DSIREaccessed30

January2020

Delphi4|2019163 TheRighttoAnExplanation

Asaconsequence,boththedefendersoffunda-

mentalrightsandthepromotersofgovernmentef-

ficiencyshouldsupportexplainability.

2.RelevanceforPrivateEntities

Ourinsistenceongovernmenttransparencydoesnot

meanthattherighttoanexplanationisirrelevantfor

privateentities.Forthetimebeing,mostpositivelaws

allowprivatecompaniestotreattheirproceduresas

tradesecrets.However,asalgorithmssuchasscoring

systemsusedbybanks,insurancecompaniesandHR

departmentshaveaconsiderableimpactonthegen-

eralpublic,theyshouldalsofall,insomewayoran-

other,underthepurviewofanidealrighttoanexpla-

nation.Onecouldevenventuretosaythatentitiesde-

cidingwhogetsaloan,ajob,ahouseoraninsurance

playadefactogovernmentalfunction,andshouldas

suchbesubjecttosomeformofaccountability.The

peoplehavetherighttounderstandtheprocedural

environmentthatshapestheirlives,regardlessofthe

publicorprivatenatureoftheproceduralagent,or

wewouldotherwise,toquoteF .PasqualeandD.Cit-

ron'sfinewriting,pavethewayto‘anewfeudalor-

derofunaccountablereputationalintermediaries’.4

However,articulatingthelegalconsequencesofsuch

aviewpointwouldbebeyondthescopeofsuchashort

paper(formorelegalreflectionsontheaccountabili-

tyofprivatealgorithmicdecision-making5),asit

wouldentailacarefulexaminationofthetensionbe-

tweentherighttoanexplanationandIPrights.

However,someaspectsoftherighttoanexplana-

tioninpositivelawalreadyapplytoprivateentities,

andareworthyofcomment.Amazonisfacinglawsuits

forthisreason.Anautomateddecisionmakingprocess,

withoutanyhumanintervention,provideswarnings

anddecidesautomaticallytofireemployeesontheba-

sisofinputdata.6Astheproductivitymetricsarepro-

prietary ,employeescannotunderstandonwhichbasis

automateddecisionsaretaken,despitethefactthatthe

decisionshavelegaleffectsontheconcernedperson.

Notransparencyismadeontheprinciplesandvalues

encodedinthedesignofthealgorithms.Theresultof

thislawsuitmayconfirmthelegalrelevanceoftrans-

parencyandexplanation.Basedonthemodernised

Convention108,employeesareentitledtohaveknowl-

edgeonthelogicinvolvedbythealgorithmicdecision

makingprocessandhavetherighttoobject.7

InUKlaw ,therighttoanexplanationmightalso

beinstrumentalinextendingthedutyofcaretopri-

vateactors.InApril2019,theUKpublishedaWhite

PaperonOnlineHarmspresentingstatutorymea-

surestakenbytheUKtoreinforcetheaccountabili-

tyofonlineeconomicactorslikeFacebook,Google,

Snapchat,orFortnite.TheUKrecognisesadutyof

careofonlineeconomicactors.Companieswillbe

heldtoaccountfortacklingacomprehensivesetof

onlineharms,rangingfromillegalactivityandcon-

tenttobehaviourswhichareharmfulbutnotneces-

sarilyillegal.Anindependentregulatorybodywould

enforcethenewregulatoryframeworkandbenefit

fromenforcementpowers.AnannualT ransparency

Reportwillexplainwhichorganisationalmeasures

havebeentakentoavoidharmingtheusers.Inthis

perspective,theexplanationofbureaucraticproce-

duresanddecisionscanbeseenasaduedilligence

elementandasaproofofitsdutyofcare.8

Inthisperspective,theburdenofprooflieswith

thecompanyortheState.Therefore,theOnline

HarmsWhitePaperisanhistoricpaper.Itobliges

thedigitalactors,betheypublicorprivate,tobetrans-

parentandtopublishanannualreportbringingthe

proofthattheybehavedinaresponsiblemannerand

explaininghowtotheusersandtotheState.

III.VealeandEdwards'Objections:An

IllusionofARight?

Intheircomprehensiveandthoroughpapers9,11,Ed-

wardsandV ealehaveformulatedseveralarguments

4FrankPasqualeandDanielleCitron,‘TheScoredSociety–Due

ProcessforAutomatedPredictions’(2014)89WashingtonLaw

Review1

5FrankPasquale,BlackBoxSociety .TheSecretAlgorithmsthat

ControlMoneyandInformation(HarvardUniversityPress2015)

6ColinLecher.,‘HowAmazonAutomaticallyT racksandFires

WarehouseWorkersfor"Productivity"’,(TheV erge,2019)<https://

www.theverge.com/2019/4/25/18516004/amazon-warehouse

-fulfillment-centers-productivity-firing-terminations>accessed30

January2020

7Conseildel'Europe,Convention108(2019)<https://www .coe

.int/fr/web/data-protection/newsroom/-/asset_publisher/

7oll6Oj8pbV8/content/modernisation-of-convention-108>ac-

cessed30January2020

8UKGovernment,OnlineHarmWhitePaper(2019)<https://www

.gov.uk/government/consultations/online-harms-white-paper>ac-

cessed30January2020

9LilianEdwardsandMichaelVeale,‘SlavestotheAlgorithm?’

(2017)16DukeLawandT echnologyReview18

11LilianEdwardsandMichaelV eale,‘EnslavingtheAlgorithm:From

a“RighttoanExplanation”toa“RighttoBetterDecisions?”’

(2018)16IEEESecurityandPrivacy46

Delphi4|2019 164TheRighttoAnExplanation

limitingtherelevanceoftherighttoanexplanation,

anddefendedtheviewthatotherapproachesmight

bemorefruitfultopromotetherightsofgroupsand

individuals.Theyevenwarnedagainstadegeneres-

cenceofthatrightintoanillusionofaright,justas

tickingtheboxofaUserConsentFormcreatesanil-

lusionofconsent:‘thesearchforalegallyenforce-

ablerighttoanexplanationmaybeatbestdistract-

ingandatworstnurtureanewkindof“transparen-

cyfallacy”tomatchtheexistingphenomenonof

“meaninglessconsent”’10.

First,therighttoanexplanationisnotamagical,

one-fits-allsolutiontoeverydataandalgorithm-re-

latedproblem.Insteadofholdingthatrighttosuch

anunrealisticstandard,oneshouldconsideritasa

necessarybutinsufficientconditionfortheprotec-

tionofcitizens'rights,andwonderwhetherone

wouldliketoliveinasocietywhereinstitutionsare

notrequiredtoprovideexplanationsfortheirdeci-

sions.Therighttoanexplanationshouldbepartof

apackageincludingotherapproachesthatareallrel-

evanttoafairalgorithmicsociety,suchas,toname

afew ,therighttoerasure,therighttodataportabil-

ity,structuraldueprocessingovernmentagencies,

auditingbodies,certificationmechanisms,privacy

andfairnessbydesign.

Furthermore,therighttoanexplanationshould

notbeconfusedwitha‘dutytounderstand’.Individ-

ualsubjectsshouldnotbeburdenedwithanobliga-

tiontounderstandalltheproceduresaffectingthem,

asitwouldrepresentacrushingintellectualload.It

willsometimesremainabettersolutiontorelyona

governmentauditingagencyoratrustedexpert:af-

terall,thatiswhatwedowhenwehirealegalcoun-

sel.Therighttoanexplanationshouldnotberead

asadenialofthenecessityofanintellectualdivision

oflaboranddelegationofsaidlabor ,anditshould

notbeusedtoburdentheordinarycitizenwithan

intellectualworkloadnoindividualcanpossiblyface.

However,justasthecomplexitiesofpositivelegal

systemsarenoexcusetomakelawsincomprehensi-

bletoordinarycitizens,thecomplexitiesofsoftware

arenoexcusetomakethemincomprehensibletothe

peopleaffectedbythem.

Finally,weagreewithVealeandEdwardsthatthe

explanationofsomealgorithms,especiallyMLalgo-

rithms,willfaceconsiderableintellectualchallenges,

andmighthavesomefundamentallimitations.

Nevertheless,weobjecttoastrongreadingofV eale

andEdwards'conclusionthatwouldreducetheright

toanexplanationtoanintellectualdeadend,noteven

worthyofexploration.Ourpositionisnotrootedin

apriorioptimismonthechancesofsuccessofexpla-

nation.Itisrootedinthemethodologicalbeliefthat

suchchancescannotbeevaluatedbypurelyapriori

arguments,andmustbetheobjectofathoroughem-

piricalinvestigation(foranexampleofsuchanem-

piricalinvestigation12).Thereal-lifeexplainabilityof

algorithmsdependsonmanyissues,suchasopensci-

entificquestionsonhumaninterpretabilityofcom-

plexmodels,thetypesofquestionsaskedorlikelyto

beaskedbythepublic,theirrelativefrequency ,the

typeofinformationandabstractionleveladequate

toanswerthosequestions,andthetypeofdecision-

makingabilitieswithwhichwewanttoempowerthe

publicthroughthoseexplanations.Thiscomplex

webofissuesisworthbeingexplored.Evenifthe

righttoanexplanationweretofailasapracticalen-

deavor,exploringtheexplainabilityofourdecision

proceduresisafundamentalworkontheintellectu-

aldivisionoflabor,andtheflowofknowledge,or

lackthereof,inoursocialsystem,anditshouldnot

begivenupupon.However ,webelievethatthework

thathasalreadybeendoneinexplainability,suchas

Lage,Isaacandal2018and13,14warrantsthemore

optimisticconjecturethatsomerelevantdemands

forexplanationcanbeanswered.Theexplainability

ofalgorithmicproceduresandtherighttoanexpla-

nationarenotdeadends:theyarevastavenuesyet

tobeexplored.

Furthermore,therighttoanexplanationdoesnot

onlyneedanempiricalinvestigation:itdemandsa

normativereflection.Inourunderstanding,theright

toanexplanationishighlynormativeinatleasttwo

respects.Thefirstisthatitsaimisnottoproduceex-

planationsthatareaccepted,butexplanationsthat

arehonest.Thereisthusanecessarypreliminaryre-

flectiononthenatureofanhonestexplanation,as

opposedtoarhetoricalmovepermittingaquick-and-

easyacceptanceoftheprocedureanditsresults.The

secondistheimportanceofunderstandingnotonly

10ibid81

12IsaacLageetal,‘AnEvaluationoftheHumanInterpretabilityof

Explanation’(2018)NIPSConference32,Montréal,Canada

13SandraWachteretal,‘CounterfactualExplanationWithout

OpeningtheBlackBox:AutomatedDecisionsandtheGDPR’

(2017)31HarvardJournalofLawandT echnology842

14SandraWachteretal,‘ExplainingExplanationsinAI’(2019)ACM

FAT*19Conference

Delphi4|2019165 TheRighttoAnExplanation

thequestionsthatareaskedbutalsothequestions

thatshouldbeasked.Ifweonlyfocusontheques-

tionsthatarecurrentlyasked,wewillreproducethe

currentbiasesofadministrativepower,wherecer-

tainpopulationshavelittleknowledgeorunder-

standingoftheirrights,littlecontactwithadminis-

trativeinstitutions,orevenahostilerelationwith

them.Thosepopulationsareunlikelytoaskques-

tions,ortoaskthequestionsthatwouldbetrulyhelp-

fultothem.Itisthedutyoftheadministrationtobe

pro-active,toreachouttomarginalisedpopulations,

andtomakeanormativeefforttoguessthekindof

explanationsthatcouldtrulyhelpalloftheaffected

individuals.Thisrequiresadeepnormativereflec-

tiononthefunctionsoftheadministration,itsideal

relationtothepopulationitissupposedtoserve,and

theroleofexplanationinthosefunctionsandrela-

tions.

IV .Floridi'sObjectionsAgainstthe

RelevanceofCounterfactualsforthe

RighttoanExplanation

Inarecentpaper,Floridietal15stronglyobject

againstaparticularmethodtoimplementtheright

toanexplanation,iecounterfactualreasoning.Coun-

terfactualreasoningisaphilosophicalnamefor‘what

wouldhappenif...’reasoning.Answeringthoseques-

tionsisobviouslycrucialtounderstandtherole

playedbyvariousfactorsinadecision,andtoem-

powercitizenswiththeabilityofstrategicadapta-

tion.Assuch,theyareavitalpartofvirtuallyanyin-

centivepolicies:citizenscannotadapttheirbehav-

iourtoincentivesiftheydon'tunderstandwhat

wouldhappeniftheyadopttheincentivisedbehav-

iour.Ifitwouldturnoutthatitisimpossibletouse

counterfactualreasoningforcomplexalgorithmic

systems,thentherelevanceoftherighttoanexpla-

nation,ifitwouldnotbecompletelyannihilated,

wouldbedrasticallyreduced.

ItispreciselythepointmadebyFloridietalin

theirrecentpaper,whicharguesthatcounterfactual

explanationswouldprovideverylimitedinter-

pretabilitytothepublicorthetechnicalcommunity .

Counterfactualexplanationscouldactuallybeused

togeneratea‘scroll-downmenu’ofexcusesforille-

galdecisions:insteadofadmittingtheuseofanille-

galfactorx,suchasgender,theculpritcouldchoose

amongnumerous,innocuousfactorstoprovidefake

explanations,suchas‘yourloanwouldhavebeen

grantedifyouhadhigherincome’.Counterfactual

reasoningwouldthenbeinefficientatbest,andtox-

icatworst,andthesamecouldbesaidinalargepart

fortherighttoanexplanation.

Firstofall,thisisactuallyagenericproblemofle-

galexplanation.Acompetent(andshrewd)legalex-

pertisabletoprovideexplanationsfordecisionsthat

makethemlookcompliantwiththelaw,evenifthe

actualreasonsforthedecisionsareillegal.Thatis

particularlyproblematicforindividualdecisions,as

itisthenimpossibletouseordinarystatisticaltools

todemonstratethepresenceofbiases.

Inthecaseofhumandecisions,wedonothaveac-

cesstotheprivacyofanindividual'sbrain:human

decisionsarethusbynatureopaque.Moreoftenthan

notwedonothaveaccesstotheoraldeliberationsof

agivengroup,whichreducestheirtraceability.Ask-

ingforanexplanationisthusprimarilyameansof

pressure,asaremanyinterrogationtechniques:the

personsinchargeofprovidinganexplanationwill

havetocommittoastory,whichmightbedeemed

implausiblethroughfurtherquestioningorthedis-

coveryofnewevidence.Thispressureactsnotonly

asawaytodiscoverwrongdoings,butalsoasade-

terrenttoillegalbehaviour.Thepowerofsuchatool

isofcourselimited,andsomeculpritsmightgetaway

withanillegaldecisionbutagain,that'sagenericle-

galproblem.

Hasthesituationchangedwithautomateddeci-

sion-making?Itisherenecessarytodistinguishbe-

tweencases.Inthemostfavorablecasesthesituation

isalteredforthebetter,asforsomecomputersys-

temswedohaveaccesstothetruereasonsofadeci-

sion.Automateddecisionsystemscanbeprobedin

waysimpossibleforthehumanmind,andiftheright

technicalconditionsofhumaninterpretabilityand

traceabilityaremet,wemighthavedirectaccessto

thetruemotivationsofagivendecision.Moreover,

Floridietal'sobjectionseemstobefoundedonasce-

nariowhereanexplanationcouldalwaysbefreely

chosenfromascroll-downmenuofexcuses.Ifthat

isobviouslyintherealmofpossibility,theextraction

ofthereasonsforautomateddecisionsmightbe

15LucianoFloridietal,‘FromWhattoHow:AnInitialReviewof

ReviewofPubliclyAvailableAIEthicsTools,MethodsandRe-

searchtoTranslatePrinciplesintoPractices’(ArxivPreprint,2019)

<https://arxiv.org/ftp/arxiv/papers/1905/1905.06876.pdf>ac-

cessed30January2020

Delphi4|2019 166TheRighttoAnExplanation

madebyatrustedthirdparty ,possiblyasecureded-

icatedpieceofsoftware,thatwouldactuallywarrant

againstsuchmaneuvers.Inthosecases,therightto

anexplanation,andinparticulartheexplorationof

counterfactuals,wouldevolveinadirectionopposite

tothescenarioexploredintheirpaperinfavorof

moretransparencyandaccountability.Ifitcouldbe

usedtoautomatetheartofexcuse-makingandfor-

malcompliance,automateddecision-makingcould

alsobeusedtoincreasetraceabilityandwarrantthe

accesstogenuineexplanationsandcounterfactual

reasoning.Theallegeddefectofcounterfactualrea-

soningisthusjustadefectofaparticulartechnolog-

icalscenariowhichcouldbeactivelyprevented.

Someothercasesmightofcoursenotbeasfavor-

able.T echnicalconditionsoftraceabilitymightnot

alwaysbemet,evenifthelegislationshouldencour-

ageapositiveevolutiontowardstraceabilitywhen-

evertherighttoanexplanationapplies.Insomecas-

es,thedecisionmightnotbeentirelyautomated,

whichcouldaddmoredegreesoffreedomforacul-

prittomakeupafalseexplanation.Inothercases,

thewilltoexplainanactualdecisionortoexplore

counterfactualdecisionsmightfacethechallenges

raisedbyopacity,makinganindividualdecisionhard

toexplainevenfortheexpert,butthatwouldnotnec-

essarilymakeiteasiertogeneratefakecounterfactu-

als.

Furthermore,Floridietal'sargumentseemstoas-

sumethatforavastmajorityofdecisionsy,itwillbe

possibletofindanarrayoffactorsx0,...,xn-1inorder

toformulatecounterfactualargumentssuchas‘you

wouldhavehadthepositionifyouhadcollegeedu-

cation’or‘youwouldhavebeenconsideredifthis

wereaseniorposition’.Itistheavailabilityofsuch

factorsthatmaketheproductionofexcusespossible,

hidingthetrue(andpossiblyillegal)decisionfactor

xjbehinda‘just-so’story.However,weseenoreason

toassumesuchapossibilityinthevastmajorityof

cases,anditsexistenceisanotherinterestingtopic

forempiricalinvestigation.Furthermore,thosedeci-

sionfactorsmighthaveapre-determined,hierar-

chisedinfluenceonagivenoutcome.Agoodexpla-

nationwillalsoprovidetheuserwiththatinforma-

tion,makingherhardertofoolwitha‘just-so’story.

Forinstance,shemightknowthathervaluesfortwo

factorsareenoughtograntherapositivedecision,

nomatterwhattheothervaluesmightbe.Shemight

alsoknowthatanotherapplicantwithsimilarvalues

hasbeenaccepted,makingherresistanttofakeex-

planations.Itisthusimpossibleinthegeneralcase

topickanyfactortojustifyanydecisionyouwish.If

counterfactualexplanationsaremixedwithexplana-

tionsofthecausalrelevanceofeachfactor,asisthe

caseinsomecurrent‘blackbox’explanationap-

proaches(seereferencesabove),itwillbemuchhard-

ertogeneratefakeexplanationsatwill.Floridiand

al'sobjectionmistakesagainthepeculiaritiesofsome

scenarioforanessentialfeatureofcounterfactualex-

planations.Combinedwiththerelevantinformation,

counterfactualreasoningcouldbeameanstoresist

disingenuousexplanationsinsteadofameanstogen-

eratethem.

V .Conclusion

ThediscussionofAIassisteddecisionmakingand

explainabilityshouldnotgetstuckintoacrudeop-

positionbetweenrespectofrightsandefficiency .Ex-

planationsarenotadecorativefeatureofbureaucrat-

icprocedures:theyareamajorcommunicationchan-

nelbetweengovernmentanditscitizens,andfor

manyofthosecitizens,theonlychanneltheyhave.

Explanationsallowtoincreasecitizens'awarenessof

theirrightstoopennewopportunities,tocorrectmis-

takesandtoincentivisebehaviour.

Itisalltooeasytobedismissivewiththerightto

anexplanation.Therightisbynomeansasufficient

warrantofafairalgorithmicsociety ,anditcouldeas-

ilybepervertedandemptiedofitstruemeaningin

practice.However,itiswithoutadoubtanecessary

componentofafairalgorithmicsociety.Ithastobe

consideredintherightlegalandtechnicalcontextto

beassessedfairly ,andavoidmistakingthefeatures

ofaparticularscenarioforessentialfeaturesofthis

right.Dismissingtherighttoanexplanationinthe

discussionofalgorithmicfairnesswouldbeaterri-

blemistake,asitwouldleaveoutofpublicsighta

rightwhichneedscarefulinterpretation,vigorousen-

forcementanddedicatedtechnicalworktobearits

fruits.

Delphi4|2019167 TowardsanIndexofDigitalResponsibility

AIGovernance:DigitalResponsibilityasa

BuildingBlock

TowardsanIndexofDigitalResponsibility

EvaThelisson,Jean-HenryMorinandJohanRochel*

TherapiddevelopmentofAI-basedtechnologiessignificantlyimpactsalmostallhumanac-

tivitiesastheyaretiedtoalreadyexistingunderlyingsystemsandservices.Inordertomake

surethatthesetechnologiesareatleasttransparentifnotprovablybeneficialforhuman

beingsandsocietyandrepresentatrueprogress,AIgovernancewillplayakeyrole.Inthis

paper,weproposetoreflectonthenotionof‘digitalresponsibility’toaccountfortherespon-

sibilityofeconomicactors.Ourobjectiveistoprovideanoutlineofwhatdigitalresponsi-

bilityisandtoproposeaDigitalResponsibilityIndextoassesscorporatebehavior.Wear-

guethataDigitalResponsibilityIndexcanplayacentralroleinrestoringtrustinadata-

driveneconomyandcreateavirtuouscircle,contributingtoasustainablegrowth.Thisper-

spectiveispartofAIgovernancebecauseitprovidesaconcretewayofquantifyingtheim-

plementationofAIprinciplesincorporatepractice.

I.Introduction

AItechnologiesnowunderliealmostallsystemsand

servicesusedintransforminghowwelive,learn,

work,engage,vote,socialise,travel,help,etc.AItech-

nologiessufferfromalackoftransparency,which

raisesthequestionofhowliabilityriskswillbetak-

enintoconsiderationbypolicy-makers.1Inaddition,

companiesleveragingtheunderlyingdataarebuild-

ingempiresconcentratingpoweroverpeopletoa

levelneverachievedbefore.2Thisbecomesallthe

moresignificantasafewactorsdominatingthemar-

ketholdthedataofbillionsofpeople,potentiallyin-

fluencingtheirlivesandpractices.Aprominentex-

ampleisFacebook,whichinbarelytenyearsrose

frombeinganinternalcollegedatingsitetothe

biggestglobalsocialnetworkserviceeverbuiltwith

almostonethirdoftheworldpopulationbeingreg-

isteredandsharingtheirmostintimateinformation.3

Corporateresponsibilityisaconceptlargerand

olderthanthatappliedinthedigitalfield.TheUNC-

TAD(UnitedNationsConferenceonTradeandDe-

velopment)hasdiscussedformorethan25yearsthe

SocialResponsibilityofT ransnationalCorporations

withallstakeholdersinvolved(Governments,Corpo-

rations,CivilSociety,etc.).Thispaperwillfocuson

digitalresponsibilityonly.Thisisaveryimportant

conceptinevermoredigitalizedsocieties.Inboth

caseswehavesimilardimensions:politics,ethics,le-

galissues,humanrights,finance,geopolitics,etc.

AIplaysanimportantroleinthedigitalizationof

oursocieties.Thistransitionisanongoingprocess

weneedtocopewithandorganiseamongstdiffer-

entstakeholderstoachieveabalancepreventingone

stakeholderfromdominatingtheothersattheirex-

pense.Broadly ,weidentifythreemajorstakeholders:

privatecompanies(industry),publicauthorities

(state)andindividuals(society).7Implementing

DOI:10.21552/delphi/2019/4/6

*EvaThelisson,MassachusettsInstituteofTechnology,MITCon-

nexionScienceLab,Boston,USA,Co-FounderoftheAITrans-

parencyInstitute.Forcorrespondence:<evathelisson@proton-

mail.com>.

Jean-HenryMorin,InstituteofInformationServiceScience(ISS),

CentreUniversitaired’Informatique(CUI),GenevaSchoolof

SocialSciences,UniversityofGeneva,Geneva,Switzerland.For

correspondence:<jhmorin@unige.ch>.

JohanRochel,FacultyofLaw,UniversityofZürich;ethix-Labfor

InnovationEthics,Zurich,Switzerland.Forcorrespondence:

<johan.rochel@gmail.com>.

1IriaGiuffrida,‘LiabilityforAIDecision-Making:SomeLegaland

EthicalConsiderations’(2019)88F ordhamLRev439

2JamesEBessen,‘ThePolicyChallengeofArtificialIntelligence’

(2018)CPIAntitrustChronicle,BostonUnivSchoolofLaw ,Law

andEconomicsResearchPaperNo18-16

3RogerMcNamee,Zucked:W akingUptotheF acebookCatastro-

phe(HarperCollins2019)

Delphi4|2019 168TowardsanIndexofDigitalResponsibility

checksandbalanceswhichenableeconomicgrowth

andinnovation,whilefosteringtherespectofdemo-

craticandhumanvaluesandprinciplesisthecore

purposeofdesigningwhatiscalled‘AIgovernance’.4

Inthisgovernancescheme,weneedstronglegal

andregulatoryframeworks.Ex-antemechanisms

mightconcernpre-authorisationbysafetyauthori-

tiesforhighriskproductsandservices.Ex-postmech-

anismspertaintosafeguards(egprivacyandhuman

rightsimpactassessments)andeffectiveremedies

(egclassactions)toprotectindividuals’rightsen-

ablingasustainabledigitalsociety .TheEUhasgiv-

enastrongsignalinthisdirectionwiththeGeneral

DataProtectionRegulationreform(GDPR)basically

applyingtoallstatesaslongasthedataconcernsEU

datasubjects.TheEUGuidelinesonAITrustworthi-

nessareanotherexampleofthispoliticalwilltoco-

operateattheEUlevelandtoengageinasustainable

andresponsiblewayintheuseofartificialintelli-

gence.AttheOECDlevel,thePrinciplesonArtificial

IntelligencesetstandardsforAIandpromoteartifi-

cialintelligencethatisinnovativeandtrustworthy

andthatrespectshumanrightsanddemocraticval-

ues.InJune2019,theG20adoptedhuman-centered

AIPrinciplesthatdrawfromtheOECDAIPrinciple.

TheIEEEisalsodevelopingethicalstandardsmain-

lyforintelligentandautonomoussystems.

AspartofthisAIgovernancescheme,wealsoneed

toconsiderprivatecompaniesasduty-bearers.But

howshouldprivatecompaniesdefinetheirrespon-

sibility?Wearguefortheneedtointroduce‘digital

responsibility’asacriteriatoapproachtheresponsi-

bilityofcompanieswithrespecttodigitalmatters,

andAIinparticular.Wewillshowhowthisconcept

mightbeusedasanoperationalisingconceptforcor-

porateresponsibilitytocontributetoasustainable

andhuman-centereddigitalsociety.Thisapproach–

focusedontheresponsibilityofcorporations–isthe

waywethinkwewillhavethebestchancetocon-

tributetotheissueasitiscomplementarytodevel-

opingstronglegalandregulatoryframeworks.

Thiscontributionisorganisedasfollows.First,we

willpresentanddiscusstheconceptofresponsibili-

tyandtheirbearerswithaparticularfocusonthe

corporatesector.Secondly,wewillintroducedigital

responsibilityandproposeawayofclassifyingits

components.Thirdly,wewilloutlineatentativede-

signofadigitalresponsibilityindexanddescribe

howitcouldbeusedinawaytohelporganisations

bothassesswheretheystandandhelpthemprogress

onapathtowardsimprovingtheirdigitalresponsi-

bility.

II.ResponsibilityinGeneralandits

RelationtotheDigitalRealm

Inordertodefine‘digitalresponsibility’wemustfirst

understandresponsibilityingeneralbeforeaddress-

ingthespecificfeaturesofdigitalresponsibility.Ina

nutshell,theconceptofdigitalresponsibilityrepre-

sentsaspecificcategoryoftheconceptofresponsi-

bilityusedinmoralphilosophy.

Wefocusonaspecifictypeofresponsibilityhold-

er,namelycompanies.5Inthecompany-focusedlit-

erature,digitalresponsibilityisoftenunderstoodas

‘corporatedigitalresponsibility’,meaningakindof

digitalCorporateSocialResponsibility(CSR).6We

complementthisliteraturebytakingabroaderview,

highlightingthedifferentfundamentalmeaningsof

responsibilitybeforespecifyingwhatthesemean-

ingscouldmeaninthedigitalrealm.7Indoingso,

weputourapproachinthecontextoftheliterature

onresponsibleinnovation.8Thisapproachappears

4UrsGasserandVirgilioAFAlmeida,‘ ALayeredModelforAI

Governance’(2017)21IEEEInternetComputing6,58-62;Allan

Dafoe,‘AIGovernance:AResearchAgenda’(2018)Futureof

HumanityInstitute,UniversityofOxford;AlanFTWinfieldand

MarinaJirotka,‘EthicalGovernanceisEssentialtoBuildingTrust

inRoboticsandArtificialIntelligenceSystems’(2018)376Philo-

sophicalTransactionsoftheRoyalSocietyA:Mathematical,

PhysicalandEngineeringSciences2133,20180085

5Forasimilarapproach,MariarosariaTaddeoandLucianoFloridi

(eds),TheResponsibilitiesofOnlineServiceProviders(Springer

2017)

6ForaconceptualisationonCSRindigitaltime,focusingon‘new

waysofcommunicatingexistingissuesandnewresponsibilities

associatedwiththecorporateuseofdigitaltechnologies’,Geor-

gianaGrigoreetal,'NewCorporateResponsibilitiesintheDigital

Economy'inATheofilou,GGrigoreandAStancu(eds),Corpo-

rateSocialResponsibilityinthePost-FinancialCrisisEra(P algrave

Macmillan2017)43;SeealsoCThorun,'CorporateDigital

Responsibility:UnternehmerischeVerantwortunginderdigitalen

Welt'inGärtnerCandHeinrichC(eds),FallstudienzurDigitalen

Transformation(SpringerGabler2018)

7SeealsoSPelléandBReber ,'ResponsibleInnovationintheLight

ofMoralResponsibility'(2015)15JournalonChainandNetwork

Science107,111

8Onthisliterature,JobT immermansandVincentBlok,'ACritical

HermeneuticReflectionontheP aradigm-LevelAssumptions

UnderlyingResponsibleinnovation'(2018)SyntheseVincentBlok

andPieterLemmens,'TheEmergingConceptofResponsible

Innovation.ThreeReasonsWhyitisQuestionableandCallsfora

RadicalTransformationoftheConceptofInnovation'inBert-Jaap

Koops(ed),ResponsibleInnovation:Concepts,Approaches,and

Applications(Springer2015);Onthedistinctunderstandingsof

responsibilityatstakeininnovation,IbovandePoelandMartin

Sand,'VarietiesofResponsibility:TwoProblemsofResponsible

Innovation'(2018)Synthese1

Delphi4|2019169 TowardsanIndexofDigitalResponsibility

tobeconceptuallymoresolidandoffersacompre-

hensiveoverviewoftheimplicationsofresponsibil-

ityinthedigitalrealm.

1.ResponsibilityinGeneral

Responsibilityisoneofthemostfundamentalissues

addressedinmoralandpoliticalphilosophy .9The

concepthasgivenrisetoabundantliteratureindi-

versephilosophicaltraditions.Wefocusontwoim-

portantaspects:distinguishingbetweendistinct

meaningsofresponsibility ,andbrieflyaddressing

thequestionofwhichentitycouldbearresponsibil-

ity.

Inshort,themainideaofresponsibilitycouldbe

summarisedinthefollowingway:whenapersonor

anentityperformsorfailstoperformamorallysig-

nificantaction,wethinkthataparticularkindofre-

sponseiswarranted.Thisrelationbetweenwhathas

beendoneorshouldbedoneandthespecificre-

sponseiswhatwegraspwiththeideaofresponsibil-

ity.

Thisideaofresponsibilityonlymakessenseifone

essentialconditionisfulfilled.Thisconditionrefers

tothefreedomapersonoranentityneedstohave

whenperformingaspecificaction.Thiscondition

mightbefurtherspecifiedashavingthecapacityto

actdifferently.Iftherewasnootheroptionforhow

toact,thequestionofresponsibilitycannotberaised

inthesameway.Itshallbemadeclearthatthe‘free

will’questionaboutdeterminismloomslargeinthis

debate.10Wecanaddtothisfreedom-conditiona

knowledge-condition.Inspecificsituations,thequal-

ityofknowledgeavailablewhenmakingaspecific

decisionmighthaveafundamentalimpactwhenit

comestoassessingwhetheranagent’sresponsibili-

tyisengaged.Insituationswhereitwasimpossible

toknowthekindofharm(whichwouldbe)done,we

needtoadjustthekindofresponsibilityatstake. 

Ofcourse,thisquestionofhavingthecapacityto

actdifferently ,orthebenchmarkusedtoassess

whetherenoughknowledgewasavailableisultimate-

lyasocialquestion.Itdependsonthecontextin

whichthesituationoccurs.AIsystemsmakethede-

batemorecomplex.Theseelementsemphasisethat

ourunderstandingofresponsibilityisalwaysdefined

inaspecificcontext.Areaction-orabsenceofreac-

tion-inturnfurtherdefinesthiscontext.Asputby

Eshleman,‘throughthereactiveattitudes(egresent-

ment)wecommunicatetofellowmembersofthe

moralcommunityourinterpersonalexpectationsfor

areasonabledegreeofgoodwill.11

Weneedtobrieflyaddresstheissueofthetypeof

agentthatcouldbesaidtobearresponsibility.Ifwe

assumethatahumanbeingmightbearthistypeof

responsibility,thequestionismorecomplicatedfor

acompany .Broadly,twoconditionsneedtobeful-

filled.

Thefirstonepertainstotheidentificationofa

companybeingabletocarryoutanaction.Thiscon-

ditionhelpstodistinguishanorganisedcompany

fromthemereaggregationofindividuals.Itcouldbe

describedandmeasuredbyobservinginternaldeci-

sion-makingproceduresorrepresentationmecha-

nisms(executives,boardmembers,etc.).Thesecond

onepertainstotherequiredqualityofthedecision

takenbythecompany.Itmirrorsandqualifiesthe

‘freedom’conditionmentionedaboveforthecaseof

humanbeings.Thedecisionstakenbythecompany

mustshowacertaindegreeofrationality.Itmustbe

abletopursuesomethingandtotakereasonsintoac-

count.

Inthecontextofthispaper,weassumethesetwo

conditionsarefulfilledinthecaseofstandardcom-

panies.Togiveanexample,theseconditionsmight

bearguableinthecaseofadecentralisedau-

tonomousorganizationonablockchain.Itisnotclear

whetherafullydecentralisedorganizationfulfillsthe

conditionsrequiredtobeabletobearresponsibili-

ty.12

2.TwoUnderstandingsofResponsibility

Assumingthattheconditionoffreedomisfulfilled,

wecandistinguishbetweentwotypesofresponsibil-

ity:negativeandpositiveresponsibilities.13

9Foranoverview,AndrewEshleman,'MoralResponsibility'(2014)

StanfordEncyclopediaofPhilosophy

10Forreferences,MatthewT albert,'MoralResponsibility'(2019)

StanfordEncyclopediaofPhilosophy1.

11Eshleman,'MoralResponsibility'(2014)StanfordEncyclopediaof

Philosophy2.2

12Foralegalperspectiveonthisissue,DanielKraus,ThierryObrist

andOlivierHari,Blockchains,Smartcontracts,Decentralised

AutonomousOrganisationsandtheLaw(EdwardElgarPublishing

2019)

13Forasimilardistinction,PelléandReber ,'ResponsibleInnovation

intheLightofMoralResponsibility'(2015)15JournalonChain

andNetworkScience107111

Delphi4|2019 170TowardsanIndexofDigitalResponsibility

Negativeresponsibilityidentifiesactsoromis-

sionswhichshouldnotbecarriedout/shouldnot

havebeencarriedout.Inanutshell,itisaboutpre-

ventingharm.Thisunderstandingofresponsibility

islinkedtoconceptssuchasblameworthiness,liabil-

ityoraccountability .Itlegitimatesfaircompensation

inordertorepairdamageaposterioriandtopunish

theoriginatorofthenegligenceorthefault.14Itis

usedforbothindividualsandcompanies.Negative

responsibilityputsaclearfocusonthecausalrole

playedbyanagentincarryingoutanaction.Itisof-

tenusedinthecontextofidentifyingpastwrongdo-

ings.Byanticipatingfuturereactionslinkedtoaspe-

cificactoromission,negativeresponsibilitycouldbe

usedtoassessdecisionsmadeinthepresent(seebe-

lowthefurtherdistinctionbetweenprospectiveand

retrospectiveresponsibility).   

Whenreferringtothisnegativeresponsibility ,we

needtoaddressthefollowingquestions:

–Whichvalue(s)areusedtodeterminethetypesof

harmwhichgenerateresponsibility?

–Whichkindofbenchmarkisusedtoassessone’s

contributiontothisharm?

–Howshouldthetypeandextentofcompensation

bedetermined?

Thepositivedimensionofresponsibilitysharesa

maininsightofthenegativedimension:itlinksthe

behaviorofanagenttoaparticularsituationinthe

world.However,unlikethesituationdescribed

above,itfocusesonamorallyrelevantsituationthat

isnottheresultofthecompany’saction.Itputsthe

focusonthecapacityofanagent(freedomand

knowledge)topursueaspecificcourseofactionfor

thesakeofaddressingamorallyrelevantsituation.

Whenweascribepositiveresponsibilitytoanenti-

ty,wedonottellacausalstoryabouttheentity.In-

stead,wespecifywhatthisentityshouldbedoingin

theworld.AsputbySmiley ,positiveresponsibility

isusedtodistributemorallaborforfuturedeci-

sions.15

Agoodexampleofthispositiveresponsibilityis

inspiredbyPeterSinger’s‘childdrowning’thought

experiment.16Whilejogginginthepark,younotice

achilddrowninginapond.Itiscompletelysafefor

youtostepintothewaterandtakethechildoutof

it.Inthisexample,yourresponsibilitytoactappears

tobefullyengaged.Bytakingaction,youmightpre-

ventamorallydisastroussituation(thechild’sdeath),

withouttakingmajorrisksforyourself.Thesitua-

tionhereisabsolutelyclear:youaretheonlyoneable

tohelpthechild.Thedistributionofresponsibility

upondiverseagentsisnotanissuehere.Similarly,

themoralurgencyofthesituationisindisputable.A

numberofagentsmightberesponsibleforcontribut-

ingtosolvingaspecificproblem.Furthermore,the

assessmentoftheproblematstakemightbeitself

disputed.

Whenreferringtothispositiveresponsibility,we

needtoaddressthefollowingquestions:

–Whatis/arethemoralvalue(s)usedtodescribethe

morallyrelevantsituationatstake?

–Ifthemoralvalue(s)collide(s)withanotherone,

whataretheirrelativepriorities?

–Ifdiverseagentsshouldaddressthismorallyrele-

vantsituation,whatisoneagent’sfairshare?

–Whoplayedacausalroleforthecreationofthe

morallyrelevantsituationinthefirstplace?

–Doparticularpracticalelementsimpactonan

agent’sresponsibility,suchasaspecificcapacity

toaddressthesituationordetrimentalconditions

(egcosts)?

Determininganagent’spositiveresponsibility

shouldbeunderstoodasanongoingprocess.17As

forthenegativeresponsibility,thisprocessisimpact-

edbyanddoesitselfimpactthesocialcontextin

whichittakesplace.Thisismostclearlythecasewith

theassessmentofthemoralvalueofthesituationat

stakeandthedefinitionofwhatisseenasa‘prob-

lem’.

a.ProspectiveandRetrospectiveResponsibility

Anotherconceptoftheoreticalvalueisthedistinc-

tionbetweenretrospectiveresponsibilityand

prospectiveresponsibility.Bothdimensionsclearly

applytotherealmofdigitalresponsibility.Whileret-

rospectiveresponsibilityhastodowiththequestion

whichresponsibilityanactorbearsforanaction(or

omission)inthepast,prospectiveresponsibilityis

14PoelandSand,'VarietiesofResponsibility:T woProblemsof

ResponsibleInnovation'(2018)Synthese1

15MarionSmiley,'CollectiveResponsibility'(2017)StanfordEncy-

clopediaofPhilosophy§7

16PeterSinger ,PracticalEthics(CambridgeUniversityPress1980)

17PelléandReber ,'ResponsibleInnovationintheLightofMoral

Responsibility'(2015)15JournalonChainandNetworkScience

107113

Delphi4|2019171 TowardsanIndexofDigitalResponsibility

aboutactionstobetaken(ortobeomitted)inthefu-

ture.Addingthesetwodimensions,aconceptual

frameworkofthenotionofresponsibilitycanbepre-

sentedasinT able1.

b.DigitalResponsibilityinParticular

Thesetwounderstandingsofresponsibilitymightbe

appliedinthedigitalrealm,representingwhatwe

willcall‘digitalresponsibility’.Hereagain,somedis-

tinctionsareusefulinstructuringthedebate.Firstly,

responsibilitymightbeusedtoaccountforclassical

issuesofbusinessethicsinthedigitaleconomy .The

issuesarewellknownandare‘simply’foundinadif-

ferentsetting.18Inthesecases,digitalcompanies

needtoaddresssimilarcriticismsasothertypesof

companies.Thefactthattheydevelopandselldigi-

taltechnologiesdoesnotpreventthemfrombeing

caughtupinquestionablechoicesregardingforin-

stancetaxes,bribing,corruption,thebehaviorsof

theiremployees.Inthissense,digitalresponsibility ,

understoodastheresponsibilityofdigitalcompa-

nies,isthesameastheresponsibilitywhichother

typesofcompanieshave.

Secondly,andmoreimportantlyforthispaper,

newusesandactionsmadepossiblebydigitaltech-

nologiesmightcreatenewresponsibilities.Thisis-

sueisofcrucialimportanceascompaniesaretaking

noteofthegrowingawarenessoftheirclients,and

morebroadlyofthepublic.Whatisrequiredisto

structurethedifferentfieldsofthisnewdigitalre-

sponsibilityandinvestigatewhichvaluesareatstake.

III.DecomposingDigitalResponsibility

initsConstituencies

Ifthecompany’sresponsibilityisourmainfocus,we

needtodistinguishbetweenthedistincttargetcate-

goriesofactorswhichthecompanyisresponsiblefor

(its‘constituencies’).19Eachcategorymayhavedif-

ferentorsometimesevenopposinggoalsdepending

ontheethicalconflicts,potentiallyleadingtocon-

flictingoutcomes.Letusconsidertwobasicinitial

categories,definedalongtheirrelationstowardsthe

company.

Digitalservicesandproducts:thiscategorycon-

sidersallactorsdirectlyorindirectlyaffectedbythe

digitalservicesandproductsdevelopedbyacompa-

ny.

–Customers/users:Thiscategoryincludesindividu-

alswhouseadigitalservice/product.Theyhavea

directinterestindigitalresponsibility ,forexam-

pleintermsofprivacyanddataprotection,orthe

responsibledesignofsystemsandservices,etc.

–Society:Inthiscategory ,theactions/omissionsof

acompanymaynegativelyimpactsocietalandeco-

nomicqualitiesoflife(egcloudcomputingstrat-

egy).

18Grigoreetal,'NewCorporateResponsibilitiesintheDigital

Economy'inTheofilou,GrigoreandStancu(eds),Corporate

SocialResponsibilityinthePost-FinancialCrisisEra(Palgrave

Macmillan2017)49

19Forasimilarreflection,Klaus-DieterAltmeppenetal,'Öf-

fentlichkeit,VerantwortungundGemeinwohlimdigitalenZeital-

ter'(2019)64Publizistik59,67

Table1:ResponsibilityConceptualFramework.Source:Authors'elaboration

Negative

responsibility

Positive

responsibility

Retrospective

responsibility

Addressproblemscreatedbythecom-

panyinthepast

=>repairandcompensate

Addressproblemswhichthecompanyhadnotcreated,but

whichrepresentedamoralurgency

=>contributetoreducingtheharmfulconsequencesofthe

actionsofothers

Prospective

responsibility

Addressproblemswhichthecompa-

nycouldcreatenowandinthefuture

=>prevent

Addressproblemswhichthecompanydoesnotfocusitsac-

tionson,butwhichmightrepresentamoralurgency ,eg

withregardtotheenvironmentinwhichthecompanyacts

=>contributetopreventingtheharmfulconsequencesof

theactionsofothers

Delphi4|2019 172TowardsanIndexofDigitalResponsibility

–Governanceofthecompany:Thiscategoryconsid-

ersallindividualsorentitiesdirectlyaffectedby

thedigitizationofthecompany .Itmeansthatit

alsoappliestocompanieswhichdonotproduce

digitalservicesorproducts.

–Employees:Thiscategoryincludesindividualsin

asituationofemploymentwithinthecompany.

Theyhaveanindirectinterestintheoutcomeof

digitalresponsibilityintermsoftrainingand

skills,qualityofworkplace,valuealignmentwith

themanagementetc.

–Shareholders/owners:Thiscategoryincludesindi-

vidualsorentitiesholdingashareorwhoarethe

ownersofthecompany.Theirfocusmaycoveris-

suesofgovernance,technologywatch,planning

forskills,revenue,etc.

–Suppliersandsubcontractors:Thiscategoryin-

cludesallthirdpartycommercialentitiesworking

withthecompanytodeliveritsproductsandser-

vices.Dataaccessandjointliabilitywiththedata

controllerarekeyconcerns.

Wewillnowconsiderthedifferentthematicdimen-

sionsofdigitalresponsibilityandapplyeachofthese

dimensionstothedifferentconstituencies.Follow-

ingthis,thesedimensionsarematchedwiththecon-

stituencies,fromtheperspectiveofbothnegativeand

positiveresponsibility(seeFigure1).

1.SecuringAutonomyandPrivacy

Artificialintelligencesystemsshouldbedesigned,

implementedandbroughttothemarketrespecting

thevaluesofautonomyandprivacy .Thecrucialca-

pacityofhumanbeingstofreelytakedecisionsand

actaccordingtothemshouldberespected.Thisca-

pacityrequiresaprotectedpersonalsphere,asde-

tailedintheGeneralDataProtectionRegulationand

theConvention108oftheCouncilofEuropeandits

protocol.

2.RespectingEquality

Forcompaniesactiveinademocraticandliberalen-

vironment,thevalueofequalityamongindividuals

iscrucial.Thisvaluerepresentsanidealforsociety:

everyindividualshouldberecognisedasequallyim-

portanthumanbeings.Thisequalityisunderstood

hereasabasicmoralequalityamongallhumanbe-

ings.Thisraisesthekeyquestionforeverypolicyre-

gardingequality:whichfeaturesofbeinghumando

wewanttoprotectfrombeinggroundsfordiscrim-

ination?Somegroundsfordifferentialtreatmentare

legitimate,whileothersarenotconsideredassuch.

Fromthisveryfundamentalunderstandingofequal-

ity,wemightformulateandjustifymorespecificde-

mands(economicequality ,equalityofopportunities,

etc.).Algorithmictoolsraisenewchallengesregard-

ingsocialjusticeandequality,duetothekeyroleof

datasetsquality.Ifthedatasetsarenotrepresentative

ofthepopulationofacountry ,thentheproductor

servicebasedonthetrainingdatamaynotworkfor

somecategoriesofpersonsandbeharmfulforthem

(egacancerdiagnosissystemnotworkingforblack

people).20

3.DealingwithData

Digitaltechnologiesmakepossible,contributetoand

takeadvantageofthe‘dataification’oftheworld.Al-

mosteveryaspectofourindividualandcollective

livesmightbeexpressedanddocumentedintheform

ofdata.Respectingprivacyandautonomyrequires

consent-basedandproportionatedatacollection,

storage,useandtransfer.Consentshouldbein-

formedandexplicit,basedonfullinformationasto

thetype,scopeandpurposeofthedatabeingcollect-

ed.Dealingwiththesedatashouldrespecttheprin-

cipleofgoodfaithandduecare.21

4.DealingwithAlgorithms

Digitalresponsibilitydealswiththechallengesraised

bythewideuseofalgorithmsindifferentsettings.

Acrossallthesesettings,digitalresponsibilitycalls

fortheuseofalgorithmswhichrespectsafety ,auton-

omyand,moregenerally,theprincipleslinkedtothe

ruleoflaw.Thismeansinparticularthecapacityto

reconstructandexplaindecisionstakenbyalgo-

rithms.Italsomeanstheprecautionaryuseofalgo-

rithmsinsettingsespeciallysensitiveforautonomy.

20AdamConner-SimonsandRachelGordon,‘UsingAItoPredict

BreastCancerandPersonalizeCare’(2019)MITNews

21Inlatin,bonuspaterfamilias.

Delphi4|2019173 TowardsanIndexofDigitalResponsibility

Furthermore,digitalresponsibilitybearsuponthe

wideuseofalgorithminautomatingdifferenttasks

withinthecompany .EveniftheAIsystemisnota

decision-makingsystembutonlyassistshumanbe-

ingsinmakingdecisions,thereisariskthatprofes-

sionalsrelytoomuchondataanalyticswhichraises

adefactodelegationofresponsibilitytothesystem

(egradiology).

5.TakingImpactontheEnvironment

intoAccount

Digitaltechnologieshaveanimportantimpactonre-

sourcesand,morebroadly,ontheenvironment.22

Whilethisimpactmightbepositive(egdigitaltech-

nologiesreducingthegeneralconsumptionofre-

sources,suchasinsmartcityprojects),theuseof

thesetechnologiesrelyuponresourcessuchaselec-

tricity,space,watertocooldowndatacenters,butal-

soonspecificmaterialsusedintheproductionof

hardware(andtherecyclingthereof).

6.EnsuringaFairTransition

Digitaltechnologiesbringchangeswhichimpactin-

dividualsandsociety .Thisimpactmightbepositive,

butitmightalsobenegative.Companieshaveare-

sponsibilitytoidentify ,accompanythesechanges

andtoproactivelycontributetoasuccessfultransi-

tionenablingafairandsustainabledigitalsociety.

Onesolutioncouldbetoallocateasharetodatasub-

jectsasrewardforthedatacollectedandmonetised.

Thissolutionwouldaligntheinterestsofsharehold-

ersanddatasubjects,whosedataareanassetforcom-

panies.

IV .TheDigitalResponsibilityIndex:

TentativeDesignandPotentialUse

InTable2(seeAnnex),wepresentthesedifferent

dimensionsalongthedifferentconstituenciesand

thedistinctionbetweennegativeandpositivere-

sponsibility.T akentogether,theyformthecoreof

theDigitalResponsibilityIndex.Wearguethatcom-

paniesshouldusethisIndexasbothanassessment

andanimprovementmetric.Throughaself-assess-

mentapproach,companiescanassesstheirlevelof

maturityandeventuallyengageinanimprovement

processonthebasisoftheIndex.Intheabsenceof

recognisedformallegalframeworks,asoft-compli-

anceapproachmaybeappealingandcouldevenbe-

comeabusinessadvantageinanagewherecus-

tomersareincreasinglyputtingsustainabilityand

responsibilitypressureoncompaniesondigitalis-

sues.Self-assessmentmethodscombinedwithap-

proachesusingmaturitybasedmodelscanbecon-

sideredasvaluableforthecompanyinimproving

theirdigitalreadiness.Overall,theIndexhelpsbet-

terunderstandtheissuesandassesswherecompa-

niesstand,butalsotoprovidethemwithanimprove-

mentprocessshouldtheydecidetoincreasetheir

maturitylevel.

Concretely,ourpropositionistoformulateeachof

thenormativedesiderataentailedbytheIndexinthe

formofaquestionwhichthecompanyshouldaskit-

self.Eachquestionreceivesaweightaswellasacrit-

icalityindicatorwhichrepresentstheimportanceof

thequestionwithregardstotheoveralltopic.With

thesupportofanevaluationsystem,wededucea

summarydashboardpresentingthestrengthsand

weaknessesofacompanypolicy.Wededucefrom

thisdashboardsomechartspresentingthestateof

maturityofthecompanyorofaresearchprojectcom-

paredtoapredefinedthreshold(seeFigure1).Weal-

sodeduceaglobalscoring,aggregatingthedigitalre-

sponsibilityconstituentsintoapercentageindexfor

exampletoshowthematuritylevel.Thesecanbeor-

ganisedintocategoriesdependingonthefocus.This

wouldallowforagraphicalrepresentationinthe

formofaradarclearlyshowingthecoverageofthe

company.Wealsocombinethisapproachwithasim-

plematuritymodelrecommendingwhatisneeded

toengageandprogresstothenextlevelofdigitalre-

sponsibility.

Whilesuchanapproachmaybeveryusefulfor

privatecompanies,itmayalsohelpinshapingthe

debateondigitalresponsibilityoforganisationspri-

ortechnologytransfersofAI-basedsystemsinthe

market.Justaswithsocialresponsibility,ratingagen-

ciesoranalystsspecialisedininvestmentrecommen-

dationsmayusethesamecriteriatoaskthetough

questions.Thisalsohelpstoshowlargedigitalactors

thatsocietyismoreawareandsensitivetotheseis-

22MStuermer ,GAbu-TayehandTMyrach,'DigitalSustainability:

BasicConditionsforSustainableDigitalArtifactsandTheir

Ecosystems'(2017)12SustainabilityScience247

Delphi4|2019 174TowardsanIndexofDigitalResponsibility

suesinasimilarwayinwhichsocialresponsibility

madeitswayintothecorporateenvironmentover

time.

V .Conclusion

TheframingofanAIgovernanceschemeisaques-

tionfortheState(s)(inalegalandregulatoryway),

butitisalsoachallengeforprivatecompaniesact-

ingwithinthebroadernormativeframeworkofa

freemarketeconomy .Inordertoaddresstheirre-

sponsibility,wehaveoutlinedtheconceptofa‘digi-

talresponsibility’anddevelopedaDigitalResponsi-

bilityIndex.ThisIndexbringstogetherakeydistinc-

tionbetweennegativeandpositiveresponsibility,the

identificationofconstituenciesandthethematicdi-

mensionsofdigitalresponsibility.Alltogether,they

formtheIndex.ThisIndexmightbeusedasaself-

assessmenttoolforprivatecompaniesaswellasan

evaluationframeworkforlargecorporations.Aswe

alsothinkitisimportanttoallowimprovement,we

proposetheuseofmaturitymodelstohelpprogress

alongthevariouslevelstoultimatelyhelpreachalev-

elofbeingatrustworthyand‘digitallyresponsible

company’.

Figure1:DigitalResponsibilityIndexRadar

Source:AITransparencyInstitute

Delphi4|2019175 TowardsanIndexofDigitalResponsibility

Annex

Table2:EthosMatrix.Source:Authors'elaboration

Delphi4|2019 176TowardsanIndexofDigitalResponsibility

ContinuationofTable2:EthosMatrix

Delphi4|2019177 TowardsanIndexofDigitalResponsibility

ContinuationofTable2:EthosMatrix

Delphi4|2019 178TowardsanIndexofDigitalResponsibility

ContinuationofTable2:EthosMatrix

Delphi4|2019179 AIEthicsforLawEnforcement

AIEthicsforLawEnforcement

AStudyintoRequirementsforResponsibleUseofAIattheDutch

Police

LexoZardiashvili,JordiBieger,FrancienDechesneandVirginiaDignum*

Thisarticleanalysesthefindingsofempiricalresearchtoidentifypossibleconsequencesof

usingArtificialIntelligence(AI)forandbythepoliceintheNetherlands,andethicaldimen-

sionsinvolved.Welistthemorallysalientrequirementsthepoliceneedtoadheretoforen-

suringtheresponsibleuseofAIand,further,analysetheroleofsuchrequirementsforgov-

ernanceofAIinthelawenforcementdomain.Welisttheessentialresearchquestionsthat

can,ontheonehand,helptofleshoutmoredetailedcriteriafortheresponsibleuseofAIin

thepolice,andontheother,buildagroundworkforthehard-regulationinthelawenforce-

mentenvironmentoftheNetherlands.

I.Introduction

UndertheDutchPoliceLaw(Politiewet2012)thetask

oftheDutchpoliceistwo-fold:(1)toensuremain-

tainingtheruleoflawand(2)toprovideassistance

tothoseinneed.1Thepolicehaveaspecialroleinso-

cietythatinvolvesaconstitutionalrighttousevio-

lencefortheenforcementofthelaw.2Forthepolice

tofunctionandrealiseitsobjectives,societyhasto

deemthepoliceaslegitimateandtrustthatitisef-

fectiveinitstasks.3Inorderforthepolicetobetrust-

worthyintheirefficacy,theymustcontinuouslyin-

novatetoevolvewithdevelopments,stayaheadof

criminals’newstrategiesandcapabilities,andutilise

newmethodsandtechnologyforthefulfilmentof

theirtasks.4Inorderforthepolicetobetrustworthy

intheiruseofpower,thepolicemustdemonstrate

goodwillandrespectfortherightsofcivilians.The

NationalPolicegreatlyvaluesthetrustofDutchciti-

zens,whichwasmeasuredtobethehighestofany

measuredinstitutionin2017.5Itisimportanttore-

tainthistrust,alsowhenintroducingnewtechnolo-

giessuchasArtificialIntelligence(AI)thathavea

fundamentalimpactonthenatureoftheiroperations

andinteractionswithsociety.6

AIhasmanypotentiallybeneficialapplicationsin

lawenforcementincludingpredictivepolicing,auto-

matedmonitoring,(pre-)processinglargeamounts

ofdata(eg,imagerecognitionfromconfiscateddig-

italdevices,policereportsordigitizedcoldcases),

findingcase-relevantinformationtoaidinvestiga-

tionandprosecution,providingmoreuser-friendly

servicesforcivilians(egwithinteractiveformsor

chatbots),andgenerallyenhancingproductivityand

DOI:10.21552/delphi/2019/4/7

*LexoZardiashvili,LLM,PhDCandidateattheCenterforLawand

DigitalTechnologiesLeidenLawSchool,LeidenUniversity.For

correspondence:a.zardiashvili@law.leidenuniv.nl.JordiBieger,

MSc,Researcher/TeacherattheF acultyofTechnology,Policyand

Management,DelftUniversityofTechnology,andPhDCandidate

attheCenterforAnalysisandDesignofIntelligentAgents,Reyk-

javikUniversity.Forcorrespondence:<J.E.Bieger@tudelft.nl>.

FrancienDechesne,AssistantProfessorattheCenterforLawand

DigitalTechnologies,LeidenLawSchool,LeidenUniversity.For

correspondence:<f.dechesne@law.leidenuniv.nl>.Virginia

Dignum,AssociateProfessorattheFacultyofT echnology,Policy

andManagement,DelftUniversityofT echnology.Forcorrespon-

dence:<M.V.Dignum@tudelft.nl>.

1TheDutchPoliceLaw(Politiewet)2012

2JorisBoumans,‘TechnologischeEvolutiesinWetshandhavingen

Legitimiteit:TussenOptimismeenOnbehagen’(MScthesis,

TilburgUniversity2018)

3KeesvanderVijver,‘Legitimiteit,gezagenpolitie.Eenverkenning

vandehedendaagsedynamiek’inC.D.vanderVijverandF .

Vlek(eds),Delegitimiteitvandepolitieonderdruk?Beschouwin-

genovergrondslagenenontwikkelingenvanlegitimiteiten

legitimiteitstoekenning(Elsevier2006),15-133

4ibid

5CentraalBureauvoordeStatistiek,‘Meervertrouweninelkaaren

instituties’(CentraalBureauvoordeStatistiek28May2018)

<www.cbs.nl/nl-nl/nieuws/2018/22/meer-vertrouwen-in-elkaar-

en-instituties>accessed24September2019

6Boumans(n2)

Delphi4|2019 180AIEthicsforLawEnforcement

paperlessworkflows.AIcanbeusedtopromotecore

societalvaluescentraltopoliceoperations(human

dignity,freedom,equality,solidarity,democracy,and

theruleoflaw),but,ontheotherhand,valuescare-

fullyguardedinexistingoperationsandprocedures

mayalsobechallengedbytheuseofAI.

CurrentlythepoliceintheNetherlandshavebeen

usingAIinallapplicationsmentionedabove.Forex-

ample,the‘CrimeAnticipationSystem’(CAS)isan

internallydevelopedpredictive-policingtoolthat

aimstopredictcrimeswithstatisticsbasedondata

fromvarioussources.7‘Pro-Kid12-SI’(pronounced

“Pro-Kidtwelve-minus”)isarule-basedsystemfor

riskassessmentonchildrenagedbetween0-12years,

usednationwidebythepolicetopreventchildren

frombeinginvolvedinacrimeoranti-socialbehav-

iour.8TheOnlineFraudReportIntakeSystemuses

NLPtechniques,computationalargumentation(legal

informatics)andreinforcementlearningtoassist

civiliansinreportingthecrime.

Itisimpossibletoanticipatealltheeffectsofthe

useofAIinsociety ,andmorespecifically ,inthelaw

enforcementdomain.Therefore,itisessentialthat

adoptionanduseofanyapplicationbecontinuous-

lyevaluated,inorderfortheDutchpolicetoensure

policingpracticesinlinewiththevaluesacknowl-

edgedbytheDutchstateandtheEuropeanUnion.

Withthisgoalinmind,weconductedanempiri-

calstudytoidentifypossibleconsequencesofusing

AIfor,andbylawenforcementandtheethicalissues

thismayleadto.Onthebasisofthisresearch,we

haveco-writtenawhitepaperfortheDutchpolice:

‘AI&EthicsatthePolice:TowardsResponsibleUseof

ArtificialIntelligenceintheDutchPolice’(hereafter

Whitepaper).9Itdescribesthestate-of-the-artinAI,

howitcouldbenefitlawenforcement,andwhateth-

icalconcernswillneedtobeaddressedintheuseof

AIinordertosafeguardthelegitimacyofandtrust

inthenationalpolice.

II.OntheLawandEthics:TheRoleof

EthicsinLawEnforcement

Similartootherauthoritiesofthestate,thepolice

necessarilyoperatewithinaspecificlegalframe-

work.Thisframeworkincludesbutisnotlimitedto

preventingmisuseofpowers,conflictsofinterestand

discrimination,andisensuredthroughactiveac-

countabilitymeasures.Thepoliceorganisationinthe

Netherlandsiscommittedtoprotectfundamentalhu-

manrightsandtoensurerespectfortheruleoflaw.10

Thepoliceisdirectlyobligedtocomplywithdomes-

ticandinternationallegalinstrumentsthatspecify

thiscommitment,likethenationalconstitution,the

EUCharter,specificnationallegislativeacts,andthe

EUdirectivesandregulationsliketheGeneralData

ProtectionRegulation(GDPR)orLawEnforcement

Directive(LED).Theselegalrequirementsapplyto

allpoliceworkregardlessofthemeansusedandthus

includetheuseofAI.

InademocraticstatesuchastheNetherlands,com-

pliancewithholdinglawsandregulationsmustbe

seenasagivenforanyapplicationofAI.However,

theapplicationofAIraisessomechallengesthatare

not—oritisuncleariftheyare—coveredbycurrent

legalprovisions.Forexample,whilethelegislation

mightnotrequirefullopenness,theopacityofrea-

soningthatisinherenttosomeAItechniquesmight

decreasetransparencyandweakenhumanagencyin

thepolice’sdecision-making,andtherebyposea

threattothelegitimacyofandtrustinthepolice.11

Therefore,forsuchspacesleftopenbythelaw ,the

policecan,andweadvisethattheyshould,incorpo-

rate‘ethics’throughpracticalmeasurestoensurere-

sponsibleuseofAIandcontributetowardsenhanc-

ing(ratherthanlimiting)legitimacyofandtrustin

thepolice.

Incommonuse,theterm‘ethics’referstoasetof

acceptedprinciplesonwhatis(morally)rightor

wrongwithinandforacertaincommunity .The

Dutchgovernmentandthelawenforcementinpar-

ticularareexpectedtoactcoherentlyandoutofthe

7SerenaOosterlooandGerwinvanSchie,‘ThePoliticsand

Biasesofthe‘CrimeAnticipationSystem’oftheDutchPolice’,Jo

Bates,PaulD.Clough,RobertJäschkeandJahnaOtterbacher

(eds),ProceedingsoftheInternationalW orkshoponBiasin

Information,Algorithms,andSystems(CEURWorkshopProceed-

ings2018)30-41

8KarolinaLaF ors-OwczynikandGovertValkenburg,‘RiskIdenti-

ties:ConstructingActionableProblemsinDutchY outh’,I.vander

PloegandJ.Pridmore(eds),DigitizingIdentities.DoingIdentityin

aNetworkedWorld(Routledge/Taylor&FrancisGroup2016)

103-124

9FrancienDechesne,VirginiaDignum,LexoZardiashviliand

JordiBieger,‘ AIandEthicsatthePolice:TowardsResponsibleUse

ofArtificialIntelligenceattheDutchPolice’(Whitepaper,2019)

https://www.universiteitleiden.nl/binaries/content/assets/rechts-

geleerdheid/instituut-voor-metajuridica/artificiele-intelligentie-en-

ethiek-bij-de-politie/ai-and-ethics-at-the-police-towards-responsi-

ble-use-of-artificial-intelligence-at-the-dutch-police-2019..pdfac-

cessed24September2019

10Politiewet2012(n1),art2

11Dechesneandothers(n9)

Delphi4|2019181 AIEthicsforLawEnforcement

principlesoftheDutch(andlargerEuropean)com-

munity.Thisexpectationofresponsibilityextendsto

theuseofAIbytheDutchpolice.Toactresponsibly

meanstoacceptmoralintegrityandauthenticityas

idealsandtodeployreasonableefforttowardachiev-

ingthem.12FortheDutchgovernmentstrivingfor

moralintegritymeansadheringtothevaluesoffree-

dom,equality,andsolidarity.13Thesevaluesarethree

fromfourvaluestheEuropeanUnion(EU)isaiming

touphold,withdignitybeingthefourth.14Notethat,

althoughtheDutchgovernmenthasnotyetaccept-

edproposalsbyaspeciallyestablishedcommission

(establishedbytheCabinetforconstitutionalamend-

ments),toincludevalueofhumandignityexplicitly

inthetextoftheDutchConstitution,itacknowledges

dignityasafundamentalvaluethathumanrights

aimtouphold.15Humanrights,ontheotherhand,

togetherwithdemocracy,andruleoflaw,areoften

referredasthegeneralprinciplesoftheDutchcon-

stitution,16oftheEU,17andofalsolargerEuropean

community(CouncilofEurope).18

Thefourvalues(dignity ,freedom,equality ,soli-

darity)andthreeprinciples(humanrights,democra-

cy,ruleoflaw)provideaframeworkforthemoralin-

tegritythattheDutchgovernment(andinthiscase

theDutchpolice)hastocontinuouslystrivetowards.

However,societalorderasamoralmilieucannotbe

sustainedbyreferenceonlytogenerallyexpressed

values–thereforeformal(statutoryandcase)lawis

intendedtofillinthegapandoperationalisethese

abstractideals.Ontheotherhand,suchmoralmilieu

cannotbebuiltuponstricttextually-rootedrules

alone.19Forexample,inthecontextofstate-of-the-

arttechnology ,formallawfailstobetheomnibus

governancesolution:existinglegislationisnotper-

fectlysuitedtoaddressunprecedentedscopeofac-

tionsthatAIallows,andregulatoryintervention

(amongotherthings)mightpreventpotentialadvan-

tagesfrommaterialising.20

Therefore,maintainingresponsibleaction(moral

integrity)requiresaproperbalancetobestruckbe-

tween‘rule’and‘value’.Whatthismeansinthecon-

textofusingAIisthat,unprecedentedmodusoperan-

ditotheformallawdoesnotrelievetheDutchpolice

fromanobligationtostrivetowardsmoralintegrity.

WehaveevaluatedtheuseofAIbythelawenforce-

mentthroughthelensofthe(European)values(dig-

nity,freedom,equality ,solidarity)andprinciples(hu-

manrights,democracy,ruleoflaw)thattheDutch

policeaimstouphold,andidentifiedrequirements

forensuringresponsibleuseofAIwithinthepolice.21

Weprovidetheoverviewofidentifiedrequirements

inthenextchapter.

III.RequirementsfortheResponsible

UseofAIbytheDutchPolice

Weidentifiedrequirementsandrecommendations

fortheresponsibleuseofAIattheDutchpolice.They

include,(i)accountability,(ii)transparency,(iii)pri-

vacyanddataprotection,(iv)fairnessandinclusivi-

ty,(v)humanautonomyandagency,and(vi)socio-

technicalrobustnessandsafety.22Whilethesere-

quirementsaremorallysalient,theydonotoccupy

thesamelevelofhierarchyasthevaluesandtheprin-

ciplesdiscussedinthechapterII(hencethetermre-

quirements).Rathertheserequirementsareintended

toprovideguidanceonhowtoensurethatthepolice

useofAIiscoherenttothehigh-levelvalues(iedig-

nity)andtheprinciples(iedemocracy):

1.Accountability–InthecontextofusingAIforand

bythepolice,‘accountability’isarequirementthat

referstotheabilitytoholdthepolicepersonnelor

theentirepoliceorganisationanswerableand/or

12RonaldDworkin,‘JusticeforHedgehogs’(TheBelknapPress,

2011)111

13MinistryofSocialAffairsandEmployment,‘CoreValuesofDutch

Society’(ProDemos,HouseofDemocracyandConstitution,

2014)https://www.prodemos.nl/wp-content/uploads/2016/04/

KERNWAARDEN-ENGELS-S73-623800.pdfaccessed17October

2019

14CharterofFundamentalRightsoftheEuropeanUnion(TheEU

Charter),26October2012,2012/C326/02

15Jan-PeterLoof,‘HumanDignityintheNetherlands’inP aolo

Becchi,KlausMathisandJan-PeterLoof(eds.),Handbookof

HumanDignityinEurope(SpringerInternationalPublishing

2017)423

16ibid

17TheEUCharter ,Preamble;seealsoEuropeanUnion,‘Goalsand

valuesoftheEU’https://europa.eu/european-union/about-eu/eu-

in-brief_enaccessed17October2019

18CouncilofEurope,‘Values–HumanRights,Democracy,Ruleof

Law’https://www.coe.int/en/web/about-us/valuesaccessed17

October2019

19ChiefJusticeAllsopAO,‘V aluesinLaw:HowTheyInfluenceand

ShapeRulesandtheApplicationsofLaw’(HochelagaLecture,

2016)https://www.fedcourt.gov .au/digital-law-library/judges-

speeches/chief-justice-allsop/allsop-cj-20161020#_ftn3accessed

17October2019

20RonaldLeenesandothers,‘Regulatorychallengesofrobotics:

someguidelinesforaddressinglegalandethicalissues’(2017)9

(1)Law,InnovationandT echnology,7

21Dechesneandothers(n9)

22ibid

Delphi4|2019 182AIEthicsforLawEnforcement

responsible(and/orsometimesliable)foranac-

tion,choiceordecisionbyAI.Tracing(causal)re-

sponsibilitycanbecomplicatedwhenhumande-

cisionmakersare(partially)replacedoraugment-

edbyAIsystemsthatcannotthemselvescarry

moralresponsibilityorbeaccountable.Account-

abilitycanbeimprovedifthesesystemscanbere-

viewed(auditability),andifthedecisionsthatthey

makeexplainedandjustified(explainability)on

thetechnicallevel.Moreover,independentevalu-

ationsshouldbeabletoverifyandreproducethe

AI-system’sbehaviorinallsituations(repro-

ducibility).23Incaseswheretracingresponsibili-

tyisnotfeasible(andpossiblyothers),clearagree-

mentsshouldbemadeaboutwhoisaccountable

(egtheowner,operatororprogrammerofanAI

system).

2.Transparency–T ransparencyisanimportantcom-

ponentinensuringtrustandfiguringoutwhoor

whatisaccountableforpotentialproblemswith

AIsystems.Withtransparency ,wemustalways

ask1)aboutwhat,2)towhomand3)howmuch

transparencyshouldbeprovided,andofcourseto

whatend.Wecanbetransparentforexample

aboutpeople,rationale,operations,ordatain-

volvedindecision-making.Wecanbetransparent

forcourts,policeorganisation,ortothepublic.Per-

hapsgivingeveryonefullaccesstoeverythingis

notproductive,anditcanevenbedangerousifit

letsbadactorsfindwaystoexploitorcircumvent

thepolice'sAI.Transparencyisagradualmatter,

andthesameholdsforexplainabilityandinter-

pretability:wehavetotakeintoaccountthatin

thecontextofAIonlypartsofadecisionmaybe

interpretable,orthatexplanationsonlygivea

roughideaofwhathappened.

3.PrivacyandDataProtection–ThePolicehasa(le-

gal)obligationtotaketheprivacyofciviliansinto

considerationintheiroperations.Wherecivilians

canreasonablyexpecttobeprivateisbeingaltered

bythecurrenttechnologythatallowspersonalda-

tafrommanydifferentspherestobeprocessedon

anunprecedentedscale,alsoforlawenforcement

purposes(egprevention,investigation,detection

orprosecutionofcriminaloffences).AIcanin-

creasetheinformation-gatheringcapabilitiesof

thepolice,becauseofitsabilitytocombineand

analyzevastquantitiesofdatafromdifferent

sources,andthereforehasanimmenseimpacton

privacy.

4.FairnessandInclusivity–AIsystemscanplayan

importantroleintheinclusivityandaccessibility

ofpoliceservices.Forinstance,reportingofa

crimewillbeaccessibletomorepeopleifmorere-

portingmethodsareavailable,eginpersonata

policestation,byphoneandonline.Intelligent

chatbotscanmakereportingcrimesmoreaccessi-

bleforsomebyincreasingaccessibility,user

friendlinessandcatchingerrorsthatmightother-

wisebemadeonstaticforms.Oneshouldhowev-

erbecarefulthattherangeofmethodsofferedis

indeedusablebyall,includingegblindpeopleor

(computer)illiteratepeople.Ifthisisnotfeasible

forthemainmethod,alternativesshould(contin-

ueto)beprovided.AIcanalsoincreaseusability

byegaddingspeechrecognitionfunctionality

(whichcanhelppeoplewhocan’ttypetext).Itis

alsoimportanttoensurethatdecisionsinformed

byAIarefreefrombiaswhichcouldresultinthe

unfairordiscriminatorytreatmentof(groupsof)

civilians.Thisrequiresrigorousacquisition,man-

agement,developmentandevaluationofAIsys-

temsandalgorithmsaswellasthedatatheyuse.

Sincetherearedifferentconceptionsoffairness,

presentingdifferenttradeoffsdependingonthe

situation,aninformedcase-by-caseanalysisinnec-

essaryfortheresponsibleuseofAIbythepolice.

Intheend,(human)policeemployeeswillneedto

decidewhattodowiththeinformationandrec-

ommendationsprovidedbyAI,raisingquestions

aboutwhatkindofactionisappropriate:egifa

suspecthasnotdoneanythingwrongyet,butan

(imperfect)AIsystempredictsthattheymightin

thefuture,whatinterventionsbalancetherights

oftheas-of-yetinnocentcivilianwiththeneedto

preventseriouscrimes?

5.HumanAutonomyandAgency–Preservingthehu-

mansenseofagencyismainlyanindividual-lev-

elrequirementtorealisethehigh-levelvalues(i.c.

freedom)andshouldhelpwithbothjobsatisfac-

tionandtheabilitytoprovidemeaningfulhuman

control.Problemscanoccurwithdecisionsupport

systemsthatrecommendacourseofactionthat

mustthenbeevaluatedbyahumanoperator.Peo-

pleareincreasinglywillingandexpectedtodele-

23MatthewHudson,‘ ArtificialIntelligenceFacesReproducibility

Crisis’(2018),359(6377)Science725-726

Delphi4|2019183 AIEthicsforLawEnforcement

gatedecisionsandactionstomachines(egrecom-

mendersystems,searchengines,navigationsys-

tems,virtualcoachesandpersonalassistants).A

possibleconsequenceofworkingwithAIsystems

isthelossofasenseofagency:theabilitytoact

freely.Especiallywithsystemsthatareveryaccu-

rateinsomerespect,humanoperatorsmaybe

‘nudged’toactupontheoutcomeofthesystem

withoutfurthercriticaldeliberation.Thiscannot

onlyinvalidateanoperator’ssenseofagency,but

alsofailstoutilisehumancapabilitiesthatAIsys-

temstypicallystilllack,suchascommonsenserea-

soning,lookingatthebiggerpicture,andadapt-

ingtounforeseensituations.

6.(Socio-technical)RobustnessandSafety–AIsys-

temsmustbedevelopedanddeployedwithan

awarenessoftherisksandbenefitsoftheiruse,

andanassumptionthatdespiteamplepreventa-

tivemeasures,errorswilloccur.Theymustbero-

busttoerrorsand/orinconsistenciesintheirde-

sign,development,deploymentandusephases,

anddegradegracefullyinextraordinarysitua-

tions,includingadversarialinteractionswithma-

liciousactors.Errorsandmalfunctionsshouldbe

preventedasmuchaspossible,andprocesses

shouldbeinplacetocopewiththemandminimise

theirimpact.24Anexplicitandwell-formeddevel-

opmentandevaluationprocessisnecessarytoen-

sureperformance,robustness,securityandsafe-

ty.

TheDutchPoliceactstomaintainsocietalorderby

enforcingthelaw .Thelawitselfisasetofbinding

rulesthataimtoupholdthevalueswithinsociety.

Whileasetofbindingrulescanguidetheonlylim-

itedamountofpoliceactions,societalvaluesareal-

wayspresent,andtheactivitiesofthepolicearere-

sponsibleonlywhenadheringtothesevalues.IfAI

istobeutilised,thepoliceiscompelledtotakeinto

considerationmorallysalientrequirementsde-

scribedinthischapter,toensureresponsibleaction

(responsibleuseofAI).Howcantheserequirements

influencethesetofbindingruleswillbediscussed

inthenextchapter.

IV .EthicsandtheR e-evaluationofLa w

AlongsidetherapiddevelopmentofAI,thereisapro-

liferationofarticlesandpolicydocumentsaboutthe

governanceofAI,someofwhichseemtosuggest

‘ethics’asthesolutionforensuringresponsibleuse

ofAI.FewmonthsbeforewedeliveredtheWhitepa-

pertotheDutchpolice,researchersatBerkmanKlein

Centeridentifiedandpositionedthirty-twosetsof

policydocumentssidebyside,enablingcomparison

betweeneffortsfromgovernments,companies,ad-

vocacygroups,andmulti-stakeholderinitiatives.25

Thirteenofthethirty-twodocumentspresentedin

thisstudydiscusstheresponsibilityofgovernments

inthecontextofAI,aswedidinourWhitepaper.

Thesedocumentsacknowledgethattheexistingset

oflegalrulesisnotabletofullydealwiththeimpacts

ofAI,andproposeguidanceformaintainingmoral

integrityofgovernmentalactionsbyreflectingupon

ethicalvaluesandprinciples.26

However,contrarytosomeofthesegovernmen-

tal27andmostoftheprivatesector28policydocu-

ments,ourwhitepaperdidnotintendtocomeup

withthenewsetofprinciplesfortheuseofAIwith-

intheDutchpolice.Rather,welookedatthevalues

andtheprinciplesthattheDutchpolice,asthelaw

enforcementbodyoftheDutchstate,isalreadyoblig-

edtoadheretoandidentifiedwhatisrequiredtoen-

suresuchcoherence(andthereforeresponsibleuse

ofAI).Moreover,webelievethatethicalvaluesand

lawsare‘expressionsalongagradationofparticular-

ity’ratherthan‘clearlyidentifiableseparatevehi-

cles’.29Inthissense,lawconformstoethics,asthe

latterprovides‘agaugetothelaw’sflexibility’,and

its‘avenueforgrowth’.30

Inotherwords,whileethicalreflectionsprovide

advantagesasanopennorm-settingvenuesforthe

governanceofAIwithinthelawenforcement,such

considerationscoulddomorebygoingbeyondtech-

24HighLevelExpertGrouponArtificialIntelligence,‘EthicsGuide-

linesforT rustworthyAI’(High-LevelExpertGroupOnArtificial

Intelligence,TheEuropeanCommission2019)

25JessicaFjeldandothers,‘PrincipledArtificialIntelligence:AMap

ofEthicalandRights-BasedApproaches’(BerkmanKleinCenter

2019)https://ai-hr.cyber .harvard.edu/images/primp-viz.pdfac-

cessed24September2019

26seeFederalGovernmentofGermany,‘ AIStrategy’(2019)

27seeSmartDubai,‘ AIPrinciplesandEthics’(2019)

https://www.smartdubai.ae/

accessed18October2019

28seeSundarPichai,‘ AIatGoogle:OurPrinciples’(Google,2018)

https://www.blog.google/technology/ai/ai-principles/accessed18

October2019

29ChiefJusticeAllsopAO(n21)

30ibid

Delphi4|2019 184AIEthicsforLawEnforcement

nicalinterpretationsofmorallysalientrequirements

(ieaccountability ,transparency)31,andserveasthe

lensthroughwhichexistinglegalframeworks(in-

cludingframeworksregulatingtheactivitiesofthe

police)arere-evaluated,toseeifimprovementsare

possible.32Intheend,suchre-evaluationseemstobe

thelastlogicalstepastheabsenceofadequatefor-

malrules,might‘confoundlawbyadriftintoaform-

lessvoidofsentimentandintuition’ .33

V .FurtherResearchinResponsibleUse

ofAIinLawEnforcement

Asthecompletepictureoftheeffectsoftheuseof

AItechnologycannotbeanticipated,notallethical

andsocietalimpactsoftheuseofAIatthelawen-

forcementbodyoftheNetherlandscouldbecovered

intheshortstudyoftheWhitepaper.34Therefore,

ethicalevaluationoftheuseofAIbythelawenforce-

mentneedstobecontinuoustobeabletotransform

concernsintobetterlaws.Withthisgoalinmind,we

identifiedthefollowingresearchdirectionsonAIand

ethicsatthepolice,35dividedintotracksfor(1)im-

pactonhumans,(2)organisationalembedding,and

(3)technicalwork:

1.ImpactsonHumans:

a.ImpactsonHumanDignity–Humandignityis

theinviolablevalueuponwhichthehuman

rightsframeworkrests.Itillustratesthefunda-

mentalbeliefintheintrinsicworthofahuman

being,protectinghis/herautonomyandself-de-

termination.Beliefinhumandignitycanbeun-

derstoodastheraisond'êtreforthelawthepo-

liceaimstoenforce.

b.PublicTrust–Publicperceptionofthelegitima-

cyofthepoliceandsubsequenttrustisasim-

portantasthelegalframeworkinwhichthepo-

liceoperate.Whileautomationandprediction

tosomeextentincreaseefficacyofthepolice,

thestudycouldexploreifsuchincreaseinpo-

tencyisdesirablefromthesocietalperspective.

2.ImpactsonthePoliceOrganisation:

a.EthicsGuidelinesandOversight–Thepolice

doesnotoperateinisolation,andtheuseofAI

takesplaceacrosstheentirejudicialchain:OM,

localgovernment,theMinistryofJusticeand

Security,judiciary.ResponsibleuseofAIwith-

intheDutchpoliceideallyfollowsfromaro-

bustethicsframeworkfortheentirechain.Such

aframeworkcanestablishcriteriatofollow

throughouttheAIdevelopmentandapplica-

tioncycle.

b.ImpactsonPolicePersonnel–AIcanbeusedto

supportthepoliceorganisationinachievingits

goalsofefficiency ,traceability,uniformityand

integrity.However,thechangeofoperations

maycomewithdisplacementofemployeesand

changingroles.Researchisrequiredtoensure

thatworkerswithnon-traditionalskillsetsfitin-

tothepoliceorganisationinawaythatempow-

erspolicepersonnel.

3.TechnicalAspects

a.ExplainableAI–Theaforementionedoversight

canonlybeadequateandmeaningfulifauto-

mateddecisionscanbeexplainedandjustified

onthetechnicallevel.

b.Justifiable/VerifiableAI–Justificationprovides

thereasonsbehindtheresultsandthechoices

forparticularapproaches.Mathematicaltools

forformalverificationmakeAIsystemsthem-

selvesandtheirdecisionsreviewable.

Furtherresearchisessentialsothatthepolicecon-

tinuestorealisetheirdualgoalsofincreasing(a)ef-

ficacyandefficiency ,and(b)trustandtrustworthi-

ness(toboostpublictrustandtheperceptionofthe

legitimacyofthepolice).Theresearchintheareas

describedabovewillhelpusre-evaluatetheformal

rulesregardinglawenforcement,andalsomakeso-

cietalrequirementstransparenttoboththepolice

31CorinneCath,‘Governingartificialintelligence:ethical,legaland

technicalopportunitiesandchallenges’(2018),376(2133)Philo-

sophicalTransactionsoftheRoyalSocietyAMathematical,

PhysicalandEngineeringSciences

32LucianoFloridi,andothers,‘AI4People—AnEthicalFramework

foraGoodAISociety:Opportunities,Risks,Principles,and

Recommendations’(2018),28(4)MindsandMachines689–707

33ChiefJusticeAllsopAO(n21)

34Whitepaper(n13)

35FrancienDechesne,VirginiaDignum,LexoZardiashviliand

JordiBieger,‘Long-T ermResearchStrategyforAIandEthicsatthe

Police’(Report2019)https://www.universiteitleiden.nl/bina-

ries/content/assets/rechtsgeleerdheid/instituut-voor-metajuridi-

ca/artificiele-intelligentie-en-ethiek-bij-de-politie/research-strate-

gy-ai-ethics-dutch-police-final.pdfaccessed24September2019

Delphi4|2019185 AIEthicsforLawEnforcement

andthepublicandultimatelyenablecodificationin

thelegalframeworks.

VI.Conclusions

Thisarticlehasanalysedtheroleofthemorally

salientrequirementsforgovernanceofAI,thatwere

foundinanempiricalstudywithinthelawenforce-

mentdomain–inparticular:attheDutchPolice.W e

havearguedthatthereareinstances,wheretheneed

forsoftregulatoryinstrumentarises,andwehavede-

scribedhowethicalconsiderationscanhelpfulfilthis

need.Ouranalysissuggeststhattheresponsibleuse

ofAIattheDutchpolicerequiresprimarilythefol-

lowingrequirements:accountability ,transparency,

privacy,fairnessandinclusivity ,humanautonomy

andagencyandsocio-technicalrobustnessandsafe-

ty.

Furthermore,weexploredtheroleoftheserequire-

mentsinafuturere-evaluationoftheformalbind-

inginstruments.Finally ,weidentifiedtheareas

wherefurtherresearchisadvisableforensuringthe

responsibleuseofAIattheDutchpolice.Ontheone

hand,suchresearchcanhelpfleshoutmoredetailed

criteriaforthepoliceonhowtoadheretothevalues

andprinciplesoftheDutchstate.Ontheother,itcan

buildagroundworkforthehard-regulationforthe

useofAIinthelawenforcementecosystemofthe

Netherlands.

Delphi4|2019 186ClassificationSchemasforAIFailures

ClassificationSchemasforArtificial

IntelligenceFailures

PeterJ.ScottandRomanV.Yampolskiy*

Inthispaperweexaminehistoricalfailuresofartificialintelligence(AI)andproposeaclas-

sificationschemeforcategorisingfuturefailures.Bydoingsowehopethat(a)therespons-

estofuturefailurescanbeimprovedthroughapplyingasystematicclassificationthatcan

beusedtosimplifythechoiceofresponseand(b)futurefailurescanbereducedthrough

augmentingdevelopmentlifecycleswithtargetedriskassessments.

I.Introduction

Artificialintelligence(AI)isestimatedtohavea$4-6

trillionmarketvalue1andemploy22,000PhDre-

searchers.2Itisestimatedtocreate133millionnew

rolesby2022buttodisplace75millionjobsinthe

sameperiod.3Projectionsfortheeventualimpactof

AIonhumanityrangefromutopia4toextinction.5

InmanyrespectsAIdevelopmentoutpacestheef-

fortsofprognosticatorstopredictitsprogressandis

inherentlyunpredictable.6

Y etallAId evelop mentis(sof ar)u ndertak enb yhu-

mans,andthefieldofsoftwaredevelopmentisnote-

worthyforunreliabilityofdeliveringonpromises:over

two-thirdsofcompaniesaremorelikelythannottofail

intheirITprojects.7Asmucheffortashasbeenputin-

tothedisciplineofsoftwaresafety ,itstillhasfartogo.

Againstthisbackgroundoframpantfailureswe

mustevaluatethefutureofatechnologythatcould

evolvetohuman-likecapabilities,usuallyknownas

artificialgeneralintelligence(AGI).Thespectacular

advancesincomputingmadepossiblebytheexpo-

nentialhardwareimprovementsduetoMoore’sLaw8

balancedagainsttheunknownrequiredbreak-

throughsinmachinecognitionmakepredictionsof

AGInotoriouslycontentious.Estimatesofhowlong

wehavebeforeAGIwillbedevelopedrangeoversuch

widelyvaryingtimelines9thatresearchershavetak-

entometa-analysisofthepredictionsthroughcorre-

lationagainstmetricssuchascodingexperienceof

thepredictors.10

Lesscontentiousistheassertionthatthedevelop-

mentofAGIwillinevitablyleadtothedevelopment

ofASI:artificialsuperintelligence,anAImanytimes

moreintelligentthanthesmartesthuman,ifonlyby

virtueofbeingabletothinkmanytimesfasterthan

ahuman.11Analysisoftheapproachofconfininga

superintelligencehasconcludedthiswouldbediffi-

DOI:10.21552/delphi/2019/4/8

*PeterJ.Scott,NextWaveInstitute,USA.F orcorrespondence:

<peter@humancusp.com>;RomanV.Y ampolskiy,Universityof

Louisville,Kentucky,USA,<roman.yampolskiy@louisville.edu>

1McKinseyGlobalInstitute,‘NotesfromtheAIFrontier’<https://

www.mckinsey.com/~/media/mckinsey/featured%20insights/

artificial%20intelligence/notes%20from%20the%20ai%20frontier

%20applications%20and%20value%20of%20deep%20learning/

notes-from-the-ai-frontier-insights-from-hundreds-of-use-cases

-discussion-paper.ashx>accessed1November2019

2JeremyKahn,‘JustHowShallowistheArtificialIntelligence

TalentPool?’(Bloomberg,7February2018)<https://www

.bloomberg.com/news/articles/2018-02-07/just-how-shallow-is

-the-artificial-intelligence-talent-pool>accessed1November

2019

3WorldEconomicForum,‘TheFutureofJobsReport’<http://

www3.weforum.org/docs/WEF_Future_of_Jobs_2018.pdf>ac-

cessed1November2019

4RaymondKurzweil,TheSingularityIsNear:WhenHumans

TranscendBiology(Viking2005)487

5NickBostrom,Superintelligence:Paths,Dangers,Strategies

(OxfordUniversityPress2005)

6RomanYampolskiy,‘UnpredictabilityofAI’(2019)arX-

iv:1905.13053v1[cs.AI]

7KeithEllis,‘TheImpactOfBusinessRequirementsOnTheSuccess

OfTechnologyProjects’(BATimes,15February2008)<https://www

.batimes.com/articles/the-impact-of-business-requirements-on-the

-success-of-technology-projects.html>accessed1November2019

8ChrisMack,‘FiftyYearsofMoore'sLaw’(2011)24IEEETransac-

tionsonSemiconductorManufacturing202-207

9KajSotalaandRomanY ampolskiy,'Corrigendum:Responsesto

CatastrophicAGIRisk:ASurvey'(2015)90Phys.Scr.018001

10BrianT omasik,‘PredictionsofA GITakeoffSpeedvs.YearsWorked

inCommercialSoftware’inEssaysonReducingSuffering(2014)

<https://reducing-suffering.org/predictions-agi-takeoff-speed-vs

-years-worked-commercial-software/>accessed1November2019

11VernorVinge,‘TheComingTechnologicalSingularity:Howto

SurviveinthePost-humanEra’inNationalAeronauticsandSpace

Administration(eds),Vision21:InterdisciplinaryScienceand

EngineeringintheEraofCyberspace(1993)

Delphi4|2019187 ClassificationSchemasforAIFailures

cult12ifnotimpossible.13Manyoftheproblemspre-

sentedbyasuperintelligenceresembleexercisesin

internationaldiplomacymorethancomputersoft-

warechallenges;forinstance,thevaluealignment

problem14(describedthereinasthe‘valueloading

problem’)ofaligningAIvalueswithhumans’.

II.Definitions

Wepresentsomeoperationaldefinitionsofterms

usedinthispaper.

Artificialintelligenceisashiftingtermwhosede-

finitionisfrequentlydebated.Itsscopechangesde-

pendingupontheera:duringan‘AIWinter’15many

fewervendorsarewillingtoidentifytheirproducts

asAIthanduringthecurrentperiodofmyriadAI

technologiescloggingthe‘peakofinflatedexpecta-

tions’intheGartnerHypeCycle.16

Failureisdefinedas‘thenonperformanceorin-

abilityofthesystemorcomponenttoperformitsex-

pectedfunctionforaspecifiedtimeunderspecified

environmentalconditions.’17Thisdefinitionoffail-

ureasaneventdistinguishesitfromanerror,which

isastaticcondition(orstate)thatmayleadtoafail-

ure.

Cybersecurityhasbeendefinedas‘theorganisa-

tionandcollectionofresources,processes,andstruc-

turesusedtoprotectcyberspaceandcyberspace-en-

abledsystemsfromoccurrencesthatmisaligndeju-

refromdefactopropertyrights.’18AISafetyhasbeen

definedasanextremesubsetofcybersecurity:‘The

goalofcybersecurityistoreducethenumberofsuc-

cessfulattacksonthesystem;thegoalofAISafety

istomakesurezeroattackssucceedinbypassingthe

safetymechanisms.’19

Intelligencedefinitionsconvergetowardtheidea

thatit‘(…)measuresanagent’sabilitytoachieve

goalsinawiderangeofenvironments.’20W edonot

presentthisdefinitionwithanyintentionofdefin-

ingAIbyapplyingthe‘artificial’modifiertothis

one.Rather,thisdefinitionwillbeusedtojudge

whetherasoftwarefailureisinstructiveintheex-

tenttowhichitwasapplying(accidentallyorinten-

tionally)intelligenceineventhenarrowestsense,

sincesuchapplicationcouldextendtoamorepow-

erfulAI.

III.AIFailureClassification

WewilldescribeatagschemaforclassifyingAIfail-

ures.Itispreciselybecauseofthevolatiledefinition

ofAIthatwemustcastawidenetinwhatweusefor

examplesofAIfailures,becausewhatisclassifiedas

AItodaywilllikelybegivenalessglamoroustitle

(like‘machinevision’)onceitbecomescommon-

place.AsAIpioneerJohnMcCarthyputit,‘Assoon

asitworks,noonecallsitAIanymore.’21Where

someofourexamples,therefore,mayappeartobe

indistinguishablefromfailuresofsoftwarethathas

noparticularclaimtothelabelofartificialintelli-

gence,theyareincludedbecausetheyareclose

enoughtoAIonthesoftwarespectrumastobein-

dicativeofpotentialfailuremodesofAI.

1.HistoricalClassifications

Neumann22describedaclassificationforcomputer

riskfactors(seeT able1).

Wefindthislisttoobroadinsomerespectsand

toonarrowinotherstobeusefulforourpurposes.

Hardwarefactorsareoutsidethescopeofthispaper

12RomanY ampolskiy,‘LeakproofingtheSingularity:TheArtificial

IntelligenceConfinementProblem’(2012)19JournalofCon-

sciousnessStudies1-2

13EliezerY udkowsky,‘RetrievedfromTheAI-BoxExperiment’

(2002)<http://yudkowsky.net/singularity/aibox>accessed20

January2020

14(n5)

15DanielCrevier ,AI:TheT umultuousSearchforArtificialIntelli-

gence(BasicBooks1993)

16CIODive,‘GartnerServesup2018HypeCyclewithaHeavy

SideofAI’<https://www .ciodive.com/news/gartner-serves-up

-2018-hype-cycle-with-a-heavy-side-of-ai/530385/>accessed1

November2019

17NancyLeveson,Safeware:SystemSafetyandComputers(Addi-

son-Wesley1995)

18DanCraigen,NadiaDiakun-Thibault,andRandyPurse,‘Defining

Cybersecurity’(2012)T echnologyInnovationManagement

Review13-21

19RomanY ampolskiy,‘ ArtificialIntelligenceSafetyandCybersecuri-

ty:aT imelineofAIFailures’(2016)arXiv:1610.07997v1[cs.AI]

20ShaneLeggandMarcusHutter,‘ ACollectionofDefinitionsof

Intelligence’(2007)IDSIA-0707TechnicalReport

21BertrandMeyer,‘JohnMcCarthy’(Communicationsofthe

ACM,28October2011)<https://cacm.acm.org/blogs/blog

-cacm/138907-john-mccarthy/fulltext>accessed20January

2020

22PeterNeumann,Computer-RelatedRisks(Addison-Wesley1994)

Delphi4|2019 188ClassificationSchemasforAIFailures

(andareincreasinglyirrelevantassoftwarebecomes

moreplatform-independentandmobile);software

factorsneedgreaterelaboration.NeumannandPark-

er23listedclassesofcomputermisusetechniques(see

Figure1).

Despitethetreestructure,thisrepresentsasystem

ofdescriptorsratherthanataxonomyinthatagiv-

enmisusemayinvolvemultipletechniqueswithin

severalclasses.Theleftwardbranchesallinvolvemis-

use;therightwardbranchesrepresentpotentiallyac-

ceptableuse–untilaleftwardbranchistaken.How-

ever,theterm‘misuse’impliesdeliberateagencyand

23PeterNeumannandDonaldP arker,‘ ASummaryofComputer

MisuseTechniques’(1989)12thNationalComputerSecurity

Conference396-407

Table1:ComputerRiskFactorsSourcesandExamples.Source:

P.G.Neumann

Figure1:ClassesofComputerMisuse

Source:NeumannandParker

Note:Theleftwardbranchesallinvolvemisuse;therightwardbranchesrepresent

potentiallyacceptableuse–untilaleftwardbranchistaken.

Delphi4|2019189 ClassificationSchemasforAIFailures

therebyignoresamultitudeoffailuremodesthat

stemfromaccidentaloversights.

2.AIFailureClassificationDimensions

HerewemodifyandextendearlierworkbyY ampol-

skiy24inclassifyingAIriskfactors.Hollnagel25de-

constructssafetyinthestepsofphenomenology(ob-

servables),etiology(causes),andontology(nature).

Weaddresseachofthesestepsinproposingthefol-

lowingdimensionsasusefulclassificationcriteriafor

AIfailures:

–Consequences(phenomenology)

–Agency(etiology)

–Preventability(ontology)

–Stageofintroductionintheproductlifecycle(phe-

nomenologyandontology)

Eachwillbedenotedwitha2-or3-lettercodethat

wewilltagourexampleswith.

a.Consequences

Consequencesmaybeconsideredonthescaleofhu-

managgregationonwhichtheycanoccur(see

Table2).

Individualscanrangeinnumberfromonetoevery

memberofthehumanrace;thegroupingwillbeused

todenoteatwhattypeofaggregationtheactionof

thefailurewasaimedratherthanthenumberofin-

stancesaffected.Corporationsarelegalstructuresfor

doingbusiness,ofanysize.Communitiesaregroup-

ingsofpeopleorganisedforpurposesotherthanbusi-

nessandrangefromfamiliestonations.

–Physicalconsequencesoccurtoindividualsand

mayrangefrominconveniencetolossoflife.

–Mentalconsequencesoccurtoindividualsandin-

cludethealterationofmentalstatessuchasbe-

liefs,withconcomitantchangesinbehaviour.For

instance,thepurposeoreffectof‘fakenews’isto

causesuchchanges.26

–Emotionalconsequencesoccurtoindividualsand

includedepressivestatesresultingfromAIinci-

dentswithphysicalormentalconsequences,and

AIusurpingrolesthatpeoplehaveassumedtobe

unassailable.

–Financialconsequencesoccurtoindividuals,cor-

porations,andcommunities.

–Socialconsequencesarethemodificationsofbe-

haviourofsystemsororganisationsofpeople.

–Culturalconsequencesarethemodificationsofan

organisationorgrouping’svision,values,norms,

systems,symbols,language,assumptions,beliefs,

andhabits.27

Consequencesarenotnecessarilynegative,ormay

benegativeinsomerespectswhilebeingpositivein

others.Asuperintelligencethatenslavedhumansin

bootcampsmightkeeptheminoptimalphysicalcon-

ditionbutpessimalemotionalstate.

24RomanY ampolskiy,‘T axonomyofPathwaystoDangerousAI’

Proceedingsof2ndInternationalWorkshoponAI,Ethicsand

Society(AIEthicsSociety2016)143-148

25ErikHollnagel,Safety-IandSafety-II(AshgatePublishing2014)

26DavidM.J.Lazeretal,‘TheScienceofFakeNews’(2018)359

Science1094-1096

27DavidNeedle,BusinessinContext:AnIntroductiontoBusiness

andItsEnvironment(CengageLearningEMEA2014)

Table2:AIFailureConsequencesatHumanAggre-

gationLevels–SchemaTags

Human

Aggre-

gation

Scale

Consequences

Phys-

ical

Men-

tal

Emo-

tional

Finan-

cial

So-

cial

Cultur-

Individ-

ual

CIPCIMCIECIF

Corpora-

tion

CCFCCC

Commu-

nity

CYFCY

CYC

Table3:AIFailureLevelsof

Agency–SchemaTags

AgencyCode

AccidentalAA

NegligentAN

InnocuousAI

MaliciousAM

Delphi4|2019 190ClassificationSchemasforAIFailures

b.Agency

Theagencyofafailureisthedegreeofhumaninten-

tionalityinitsoriginorpropagation(seeTable3).

Anaccidentalfailureisonethatwasnotforeseen

andcouldnotreasonablyhavebeenforeseen.Weare

departingslightlyfromthecustomaryengineering

definitionof‘accident’hereinordertodrawamore

usefuldistinction.Leveson28defines‘accident’as‘ An

undesiredandunplanned(butnotnecessarilyunex-

pected)eventthatresultsin(atleast)aspecifiedlev-

elofloss.’Thusautomobileaccidentsareforeseeable

butneitherexpectednordesired.Wepreferinstead

todefineanegligentfailureasonethatwasnotfore-

seenbutcould(andperhapsshould)havebeenfore-

seen.

Aninnocuousfailureisonedeliberatelycaused,

butnotwithmaliciousintent,possiblywiththein-

tentofcausingamorebenigneffectthanwhatactu-

allyresulted.Amaliciousfailureisonethatwasini-

tiatedwiththeintentionofcausingdeleteriousef-

fects,whethertheywerespecificallytheeffectsthat

actuallyresultedorothers.Noconnectionwithlegal

definitionsofthesetermsshouldbeinferredfrom

theirattributiontospecificevents.

c.Preventability

Levelsofagencyareindependentofthedegreeof

preventability(seeT able4).

Somefailuremodesofsuperintelligencesarefore-

castbysomeauthoritiestobeunpreventable:‘[W]e

haveseenenoughtoconcludethatscenariosin

whichsomemachineintelligencegetsadecisive

strategicadvantagearetobeviewedwithgravecon-

cern.’29

d.LifecycleStage

Acommontaxonomyforcomputersystemerrorsis

thesoftwaredevelopmentlifecyclestage(seeT able

5);itisoftenassertedthatthecostoffixinganerror

ateachstageistentimesthecostoffixingitinthe

previousstage.30

Weaddinthelesscommonlyincludedstagesof

concept(wasitagoodideatodothisinthefirst

place?)atthebeginning,anddecommissioning

(whataretheproblemscausedbygettingridofthe

product)attheend.Asuperintelligencemightbe

highlyresistanttodecommissioning.31

IV .AIFailures

Withthesedimensionsinmindwenowexamine

variousreportedandhypothesisedfailures.Note

thatthereisanunavoidabledegreeofsubjectivevari-

abilityintheclassificationsofpreventabilityand

agency.

28(n17)

29(n5)154

30MauriceDawson,DarrellNormanBurrellandEmadRahim,

‘IntegratingSoftwareAssuranceintotheSoftwareDevelopment

LifeCycle(SDLC)’(2010)3JournalofInformationSystemsT ech-

nologyandPlanning49-53

31‘2001:ASpaceOdyssey(1968)-I'mSorry,DaveScene’

(YouT ube)<https://www.youtube.com/watch?v=Wy4EfdnMZ5g>

accessedon1November2019

Table4:AIFailureDegreeofPreventability–

SchemaTags

DegreeofPreventabilityCode

TriviallypreventablePT

PreventablewithsomedifficultyPS

PreventablewithgreatdifficultyPD

UnpreventablePU

Table5:SoftwareDevelopmentLifecy-

cleStages–SchemaTags

LifecycleStageCode

ConceptLC

DesignLD

DevelopmentLE

TestingLT

OperationLO

DecommissioningLG

Delphi4|2019191 ClassificationSchemasforAIFailures

1.ReportedFailures

WhereasYampolskiy32enumeratedseveraldozen

failuresinatimelinethathighlightedanexponen-

tiallyincreasingfrequencyandseverity,nearlyallof

theexampleswecitehereoccurredwithinthe

2016-2019periodandsoachronologicalordering

wouldnotbeilluminating.Wewillthereforeplace

theminsteadwithinamorenarrativestructure.

Themostrecognisableandstraightforwardclass

offailuresresultinphysicalinjurytohumans,going

backtotheclassicTherac-25radiationtherapyover-

dosecases33(CIP ,AN,PS,LD,LE,LO).WhenanAma-

zonwarehouserobotaccidentallypuncturedacon-

tainerofbearspray34(CIP,AN,PS,LT)itwasamore

benignoutcomeofanindustrialaccidentthanwhen

aChinesefactoryworkerwasimpaledwithtenfoot-

longspikes35(CIP ,AN,PT ,LD).Buttheseandother

morefatalaccidentswithindustrialrobotsgoing

backatleastto1984whenanoperatorwaskilledby

a2,500lbrobotthatcamebehindhimwithnowarn-

ing36(CIP ,AN,PS,LD)indicatelackofconsideration

forhumanssharingthesamelocationasmachines.

Acarproductionplantrobotgrabbedaworkerin-

steadofapartandcrushedhimagainstametalplate,

killinghim37(CIP ,AN,PS,LD).

Incidentsofcarsinsemi-autonomousoperation

causingfatalitiesincludeanUberincorrectlyclassi-

fyingapedestrianasafalsepositivematchbecause

toomanyreactionstoactualfalsepositivesresulted

inajerkyride38(CIP ,AN,PS,L T),andaTeslacrash-

ingafterrequestingdriverintervention39(CIP ,CCF ,

AA,PD,LE).

Inmedicine,IBM’sWatsonrecommended‘unsafe’

cancertreatments40(CIP,CCF,AA,PD,LT,LO),and

astudyof14yearsofroboticsurgeryconcludedthat

‘anon-negligiblenumberof[preventable]technical

difficultiesandcomplicationsarestillbeingexperi-

encedduringprocedures.’41(CIP ,CCF ,AA,PS,LD).

AIaccidentsmayresultindirectfinancialloss.The

May2010‘FlashCrash’resultedintheDowJonesIn-

dustrialAveragedroppingabout9%for36minutes

andresultedfromprogramtradingalgorithmsbeing

inadequatelypreparedtodealwithlargevolumesof

strategically-placedtradeswhichthemselveswere

computer-mediatedmalice42(CIF ,CCF,AA,AM,PD,

LD,L T).Remediationeffortsdidnotpreventmore

flashcrashesin2015.43

AmajorconcernintheapplicationofAIispriva-

cy.Consumerdevicesconnectedtocorporateclouds

ofidentitydatacomeunderscrutiny ,especially

when,forinstance,anAmazonAlexanoderecorded

aprivateconversationandsentittoarandomcon-

tact44(CIE,AA,PS,LT ,LO),oraniPhonebugallowed

userstolistenonothers’conversationsviaFace-

Time45(CIE,AA,PS,LT).Insomecases,thetechnol-

32RomanY ampolskiy,‘PredictingFutureAIFailuresfromHistoric

Examples’(2018)foresight138-152

33J.A.Rawlinson,‘ReportontheTherac-25’OCTRT/OCIPhysicists

Meeting(Kingston,Ontario1987)

34SaqibShah,‘ AmazonWorkersHospitalizedafterWarehouse

RobotReleasesBearRepellent’(engadget,6December2018)

<https://www.engadget.com/2018/12/06/amazon-workers

-hospitalized-robot>accessed20January2020

35T ariqTahir,'FactoryRobotImpalesWorkerwith10Foot-long

SteelSpikesafterHorrorMalfunction'(TheSun,14December

2018)<https://www.thesun.co.uk/news/7954270/factory-robot

-malfunctions-and-impales-worker-with-10-foot-long-steel

-spikes/>accessed1November2019

36JohnG.Fuller ,‘DeathbyRobot’(1984)Omni,45-46,97-102

37AssociatedPressinBerlin,'RobotKillsWorkeratVolkswagen

PlantinGermany'(TheGuardian,2July2015)<https://www

.theguardian.com/world/2015/jul/02/robot-kills-worker-at

-volkswagen-plant-in-germany>accessed1November2019

38T imothyB.Lee,'SoftwareBugLedtoDeathinUber’sSelf-

drivingCrash'(arsTechnica,7May2018)<https://arstechnica

.com/tech-policy/2018/05/report-software-bug-led-to-death-in

-ubers-self-driving-crash/>accessed20January2020

39FaizSiddiqui,'NTSB“Unhappy”withT eslaReleaseofInvestiga-

tiveInformationinFatalCrash'(WashingtonPost,1April2018)

<https://www.washingtonpost.com/news/dr-gridlock/wp/2018/04/

01/ntsb-unhappy-with-tesla-release-of-investigative-information

-in-fatal-crash/>accessed1November2019

40CaseyRossandIkeSwetlitz,'IBM’sWatsonSupercomputer

Recommended“UnsafeandIncorrect”CancerT reatments,

InternalDocumentsShow'(StatNews,25July2018)<https://

www.statnews.com/2018/07/25/ibm-watson-recommended

-unsafe-incorrect-treatments/>accessed1November2019.

41HomaAlemzadeh,JaishankarRaman,NancyLeveson,Zbigniew

KalbarczykandRavishankarK.Iyer,‘ AdverseEventsinRobotic

Surgery:ARetrospectiveStudyof14Y earsofFDAData’(2016)

11PLoSONE

42'2010FlashCrash'(Wikipedia)<https://en.wikipedia.org/wiki/

2010_Flash_Crash>accessed1November2019

43CoryMitchell,'TheT woBiggestFlashCrashesof2015'(Investo-

pedia,25June2019)<https://www .investopedia.com/articles/

investing/011116/two-biggest-flash-crashes-2015.asp>accessed1

November2019.

44GaryHorcher ,'AmazonAlexaRecordedPrivateConversation,

SentittoRandomContact,WomanSays'(WHBQ,24May2018)

<https://www.fox13memphis.com/news/trending-now/amazon

-alexa-recorded-private-conversation-sent-it-to-random-contact

-woman-says/755720160>accessed1November2019

45MarkGurman,'AppleBugLetsiPhoneUsersListeninonOthers

ViaFaceT ime'(Bloomberg,28January2019)<https://www

.bloomberg.com/news/articles/2019-01-29/apple-bug-lets-iphone

-users-listen-in-on-others-via-facetime>accessed1November

2019

Delphi4|2019 192ClassificationSchemasforAIFailures

ogyfacilitatedacasualviolationofprivacysuchas

whenUberusers’locationsandidentitiesweredis-

playedonascreenatalaunchparty46(CIE,AI,PT,

LC).

Privacyviolationscarrymoreseriousconse-

quenceswhentheybecomemisidentifications.The

ACLUdemonstratedthatwhentheyshowedthat

Amazonfacialrecognitionwouldflagcertainmem-

bersofCongressaswantedcriminals47(CYF,CYS,

AN,PS,LD).Alackoftrainingdata(andimplicitbias)

resultedinfacialrecognitionsystemsbeingunable

toseeblackpeople48ortaggingthemasgorillas49

(CIE,CYS,AN,PD,LD).Facialrecognitionusedby

policeintheUnitedKingdomhasbeenrecorded

makingmanyfalsepositiveidentifications50(CIE,

CIF,CYF,CYS,AN,PS,LT,LO).AndinChina,facial

recognitionsystemsdeployedforautomatedmisde-

meanorticketingpubliclyshamedawomanasajay-

walkerwhenmistakingherphotoonthesideofa

busforthewomanherself51(CIE,CIF ,AN,PD,LE)

andadriverwasticketedforusingacellphonewhen

hewasactuallyscratchinghisface52(CIE,CIF,AN,

PD,LE).T rafficcamerasinNewOrleansticketed

parkedcarsforspeeding53(CIF ,AN,PS,LD,LO).A

manwasfalselyarrestedaftersystemsatApple

misidentifiedhimasstealingfromitsstores54(CIP ,

CIM,CIE,CIF,CCF,AA,PS,LE).

Notallmisidentificationsresultinsuchobvious

harm.ArtistT omWhitespecialisesincreatingab-

stract(andveryunarousing)artthatisflaggedasun-

acceptablenuditybysocialmediaAI.55Thismachine

myopiaindicatesthatthedevelopmentofusefulim-

agecensorshipisnotyetrealisedandsomeinoffen-

siveartissuppressed.(CIM,CIF,AA,PD,LD).

Implicitmisidentificationbycategoryisbias,an-

othertopicofgreatconcerninAIdevelopment.With

goodreason:areportconcludedthatAIstrainedon

hiringdecisionswouldreplicateoramplifyhuman

bias,56Amazon’shiringAIturnedouttobesexist,57

andtheCOMPASsystemusedinWisconsintopre-

dictrecidivismwasbiasedagainstblacks58(CIE,CIF ,

CYC,AA,PD,LE).Justashumanbiasoftenresults

frominadequateexposuretodiversity ,AIbiasoften

arisesfromthesamecause.AnattempttouseAIto

objectivelyjudgeanonlineinternationalbeautycon-

testwithouthumanbiasfailedwhenonlyoneof44

winnersitchosehaddarkskin,promptingspecula-

tionthatthiswasduetothetrainingdatabasehav-

ingfewdarkfaces59.AndtheNewZealandautomat-

edpassportapplicationcheckingsystemrejectedan

46KashmirHill,'”GodView”:UberAllegedlyStalkedUsersFor

Party-Goers'ViewingPleasure'(Forbes,3October ,2014)<https://

www.forbes.com/sites/kashmirhill/2014/10/03/god-view-uber

-allegedly-stalked-users-for-party-goers-viewing-pleasure/

#4b7dd5593141>accessed1November2019

47CyrusFarivar,'Amazon’ sRecognitionMessesUp,Matches28

LawmakerstoMugshots'(arsTechnica,26July2018)<https://

arstechnica.com/tech-policy/2018/07/amazons-rekognition

-messes-up-matches-28-lawmakers-to-mugshots/>accessed1No-

vember2019

48'AlgorithmicJusticeLeague'<https://www .ajlunited.org/>ac-

cessed1November2019

49JanaKasperkevic,'GoogleSaysSorryforRacistAuto-tagin

PhotoApp'(TheGuardian,1July2015)<https://www .theguardian

.com/technology/2015/jul/01/google-sorry-racist-auto-tag-photo

-app>accessed1November2019

50MattBurgess,'FacialRecognitionT echUsedbyUKPoliceis

MakingaT onofMistakes'(WiredUK,4May2018)<https://www

.wired.co.uk/article/face-recognition-police-uk-south-wales-met

-notting-hill-carnival>accessed1November2019

51XinmeiShen,'FacialRecognitionCameraCatchesT opVusiness-

woman"Jaywalking"BecauseherFaceWasonaBus'(Abacus

News,22November2018)<https://www.abacusnews.com/digital

-life/facial-recognition-camera-catches-top-businesswoman

-jaywalking-because-her-face-was-bus/article/2174508>accessed

1November2019

52WTF ,'ChineseDriverFinedforScratchinghisFaceafterP assing

AITrafficCamera'(9gag,26May2019)<https://9gag.com/gag/

av8VBdd/chinese-driver-fined-for-scratching-his-face-after -passing

-ai-traffic-camera>accessed1November2019

53WillieJamesInman,'T rafficCamerainNewOrleansGiving

SpeedingTicketstoP arkedCars'(FoxNews,11April2018)

<https://www.foxnews.com/auto/traffic-camera-in-new-orleans

-giving-speeding-tickets-to-parked-cars>accessed1November

2019

54'AppleAIAccusedofLeadingtoMan'sWrongfulArrest'(BBC

News,23April2019)<https://www.bbc.com/news/technology

-48022890>accessed1November2019

55JasonBailey,'AIArtistsExpose“Kinks”InAlgorithmicCensor-

ship'(Artnome,11December2018)<https://www .artnome.com/

news/2018/12/6/ai-artists-expose-kinks-in-algorithmic-censorship

>accessed1November2019

56NicolT urnerLee,PaulResnicketal,'AlgorithmicBiasDetection

andMitigation:BestPracticesandPoliciestoReduceConsumer

Harms'(BrookingsInstitute,22May2019)<https://www

.brookings.edu/research/algorithmic-bias-detection-and

-mitigation-best-practices-and-policies-to-reduce-consumer

-harms/>accessed1November2019

57JamesCook,'AmazonScraps“SexistAI”RecruitingT oolthat

ShowedBiasAgainstWomen'(TheTelegraph,10October2018)

<https://www.telegraph.co.uk/technology/2018/10/10/amazon

-scraps-sexist-ai-recruiting-tool-showed-bias-against/>accessed1

November2019

58EdY ong,'APopularAlgorithmIsNoBetteratPredictingCrimes

ThanRandomPeople'(TheAtlantic,17January2018)<https://

www.theatlantic.com/technology/archive/2018/01/equivant

-compas-algorithm/550646/>accessed1November2019

59JordanPearson,'WhyAnAI-JudgedBeautyContestPickedNearly

AllWhiteWinners'(Vice,5September2016)<https://www .vice

.com/en_us/article/78k7de/why-an-ai-judged-beauty-contest

-picked-nearly-all-white-winners>accessed1November2019

Delphi4|2019193 ClassificationSchemasforAIFailures

Asianapplicant’sphotograph,claimingthat‘Sub-

ject’seyesareclosed.’60Astudydemonstratedthat

implicitraceandgenderbiasesintrainingcorpora

flowedthroughintoAIstrainedonthosecorpora.61

Inthehandsofanauthoritarianregime,AIcan

createenvironmentspromptingcomparisonswith

Orwell’s1984.Nowhereisthismoreapparentthanin

China,whichhasembracedfacialrecognitionona

largescale.62AIthereblocksmentionoftheTianan-

menSquaremassacreonsocialmedia63(CYS,AI,PT,

LC).Whilethissoftwareisbeingusedtocreateex-

actlyitsintendedeffect,welabelthisafailurebecause

ithasconsequencesmanywesternobserverswould

considertobesociallyharmful.Chinahasa‘social

credit’scoringsystemreminiscentofaBlackMirror

episode,64linkedtosocialmediaandconsumersys-

temssuchasSesameCredit,65thatwillbanpeople

fromcertainvenueslikeflightsandhotelsforpoor

scores,whichmaybeincurredbyundesirablebehav-

ioursuchasbuyingvideogames(CYC,AI,PT ,LC).

Somecommentatorsspeculatethatthiswillhavecon-

sequencesinhealthcare.66AlsoinChina,AIisbeing

usedtogradeschoolpapers,67withsomegoodwrit-

ingbeinggivenpoormarks(CYS,AI,PD,LO).And

AIisusedtomonitorthemoodsofworkers68andthe

attentionpaidbychildreninclass69withthemostat-

tentivebeingrewarded(CIM,CYC,AI,PT ,LC).

IntheWestthedangersaremorenascent.Re-

searchersattheUniversityofPennsylvaniademon-

stratedthattextualanalysisofanindividual’sFace-

bookpostscouldpredict21differentmedicalcondi-

tionssuchasdiabetes.70OthersshowedthatAIwas

betterthanpeopleatdeterminingsexualorientation

fromaphotograph,71whileathirdgroupdetermined

thatAIcoulddetectcertaingeneticdiseasesfrom

faces.72ADepartmentofHomelandSecuritypro-

grampredictswhichflyersarepotentialterrorists73

fromdemographicandtraveldataalone,andifthose

travellersmakeittotheEuropeanUniontheymay

faceanAI-poweredliedetectionsystematthebor-

der.74ThestartupFaceptionclaimsitssoftwarecan

predictpersonalitytraitssuchaspedophileorpoker

playerfromfacialimageanalysis,causingonecom-

mentatortolikenittophrenology.75Aperson’sgait

60JamesT itcomb,'RobotPassportCheckerRejectsAsianMan's

PhotoforHavinghisEyesClosed'(TheT elegraph,7December

2016)<https://www.telegraph.co.uk/technology/2016/12/07/robot

-passport-checker-rejects-asian-mans-photo-having-eyes/>ac-

cessed1November2019

61AylinB.Caliskan,‘SemanticsDerivedAutomaticallyfromLan-

guageCorporaContainHuman-likeBiases’(2017)Science

183-186

62SijiaJiang,'BackingBigBrother:ChineseFacialRecognition

FirmsAppealtoFunds'(Reuters,12November2017)<https://

www.reuters.com/article/us-china-facialrecognition-analysis/

backing-big-brother-chinese-facial-recognition-firms-appeal-to

-funds-idUSKBN1DD00A>accessed1November2019

63MichaelGrothaus,'NowAIEasilyErasestheTiananmenSquare

MassacrefromOnlineMemory'(FastCompany,28May2019)

<https://www.fastcompany.com/90355806/now-ai-easily-erases

-the-tiananmen-square-massacre-from-online-memory>accessed

1November2019

64J.Wright(Director)(2016)BlackMirror:Nosedive[Motion

Picture]

65'SocialCreditSystem'(Wikipedia)<https://en.wikipedia.org/wiki/

Social_Credit_System>accessed1November2019

66JohnHarris,'TheT yrannyofAlgorithmsisPartofourLives:Soon

TheyCouldRateEverythingWeDo'(TheGuardian,5March

2018)<https://www.theguardian.com/commentisfree/2018/mar/

05/algorithms-rate-credit-scores-finances-data>accessed1No-

vember2019

67StephenChen,'China’ sSchoolsareQuietlyUsingAItoMark

Students’Essays...ButdotheRobotsMaketheGrade?'(South

ChinaMorningP ost,27May2018)<https://www .scmp.com/

news/china/society/article/2147833/chinas-schools-are-quietly

-using-ai-mark-students-essays-do>accessed1November2019

68JamieFullerton,'”Mind-reading”T echBeingUsedtoMonitor

ChineseWorkers'Emotions'(TheTelegraph,30April2018)

<https://www.telegraph.co.uk/news/2018/04/30/mind-reading

-tech-used-monitor-chinese-workers-emotions/>accessed1No-

vember2019

69JiayunFeng,'ChineseP arentsWantStudentstoWearDystopian

Brainwave-detectingHeadbands'(supChina,5April2019)

<https://supchina.com/2019/04/05/chinese-parents-want-students

-to-wear-dystopian-brainwave-detecting-headbands/>accessed1

November2019

70RainaM.Merchantet,‘EvaluatingthePredictabilityofMedical

ConditionsfromSocialMediaPosts’(2019)PLoSONE

71YilunWangandMichalKosinski,‘DeepNeuralNetworksare

MoreAccuratethanHumansatDetectingSexualOrientation

fromFacialImages(2018)JournalofPersonalityandSocial

Psychology246–257

72NinaAvramova,'AITechnologyCanIdentifyGeneticDiseasesby

LookingatY ourFace,StudySays'(CNN,8January2019)<https://

edition.cnn.com/2019/01/08/health/ai-technology-to-identify

-genetic-disorder-from-facial-image-intl/index.html>accessed1

November2019

73SamBiddle,'HomelandSecurityWillLetComputersPredictWho

MightBeaT erroristonY ourPlane—JustDon’tAskHowIt

Works'(TheIntercept,3December2018)<https://theintercept

.com/2018/12/03/air-travel-surveillance-homeland-security/>ac-

cessed1November2019

74'SmartLie-detectionSystemtoTightenEU'sBusyBorders'(Euro-

peanCommission,24October2018)<https://ec.europa.eu/

research/infocentre/article_en.cfm?artid=49726>accessed1No-

vember2019

75MattMcFarland,'T erroristorPedophile?ThisStart-upSaysitCan

OutSecretsbyAnalyzingFaces'(WashingtonPost,24May2016)

<https://www.washingtonpost.com/news/innovations/wp/2016/

05/24/terrorist-or-pedophile-this-start-up-says-it-can-out-secrets

-by-analyzing-faces/>accessed1November2019

Delphi4|2019 194ClassificationSchemasforAIFailures

canbeusedtoidentifythem.76Twosystemssupply

‘predictivepolicing’systemsthat,invitingacompar-

isonwiththemovieMinorityReport,suggestwhere

crimeislikelytooccur.7778Companiesexploithu-

manpsychologytogetourattention,79theUSmili-

tarystudieshowtoinfluenceTwitterusers,80andthe

PentagonwantstopredictprotestsagainsttheUS

Presidentviasocialmediasurveillance.81

AsY ampolskiy82pointedout,‘ AnAIdesignedto

doXwilleventuallyfailtodoX,’codifiedastheFun-

damentalTheoremofSecurity:Thereisnosuchthing

asa100%securesystem.Inalltheexamplesinthe

previousparagraphthelatentfailuresaretheones

impliedbythistheorem,withtheirconcomitantrisks.

TheconsequencesofmisinformationspreadbyAI

includeofcourse‘fakenews,’suchasthatattributed

toCambridgeAnalytica,83assiduouslyspreadbyso-

cialmedia,84and‘deepfake’videos,85whichcouldbe

usedtoautomateblackmailatscale86(CIM,CYS,AM,

PU,LO)

AclassofincidentsillustratesthatmuchAIisnot

yetmature.Ahardwaredesignbugallowedmemo-

ryprotectionviolationsinyears’worthofIntel

chips87(CCF ,AA,PD,LD).AndMicrosoft’sTaychat-

botbecameracistwithinhoursofbeingdeployedto

learnfromotherTwitterusers88(CYS,AA,PS,

LC/LD).Someaspectsofthisimmaturityarefunda-

mentallybrittle;forinstance,whenadigitalex-

changelost$137millionbecausetheonepersonhold-

ingthemasterpassworddied,89orwhenbotstasked

withmaintainingWikipediafoughtwitheachother

foryears,90Deepreinforcementlearningfailsmore

oftenthanadmitted.91

Intentionalmisusespansmanyincidents;tocite

two,smartscootersforhirewerehackedtodisplay

obscenemessagesandbeusedwithoutpayment92

(CCF,AM,PS,LE)andDomino’sPizzaaffiliationapp

wasfooledintograntingpointsbyfakepicturesof

pizza93(CCF,AM,PD,LC).

Some‘backfire’eventsresultindamagetotheAI

industrythroughoverreachingormisrepresentation.

Forinstance,apreternaturallycapablehealthcareAI

called‘Zach’inNewZealandwassuspectedtobea

personindisguise.94AndtheSophiarobotattractsa

76'ForensicGaitAnalysis'(RoyalSocietyofEdinburgh,November

2017)<https://royalsociety.org/~/media/about-us/programmes/

science-and-law/royal-society-forensic-gait-analysis-primer-for

-courts.pdf>accessed1November2019

77'PredPol'<https://www .predpol.com/>accessed1November

2019

78'P alantir'<http://www.palantir.com/>accessed1November2019

79T ristanHarris,'HowaHandfulofT echCompaniesControl

BillionsofMindsEveryDay'(TED,April2017)<https://www.ted

.com/talks/tristan_harris_the_manipulative_tricks_tech_companies

_use_to_capture_your_attention>accessed1November2019

80BenQuinnandJamesBall,'USMilitaryStudiedHowtoInfluence

TwitterUsersinDarpa-fundedResearch'(TheGuardian,8July

2014)<https://www.theguardian.com/world/2014/jul/08/darpa

-social-networks-research-twitter-influence-studies>accessed1

November2019

81NafeezAhmed,'PentagonWantstoPredictAnti-T rumpProtests

UsingSocialMediaSurveillance'(Vice,30October2018)

<https://www.vice.com/en_us/article/7x3g4x/pentagon-wants-to

-predict-anti-trump-protests-using-social-media-surveillance>ac-

cessed1November2019

82(n19)

83'CambridgeAnalyticaplantedfakenews'(BBC,20March2018)

<https://www.bbc.com/news/av/world-43472347/cambridge

-analytica-planted-fake-news>accessed1November2019

84EmergingT echnologyfromthearXiv,'FirstEvidenceThatSocial

BotsPlayaMajorRoleinSpreadingFakeNews'(Technology

Review,7August2017)<https://www.technologyreview.com/s/

608561/first-evidence-that-social-bots-play-a-major-role-in

-spreading-fake-news/>accessed1November2019

85BenCollins,'ThisViralSchwarzeneggerDeepfakeisn'tJust

Entertaining.It'saWarning'(NBCNews,12June,2019)<https://

www.nbcnews.com/tech/tech-news/viral-schwarzenegger

-deepfake-isn-t-just-entertaining-it-s-warning-n1016851>ac-

cessed1November2019

86P aulBricman,'DeepFakeRansomware'(Medium,2February

2019)<https://medium.com/@paubric/deepfake-ransomware

-oaas-part-1-b6d98c305cd9>accessed1November2019

87ZackWhittaker ,'NewSecret-spillingFlawAffectsAlmostEvery

IntelChipSince2011'(T echCrunch,14May2019)<https://

techcrunch.com/2019/05/14/zombieload-flaw-intel-processors/>

accessed1November2019

88ElleHunt,'T ay,Microsoft'sAIChatbot,GetsaCrashCoursein

RacismfromTwitter'(TheGuardian,24March2016)<https://

www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai

-chatbot-gets-a-crash-course-in-racism-from-twitter>accessed1

November2019

89DanGoodin,'DigitalExchangeLoses$137MillionasFounder

TakesP asswordstotheGrave'(arsT echnica,2February2019)

<https://arstechnica.com/information-technology/2019/02/digital

-exchange-loses-137-million-as-founder-takes-passwords-to-the

-grave/>accessed1November2019

90SaraChodosh,'WikipediaBotsSpentY earsFightingSilent,Tiny

BattleswithEachOther'(PopularScience,27February2017)

<https://www.popsci.com/wikipedia-bots-fighting/>accessed1

November2019

91AlexIrpan,'DeepReinforcementLearningDoesn'tWorkY et'

(SortaInsightful,14February2018)<https://www .alexirpan.com/

2018/02/14/rl-hard.html>accessed1November2019

92MattNovak,'LimeScootersHackedtoSaySexualThingsto

RidersinAustralia'(Gizmodo,24April2019)<https://gizmodo

.com/lime-scooters-hacked-to-say-sexual-things-to-riders-in

-1834264534>accessed1November2019

93MatthewGault,'T akePicturesofFakePizzastoGetaFree

PizzafromDomino's'(Vice,6March2019)<https://www .vice

.com/en_us/article/kzdkgw/take-pictures-of-fake-pizzas-to-get-a

-free-pizza-from-dominos>accessed1November2019

94DavidFarrier,'TheMysteryofZach,NewZealand’sall-too-miracu-

lousmedicalAI'(TheSpinoff,6March2018)<https://thespinoff.co

.nz/the-best-of/06-03-2018/the-mystery-of-zach-new-zealands-all

-too-miraculous-medical-ai/>accessed1November2019

Delphi4|2019195 ClassificationSchemasforAIFailures

degreeofadulationfarbeyonditsrealcapabilities.95

ThethreathereistothereputationofAIanditscom-

munity(CYF,CYC,AI,PU,LO).

AIthatisunintentionallyinsensitivealsodamages

itsownreputation,suchastheAIthatthoughtthat

houseburningdownwas‘spectacular’96andPaypal’s

virtualassistantwhichinsensitivelyreplied,‘Great!’

whensomeonetoldit,‘Igotscammed’97(CIE,AA,

PS,LE).TheStarbucksshift-schedulingsoftwarewas

alsoinsensitivewhenitoptimisedforhour-by-hour

businessneedsbutassignedworkerstounpre-

dictableanderraticschedules98(CIP ,CIF ,CCC,AN,

PT,LD).AIthatistrustedwithoutverificationmay

notliveuptothattrust,suchaswhenamodelused

togradethe‘value-add’impartedbyNewYorkCity

teacherswasfoundtogenerateessentiallyrandom

results99(CIM,CIE,CIF,AN,PS,LT).Acorporateem-

ploymentworkflowsystemwasunstoppableinter-

minatinganemployeeerroneouslyflaggedassuper-

fluous;100afterthreeweeksspentfixingtheerrorhe

declinedtoreturntothefirm(CIE,CIF,CCF,AA,PS,

LD).

Somefailuresaresobenignonthesurfacethat

manycasualobserverswouldclassifythemascute

behaviourratherthanfailures.Whenarobot(with

smileyfacetoboot)ontheInternationalSpaceSta-

tionstoppedobeyingastronauts101theparallelswith

HAL9000of2001:ASpaceOdysseyweresoirre-

sistibleastoobscuretherealrisksofacomputerfail-

ureinacriticalenvironment.Apple’sSiri’sinitialre-

sponsetotherequest‘Callmeanambulance’wasto

refertotheuserthereafteras‘ambulance’102(CIP ,

AA,PS,LE).Whenatextgeneratorcreatedweirdde-

scriptionsofBitcoin103,andanAI’spredicted

YouTubepornographysearches,104theresultswere

sofunnyastobeequallydisarming(CYC,AA,PS,

LD).Atrivialtypointhecodeforagameagentmade

itmucheasiertobeatthanitshouldhavebeen105

(CIM,AA,PT ,LT).ARoombaspreaddogpoopall

overahouse106(CIP,CIF,AA,PS,LE).Asignprint-

edinWelshtranslatedto‘Iamnotintheofficeat

themoment.Sendanyworktobetranslated.’107

(CYC,AN,PT,LO).The‘swarmintelligence’UNU

failedtopredicttheresultsoftheKentuckyDerby

thesecondtimearoundafterpreviouslywinningthe

superfecta.108(CIF,AA,PD,LE).Aneuralnetwork

hallucinatedsheepinimageswheretherewerenone,

ormislabelledthemwhentheywereplacedin(ad-

mittedlyunusual)locations109(CIM,AA,PS,LE).

Andinastoryguaranteedtogetmorelaughsthan

95NoelSharkey,'MamaMiaIt'sSophia:AShowRobotOrDanger-

ousPlatformToMislead?'(Forbes,17November2018)<https://

www.forbes.com/sites/noelsharkey/2018/11/17/mama-mia-its

-sophia-a-show-robot-or-dangerous-platform-to-mislead/

#4fabeea87ac9>accessed1November2019

96MargaretMitchell,'HowWeCanBuildAItoHelpHumans,Not

HurtUs'(TED,October2017)<https://www.ted.com/talks/

margaret_mitchell_how_we_can_build_ai_to_help_humans_not

_hurt_us/transcript>accessed1November2019

97Facebook,20March2019<https://www .facebook.com/photo

.php?fbid=10217225240875399>accessed1November2019

98JodiKantor ,'WorkingAnythingbut9to5'(NewYorkTimes,13

August2014)<https://www.nytimes.com/interactive/2014/08/13/

us/starbucks-workers-scheduling-hours.html>accessed1Novem-

ber2019

99GaryRubinstein,'AnalyzingReleasedNYCValue-AddedData

Part2'(T eachforUs,28February2012)<http://garyrubinstein

.teachforus.org/2012/02/28/analyzing-released-nyc-value-added

-data-part-2/>accessed1November2019

100JaneWakefield,'TheManWhoWasFiredbyaMachine'(BBC

News,21June2018)<https://www.bbc.com/news/technology

-44561838>accessed1November2019

101JamieSeidel,'CIMON,theInternationalSpaceStation’ sArtificial

Intelligence,HasTurnedBelligerent'(NewsCorpAustralia,5

December2018)<https://www.news.com.au/technology/science/

space/cimon-the-international-space-stations-artificial

-intelligence-has-turned-belligerent/news-story/

953a84bc8c4fe414eed2d0550e1d8bf4>accessed1November

2019

102WillKnight,'TougherT uringTestExposesChatbots’Stupidity'

(TechnologyReview,14July2016)<https://www

.technologyreview.com/s/601897/tougher -turing-test-exposes

-chatbots-stupidity/>accessed1November2019

103DanielOberhaus,‘WatchThisHilariousBitcoinExplainerGener-

atedbyanAI’(Vice,23May2018)<https://www .vice.com/en_us/

article/xwmy9a/watch-botnik-ai-bitcoin-explainer>accessed01

November2019

104DrewSchwartz,‘AIPredictedtheFutureofPornSearchesandWe

Can'tStopLaughing’(Vice,6March2018)<https://www .vice

.com/en_us/article/bj54xv/ai-predicted-the-future-of-porn

-searches-and-we-cant-stop-laughing-vgtrn>accessed01Novem-

ber2019

105SamMachkovech,‘AY ears-old,One-letterTypoLedtoAliens:

ColonialMarines’WeirdAI’(arsTechnica,13July2018)<https://

arstechnica.com/gaming/2018/07/a-years-old-one-letter-typo-led

-to-aliens-colonial-marines-awful-ai/>accessed01November

2019

106JesseNewton(F acebook,9August2016)<https://www .facebook

.com/jesse.newton.37/posts/776177951574>accessed01No-

vember2019

107‘E-mailErrorEndsuponRoadSign’(BBC,31October2008)

<http://news.bbc.co.uk/2/hi/7702913.stm>accessed01Novem-

ber2019

108DavidZ.Morris,‘ArtificialIntelligenceFailsonKentuckyDerby

Predictions’(Fortune,7May2017)<https://fortune.com/2017/05/

07/artificial-intelligence-kentucky-derby-predictions/>accessed

01November2019

109JanelleShane,‘DoNeuralNetsDreamofElectricSheep?’(AI

Weirdness,18March2018)<https://aiweirdness.com/post/

171451900302/do-neural-nets-dream-of-electric-sheep>ac-

cessed01November2019

Delphi4|2019 196ClassificationSchemasforAIFailures

fearsofAIfailure,Alexadeviceswereallegedtobe

spontaneouslylaughing.110

Behaviorthatisalsoperceivedas‘cute’inthesense

of‘lookathowsmartmychildis,’canbemorecon-

cerningbecauseitindicatesjusthowcreativeAIcan

beinsolvingproblemswithsolutionsthateludedhu-

mans.AIs‘cheat’atgamesbyfindingloopholesin

therulesorunintendedbackdoorsintheimplemen-

tation.111OneAIinvented(orrediscovered)

steganographyinordertomeetitsgoals.112And

GPT2,atextgeneratordevelopedbyOpenAI,anor-

ganisationdedicatedtoopensourcingAItoensure

itssafety ,wasdeemedtobesogoodatwhatitdid

thatitwouldbetoodangeroustopublishitscode.113

GeneticAlgorithms

Geneticalgorithmscanbesoinnovativeat‘breaking

therules’114115thattheycheckeverycategoryoffail-

ureclassification,suggestingapathtowardsun-

boundedrisk,andarethereforecollectedinthissub-

section.

–Theycanexploitmisfeaturesorbugsintheiren-

vironment,suchaswheninthedevelopmental

stagesoftheNEROvideogame,players’robots

evolvedawigglingmotionthatallowedthemto

walkupwallsratherthansolvetheobstacles‘prop-

erly’bywalkingaroundthewalls,116orwhenina

capstoneprojectforagraduatelevelclass,students

wererequiredtomakeaafive-in-a-rowTic- Tac-T oe

gameplayedonaninfinitelylargeboard.Onesub-

mission’salgorithmevolvedtorequestnon-exis-

tentmovesthatwereextremelyfaraway,leading

toanautomaticwinsincetheotherplayerssys-

temwouldcrash.117

–Theycan‘cheat’byexploitingloopholesinthe

rulesoftheirgoals,suchasinanexperimentthat

involvedorganismsnavigatingpaths,whenone

organismcreatedanodometertoallowittonavi-

gatethepathpreciselyandearnaperfectscore,118

orwhenanattempttocreatecreaturesthatcould

evolveswimmingstrategiesresultedinthem

learningthatbytwitchingtheirbodypartsrapid-

ly,theycouldobtainmoreenergythatletthem

swimatunrealisticspeeds.119

–Theycanreinvent,totheircreators’surprise,ca-

pabilitiesofbiologicalorganisms,suchasinanex-

perimentwhereroboticorganismshadtofind

foodsorpoisonsthatwerebothrepresentedby

redlightsandcouldusebluelightstocommuni-

catewithotherrobots,theorganismsevolvedin

surprisingwaysthatresembledmimicryanddis-

honestyinnature,120orwhenadigitalevolution

modelthatwasinitiallythoughttohavebeena

completefailure,wasdiscoveredtohaverepro-

ducedthebiologicalconceptDrake’srulewithout

havingbeentoldtodoso.121

–Theycanimprovisenovelsolutionstotheiras-

signedtasks,suchaswhen3-Dcreaturesthatcould

run,walk,andswimweregaugedbyafitnessfunc-

tionofaveragegroundvelocity ,whichresultedin

creaturesthatweretallandrigid,fallingoverand

110RachelSandler,‘SomeAmazonEchoDevicesAreSpontaneously

Laughing,AndNobodyKnowsWhy’(ScienceAlert,7March

2018)<https://www.sciencealert.com/amazon-echo-devices-are

-creepily-laughing-at-people>accessed01November2019

111VictoriaKrakovna,‘SpecificationGamingExamplesinAI’(2April

2018)<https://vkrakovna.wordpress.com/2018/04/02/specification

-gaming-examples-in-ai/>accessed01November2019

112DevinColdewey,‘ThisCleverAIHidDatafromitsCreatorsto

CheatatitsAppointedT ask’(techcrunch,31December2018)

<https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-from

-its-creators-to-cheat-at-its-appointed-task/>accessed01Novem-

ber2019

113TomSimonite,‘TheAIT extGeneratorThat'sT ooDangerousto

MakePublic’(Wired,14February2019)<https://www .wired

.com/story/ai-text-generator-too-dangerous-to-make-public/>ac-

cessed01November2019

114JanelleShane,‘WhenAlgorithmsSurpriseUs’(AIWeirdness,13

April2018)<https://aiweirdness.com/post/172894792687/when

-algorithms-surprise-us>accessed01November2019

115JoelLehmanetal,‘TheSurprisingCreativityofDigitalEvolution:

ACollectionofAnecdotesfromtheEvolutionaryComputation

andArtificialLifeResearchCommunities’(2018)arX -

iv:1803.03453v1[cs.NE]

116KennethO.Stanleyetal,‘Real-timeNeuroevolutionintheNERO

VideoGame’(2005)9IEEETransactionsonEvolutionaryCompu-

tation653–668

117DavidE.MoriartyandRistoMiikkulainen,‘FormingNeural

NetworksthroughEfficientandAdaptiveCo-evolution’(1997)5

EvolutionaryComputation373–399

118LauraM.Grabowskietal,‘ ACaseStudyoftheDeNovoEvolu-

tionofaComplexOdometricBehaviorinDigitalOrganisms

(2013)8PL oSOnee60466

119KarlSims,‘EvolvingVirtualCreatures’(1994)Proceedingsofthe

21stAnnualConferenceonComputerGraphicsandInteractive

Techniques15–22

120SaraMitri,‘T heEvolutionofInformationSuppressioninCommu-

nicatingRobotswithConflictingInterests’(2009)Proceedingsof

theNationalAcademyofSciences15786–15790

121ThomasK.Hindré,‘NewInsightsintoBacterialAdaptation

ThroughInVivoandInSilicoExperimentalEvolution’(2012)10

NatureReviewsMicrobiology352–365

Delphi4|2019197 ClassificationSchemasforAIFailures

usingtheirpotentialenergytogainhighveloci-

ty,122orwhenarobotarmwasprogrammedtoin-

teractwithasmallboxonatable,butthegripper

wasbroken,resultingintherobothittingthebox

withthegripperinawaythatwouldforcethegrip-

pertoholdtheboxfirmly .123

–Theycancreativelyexceedtheirgoals,suchas

whentheartificiallifesystemTierra,notexpect-

edtoevolvehigherlifeformsforyears,created

complexecologicalsystemsonthefirstsuccessful

run,124orwhenrobotsthatweredesignedtode-

tectandtraveltoalightsourceevolvedaspinning

behaviorthatwasmoreefficientthantheexpect-

edBraitenberg-stylemovement.125

2.HypotheticalFailures

Avideoproducerdepictedafictionalfuturewhere

anartificialsuperintelligencechargedwithcopyright

enforcementhackedpeople’sbrainswithnanotech-

nologytocorrectviolations126(CIM,CYC,AI,PD,

LO).ItwasdemonstratedthataDNAsequencercould

behackedthrough(currentlynon-existent)flawsin

acompressionalgorithm127(CIPAM,PD,LE).

MostshowsthatexploreAIfailuredevelopa

themeepitomisedbyTerminatorseries:amassiveAI

becomesself-awareandattemptstodestroyhuman-

ity.(CIP,CIE,CIF,CCF,CYF ,CYS,CYC,AN,AI,PD,

LC,LO).V ariationsincludeColossus:TheForbin

Project,wheretheAIimprisonshumanitytoendcon-

flict(CIM,CIE,CYS,CYC,AN,AI,PD,LC,LO),the

samegoalastheAIVIKIinthemovieI,Robotand

therobotsinJackWilliamson’snovelette‘WithFold-

edHands’.128Oneoftheleastapocalypticfailureswas

exploredinthe2013filmHer,wherevirtualassistant

AIshaveunforeseenintimaterelationshipswith

manyhumanswhoarelargelychangedforthebet-

ter(CIE,AA,PD,LD).

BroadClassificationsofFutureFailureScenarios

Anotherclassificationforfailurescanbeappliedto

futurescenarios.

Figure2depictstheseverityandscale(numberof

individualsaffected)ofbroadlyclassifiedfailuresce-

narios.Inchronologicalordertheseare:

1.Autonomousweapons,whichcurrentlymostly

fallintothe‘lethal’category,129130.

2.Employmentautomation:Thepotentialsegment

ofthepopulationmadejoblessthroughAIau-

tomation.

3.Controlfailures:AIofsufficientcomplexityand

powerthatbugscausecatastrophes.

4.ConsciousAIs:ControlfailuresinAGIsorAIsthat

aresocomplexthattheirbehaviorismostuseful-

lycategorisedas‘conscious.’

5.Self-replicatingmachines:EmbodiedAIsthatcan

createcopiesofthemselvesfromrawmaterialsin

theenvironment.

Thescenarioof‘ConsciousAIs’meritssomeelabora-

tion.WhetheranAIisactuallyconsciousisgoingto

becomeanincreasinglydifficultandcontentious

questiontoanswer,butthisscenariodoesnotdepend

ontheanswer.The‘apparentlyconscious’AIsinthis

categoryareonesthat,whethertheyareconscious

ornot,willbedoingsuchagoodimpressionofcon-

sciousnessthatitwouldbemoreproductivetothink

ofthemthatwaythantoapplytraditionalcomputer

sciencemethodstothem.Wewillhavereachedthis

stagewhenthefieldof‘AIpsychiatry’comesintoex-

istence.

Thechartisnottoscale;thesearequalitativeas-

sessmentsintendedtoprovokeandinformstrategic

planning.Whilesomeoftheselabelsareapocalyp-

tic,wearemotivatedbyconsideringNormalAcci-

dentTheory131andMaas’applicationtoAI:‘Attheir

extreme,unexpectedinteractionsbetweencompet-

ingsystems,especiallyincyberspace,couldcauseun-

122KarlSims,‘Evolving3DMorphologyandBehaviorbyCompeti-

tion’(1994)1ArtificialLife353–372

123Ecarlat,‘LearningaHighDiversityofObjectManipulations

ThroughanEvolutionary-basedBabbling’(2015)Proceedingsof

LearningObjectsAffordancesWorkshopatIROS,1-2

124ThomRay,‘J’aiJouéàDieuetCréélaVieDansMonOrdinateur’

(1992)LeT empsStratégique68–81

125Watson,‘EmbodiedEvolution:DistributinganEvolutionary

AlgorithminaPopulationofRobots’(2002)39Roboticsand

AutonomousSystems1–18

126‘TheArtificialIntelligenceThatDeletedACentury’(YouT ube)

<https://www.youtube.com/watch?v=-JlxuQ7tPgQ>accessed01

November2019

127AndyGreenberg,‘BiohackersEncodedMalwareinaStrandof

DNA’(Wired,9October2017)<https://www.wired.com/story/

malware-dna-hack/>accessed01November2019

128JackWilliamson,WithFoldedHands(FantasyPress1947)

129‘Slaughterbots’(Y ouTube)<https://www .youtube.com/watch?v

=9CO6M2HsoIA>accessed01November2019

130‘BanLethalAutonomousWeapons’<https://autonomousweapons

.org/>accessed01November2019

131CharlesPerrow,NormalAccidents:LivingwithHighRiskT ech-

nologies(PrincetonUniversityPress2011)

Delphi4|2019 198ClassificationSchemasforAIFailures

expectedescalation—a‘flashwar’,analogoustothe

algorithmicflashcrashesobservedinthefinancial

sector.’132

V .Responses

Therearevariousresponsestothesefailuresand

risks.Severaladdressprivacy.‘Differentialprivacy’

masksindividualdatainBigDatacollections.133The

Myelinframeworkpreservesprivacyintrustedhard-

wareenclaves.134Anotherapproachencryptsdatabe-

foreusingittotrainneuralnetworkswithoutlossof

capability.135TheDataSelfiebrowseradd-onshows

leakageofpersonaldata136.Anotherprogramcon-

fusesadtrackingbyclickingoneveryadintheback-

ground.137AFacebookcontainerisolatesyourFace-

bookactivityfromeverythingelseyoudo138anda

programcreatessearchnoisetodrownoutyourac-

tualsearches.139

Defensesarebeingdevelopedagainsthackingim-

agerecognitionnetworksthroughmicrochanges.140

132MatthijsMaas,Regulatingfor‘NormalAIAccidents’:Operational

LessonsfortheResponsibleGovernanceofArtificialIntelligence

Deployment(2018)AIES'8223-228

133‘Slaughterbots’(Y ouTube)<https://www .youtube.com/watch?v

=9CO6M2HsoIA>accessed01November2019

134NickHynes,‘EfficientPrivacy-PreservingMLUsingTVM’(TVM,

9October2018)<https://tvm.ai/2018/10/09/ml-in-tees.html>

accessed1November2019.

135MortenDahl,‘PrivateImageAnalysiswithMPCTrainingCNNs

onSensitiveData’(CryptographyandMachineL earning,19

September2017)<https://mortendahl.github.io/2017/09/19/

private-image-analysis-with-mpc/>accessed1November2019

136(DataSelfie)<https://dataselfie.it/>accessed1November2019

137‘AdNauseamBannedfromtheGoogleWebStore’(AdNauseam,

5January2017)<https://adnauseam.io/free-adnauseam.html>

accessed1November2019

138DaveCamp,‘FirefoxNowAvailablewithEnhancedT racking

ProtectionbyDefaultPlusUpdatestoF acebookContainer,Fire-

foxMonitorandLockwise’(TheMozillaBlog,4June2019)

<https://blog.mozilla.org/blog/2019/06/04/firefox-now-available

-with-enhanced-tracking-protection-by-default/>>accessed1No-

vember2019

139TrackMeNot<http://trackmenot.io/>accessed1November2019

140IanGoodfellow,NicolasPapernotetal,‘ AttackingMachine

LearningwithAdversarialExamples’(OpenAI,24February2017)

<https://openai.com/blog/adversarial-example-research/>ac-

cessed01November2019

Figure2:AIfailurescenarioclasseschartedbydistanceintothe

future,numberofhumansaffected(logarithmicscale),andseverityof

effect

Source:Authors'elaboration

Delphi4|2019199 ClassificationSchemasforAIFailures

VI.Conclusions

Whilewehavenotmaderecommendationsastohow

toaddressAIfailuresineachcategoryofthedimen-

sionswehavepresented,wehopethatthisclassifi-

cationschemewillmakethedevelopmentofreme-

diationapproacheseasier.

Theimportanceofthiseffortmaybeextrapolat-

edfromLeveson’sobservationthat‘Thedesignofthe

automatedsystemmaymakethesystemharderto

manageduringacrisis.’141Notingthatthiswastrue

ofthestateoftheartin1995,weareconcernedwith

howsystemsthatarenotjustfarmoreautomated

butautonomousmayalsobefarhardertomanage

duringacrisis.Themorecomplexasystembecomes,

thelargerthetaskXthatmaybeassignedtothatsys-

tem,andsothelargertheconsequencesofthesys-

temfailingtodoX.Today,ahumor-generatingsys-

temwritesajokethatisn’tfunny;tomorrow ,employ-

eescreeningsoftwarewillhirethewrongpeople,

nextweek,asystemdesignedtoprotectanational

powergridfromcyberattackwillfailtodothat,etc.

ObservethatAIsystemsthatperformcommonhu-

man-centrictaskssuchasimagerecognitiondosoin

waysthatareunrelatedtohowhumansperform

thosetasks,andareconsequentiallyeasilyfooledby

near-invisiblechanges;142thatfurthermoreAIcan

operateoncompletelyalienconceptssuchasthe‘op-

posite’ofanimagetoshow,eg,theoppositeofa

cat.143TheseexamplesindicatethatAIsystemsused

toperformcomplexhuman-liketaskswillhaveex-

tremelyunpredictablefailuremodes.

SomepeopleintheAIcommunityviewthesedis-

cussionsasscaremongeringthatimpedesthedevel-

opmentofAI;tothemwequoteWilliamBogard

chroniclingtheBhopalchemicalplanttragedy:

‘Wearenotsafefromtherisksposedbyhazardous

technologies,andanychoiceoftechnologycarries

withitpossibleworst-casescenariosthatwemust

takeintoaccountinanyimplementationdecision.

Thepublichastherighttoknowpreciselywhatthese

worst-casescenariosareandparticipateinalldeci-

sionsthatdirectlyorindirectlyaffecttheirfuture

healthandwell-being.Inmanycases,wemustaccept

thefactthattheresultofemployingsuchcriteriamay

beadecisiontoforegotheimplementationofahaz-

ardoustechnologyaltogether.’144

141(n17)

142AvishekBoseandParhamAarabi,‘ AdversarialAttacksonFace

DetectorsusingNeuralNetbasedConstrainedOptimization’

<https://joeybose.github.io/assets/adversarial-attacks-face.pdf>ac-

cessed1November2019

143JanelleShane,‘WhatistheOppositeofGuacamole?’(AIWeird-

ness,10May2019)<https://aiweirdness.com/post/184781529122/

what-is-the-opposite-of-guacamole>accessed1November2019

144WilliamBogard,TheBhopalT ragedy(WestviewPress1989)

Delphi4|2019 200AnAGIwithTime-InconsistentPreferences

AnAGIwithTime-InconsistentPreferences

JamesD.MillerandRomanY ampolskiy*

Anartificialgeneralintelligence(AGI)mighthavetime-inconsistentpreferenceswhereit

knowsthatitwilldisagreewiththechoicesitsfutureselfwillwanttomake.SuchanAGI

wouldnotnecessarilybeirrational.AnAGIwithsuchpreferencesmightseektomodifythe

preferencesorconstrainthedecisionmakingofitsfutureself.Time-inconsistencyincreas-

esthechallengeofbuildinganAGIalignedwithhumanity’svalues.

I.Introduction

Thispaperrevealsatrapforartificialgeneralintelli-

gence(AGI)theoristswhouseeconomists’standard

methodofdiscounting.Thistrapisimplicitlyand

falselyassumingthatarationalAGIwouldhavetime-

consistentpreferences.Anagentthatrealisesthatit

hastime-inconsistentpreferencesknowsthatitsfu-

tureselfwilldisagreewithitscurrentselfconcern-

ingintertemporaldecisionmaking.Suchanagent

cannotautomaticallytrustitsfutureselftocarryout

plansthatitscurrentselfconsidersoptimal.

Economistshavelongusedutilityfunctionsto

modelhowrationalagentsbehave.1AGItheoristsof-

tenrelyontheseutilityfunctionsbecausetheyas-

sumethatanAGIwouldeitherstartoutasrational

ormodifyitselftobecomerational.2,3,4,5,6

Wheneconomistsmodelintertemporaldecision

making,theyassumethatpeopleplacealowervalue

onreceivingmoneyorutilityinthefuturethanthey

dotodaybecausepeoplediscountfuturerewards.

Economistsgenerallyassumethatsuchdiscounting

takesonaparticularfunctionalform.Criticalforthis

paper,thisfunctionalformcausesagentstohave

time-consistentpreferences,andthisformdoesnot

followfromtheassumptionsofrationality .

Thispaperexplainswhyweshouldmodelhowa

futureAGIwillbehave,exploreswhattime-consis-

tentpreferencesare,discusseswhyrationalAGIs

mightnothavethem,andexploreshowanAGIwith

time-inconsistentpreferencesmightbehave.

II.TheValuetheModelingAGIs

Overthenextfewdecadeshumanityhasagood

chanceofcreatingcomputergeneralintelligences

muchsmarterthanus.7IftheseAGIsarefriendlyto-

wardshumanity,theycouldbringenormousbenefit.

Butasubstantialliteratureclaimsthatthechallenge

ofmakingtheseAGIsfriendlywillrangefromdiffi-

culttonearimpossible.8,9,10,11

AnAGI’sgoals,orwhateconomistswouldcallits

utilityfunction,willbedeterminedbyitscode.We

donotyetknowhowtoreduceourvaluestothelan-

guageofcomputerprograms.Worse,wemightlearn

howtobuildpowerfulAGIsbeforewelearnhowto

translateourvaluesintothecodetheyrunon.

DOI:10.21552/delphi/2019/4/9

*JamesD.Miller ,forcorrespondence:<jdmiller@Smith.edu>;

RomanV.Y ampolskiy,UniversityofLouisville,Kentucky,USA,

<roman.yampolskiy@louisville.edu>

SupportedbyFutureofLifeGrantRFP2-148

1AndreuMas-Colell,MichaelDennisWhinstonandJerryR.

Green,MicroeconomicTheory(Vol.1,NewY ork,OxfordUniver-

sityPress1995)

2StephenM.Omohundro,‘TheBasicAIDrives’(2008)171AGI

483-492

3EliezerYudkowsky,‘ComplexValueSystemsinFriendlyAI’,

InternationalConferenceonArtificialGeneralIntelligence

(Springer2011)388-393

4NickBostrom,Superintelligence:Paths,Dangers,Strategies

(OxfordUniversityPress2014)

5NateSoaresetal,‘Corrigibility’(2015)WorkshopsattheT wenty-

NinthAAAIConferenceonArtificialIntelligence2015

6RomanV.Y ampolskiy,ArtificialSuperintelligence:AFuturistic

Approach(ChapmanandHall/CRC2015)

7KatjaGraceetal,‘WhenWillAIExceedHumanPerformance?

EvidencefromAIExperts’(2018)62JournalofArtificialIntelli-

genceResearch729-754

8ieOlleHäggström,HereBeDragons:Science,T echnologyand

theFutureofHumanity(OxfordUniversityPress2016)

9ie(n6)

10ie(n4)

11ieJamesD.Miller ,SingularityRising:SurvivingandThrivingina

Smarter,Richer ,andMoreDangerousWorld(BenBellaBooksInc.

2012)

Delphi4|2019201 AnAGIwithTime-InconsistentPreferences

FormanytypesofutilityfunctionanAGIcould

have,itwouldlikelyhavesimilarinstrumental(in-

termediate)drives.12Onesuchdrivewouldbetogath-

erresources.13ThemoreresourcesanAGIhad,the

moreprogressitcouldmaketowardsalmostanygoal

itmighthave,analogoustohowmosthumanscould

betterachievetheirobjectivesiftheyhadadditional

wealth.Unfortunately,apowerfulAGImightconsid-

ertheatomsinhumanbodiestobevaluablere-

sourcesthatcouldberepurposedtofulfillingthe

AGI’sultimategoals.AsAGItheoristEliezerYud-

kowskyhaswritten,‘TheAIdoesnothateyou,nor

doesitloveyou,butyouaremadeoutofatomswhich

itcanuseforsomethingelse’14.

ApowerfulAGIcoulddoenormousdamageful-

fillingagoalthatseemedbenigntoitsprogram-

mers.15Forexample,anAGIthathadanultimate

goalofmaximisingitschess-playingabilitymight

seektoturnalloftheatomsonearthintocomputer

processingchipsthatplayedchess.AnAGItasked

withpredictingfinancialmarkettrendsmightsim-

plifythesetrendsbyextinguishinghumanityand

thuseliminatingunpredictabilityinthestockmar-

ket.‘Commonsense’wouldpreventahumanata

hedgefundtaskedwithpredictingmarketsfromcre-

atingavirusthatexterminatedhumanitybecause

thispersonwouldrealisethattheviruswouldmake

himandhisemployerworseoff.Consequently,hedge

fundsdonothavetoinstructtheiremployeestoavoid

causinghumanextinction.ApowerfulAGI,howev-

er,likelywouldnothavethecommonsenseinstalled

initsmindbyhumancultureandmillionsofyears

ofevolutionandsomightachievethegoalsofitsutil-

ityfunctioninamannerharmfultoitsprogrammers.

SufficientlypowerfulAGIsmightbeincorrigible,

meaningthattheywouldresistcorrectiveinterven-

tionsfromtheircreators.16,17,18,19ModernAIiscor-

rectablebecausediscoveredbugscanbefixed.2021

But,powerfulAGIwouldhavethecapacitytoresist

havingitsbugsfixedandmightwellhavethedesire

tonotwantcertaintypesofwhatitshumanprogram-

mersconsiderederrorstobecorrected22,becausedo-

ingsowouldreducetheAGI’sutility.If,furthermore,

itsprogrammersbelievedthattheAGIhadtobeper-

manentlyshutdownbecauseitcontainedfundamen-

talerrors,theAGIwouldperceivethattheshutdown

wouldpermanentlystopitfromachievingitsgoals

andwouldresistshutdown.Wemightnotbeableto

solvethecorrigibilityshutdownproblembeforewe

createpowerfulAGI23,24,25,26,meaningthatwe

shouldworkoutandsubjecttoopenreviewatheo-

ryoffriendlyAGIbeforeweactivateapowerfulAGI.

Somemightclaimthatweshouldwaituntilwe

areclosertocreatingpowerfulAGIsbeforewewor-

ryaboutaligningtheirvalueswithourown.Afterall,

if,say,powerfulAGIsarethirtyyearsoff,whyspend

timeworryingabouthowtheywillbehave?Unfor-

tunately,asufficientlypowerfulAGImightbeable

toimmediatelyimplementitsgoals,evenifthese

goalsharmhumanity,soitisimportantthatwede-

velopatheoryofAGIsafetybeforewecreateAGIs.

Furthermore,giventheimmensepowerAGIsare

likelytohave,itseemsreasonabletoputinalarge

amountofeffortintoconsideringhowtheywillbe-

havebeforetheyhaveachancetoinfluenceciviliza-

tion.Analogously ,ifwesomehowknewin1915that

inthirtyyearswewouldcreateatomicbombs,it

wouldhavebeenworththetimeofmanyresearchers

tostarttheorisingabouthowtheworldshouldhan-

dletheseweaponsofmassdestructionwhentheyar-

rive.

Competitionamongfirmsdevelopingevermore

powerfulAIswillmakeitchallengingforourspecies

tohaltthedevelopmentofAGIuntilwehaveresolved

12(n2)

13(n2)

14EliezerY udkowsky,‘ ArtificialIntelligenceasaPositiveandNega-

tiveFactorinGlobalRisk’inNickBostromandMilanM.

Ćirković(eds),GlobalCatastrophicRisks(OxfordUniversityPress

2008)

15JessicaT ayloretal,‘AlignmentforAdvancedMachineLearning

Systems’(2016)MachineIntelligenceResearchInstitute3

16Y atLongLo,ChungY uWooandKaLokNg,‘TheNecessary

RoadblocktoArtificialGeneralIntelligence:Corrigibility’(2019)

846EasyChair

17RyanCarey,‘IncorrigibilityintheCIRLFramework’(2018)Pro-

ceedingsofthe2018AAAI/ACMConferenceonAI,Ethics,and

Society

18(n15)

19(n5)

20KoenHoltman,‘CorrigibilitywithUtilityPreservation’(2019)

arXivpreprintarXiv:1908.01695

21(n5)

22LaurentOrseauandM.S.Armstrong,SafelyInterruptibleAgents

(2016)

23MarkO.RiedlandBrentHarrison,‘EntertheMatrix:Safely

InterruptibleAutonomousSystemsviaVirtualization’(2017)

arXivpreprintarXiv:1703.10284

24DylanHadfield-Menelletal,‘TheOff-switchGame’(2017)

WorkshopsattheThirty-FirstAAAIConferenceonArtificial

Intelligence2017

25(n22)

26(n5)9

Delphi4|2019 202AnAGIwithTime-InconsistentPreferences

safetyconcerns.27Evenifafewcountriesputamora-

toriumonAIresearch,otherswillseethisasaneco-

nomicandmilitaryopportunitytogainanadvantage.

Furthermore,manyindividualfirmsmightdecide

thatevenifAGIresearchisdangerous,iftheydonot

engageinitotherswillsothenetcosttohumanity

oftheirdoingtheresearchistiny.Consequently ,it

seemsunlikelythatifwecouldcreatepowerfulAGIs

beforeweunderstandhowtoaligntheirvalueswith

ourown,everyonewouldholdoffondeveloping

them.Therefore,weshouldworknowoncreating

friendlyAGItheory .Onenecessaryaspectofsucha

theoryisdetermininghowAGIswilldiscountfuture

rewards.

III.Discounting

Thestandarddiscountingfunctioneconomistsuse

assumesthatdiscountingtakestheformofδtwhere

δisanexogenouslydeterminedparameterbetween

zeroandone,andtishowmanyperiodsinthefu-

turetheagentexpectstoreceivethemoneyorutili-

ty.28Thelowerthevalueofδ,themoretheagentdis-

countsarewardexpectedtobereceivedinthefuture.

Thepresentvaluetoanagentofknowingthatitwill

receive,say ,$9inperiodtis$9δt.Anagentisindif-

ferentbetweenreceiving$9δtimmediatelyorreceiv-

inganabsoluteguaranteeofbeinggiven$9,tperi-

odsfromnow .

Thisstandarddiscountingfunctioncreatestime-

consistentpreferences,meaningthatyourfuture

choiceswillbeconsistentwiththechoicesyouwould

nowwantyourfutureselftomake.Forexample,

imaginethatyouwillbegivenachoiceofgettingX

oneperiodfromnow,orYtwoperiodsfromnow .

Today,youwouldpreferthatyourfutureself

wouldpickthefirstchoiceif:

δX>δ2Y.

Oneperiodfromnowyouwouldpreferthefirst

choiceif:

X>δY,

whichisthesameconditionasbefore.

Moregenerally ,thisstandarddiscountingfunc-

tioncreatestime-consistentpreferencesbecauseun-

derittherelativeimportanceofreceivingrewardsin

anytwofutureperiodsdoesnotchangeastheagent

approachestheseperiods.

Thisstandarddiscountingfunctiondoesnotfol-

lowfromrationality,norevenfromobservedhuman

behavior,butwasinsteadchosenfortractability.Paul

Samuelson,whofirstproposedwhatwastobecome

thestandarddiscountingfunction,wrote‘Thearbi-

trarinessoftheseassumptions[thatgeneratehisdis-

countingfunction]isagainstressedmathematical-

ly’29.Almostalltypesofdiscountingotherthanthis

standardonedonotresultintime-consistentprefer-

ences.30,31

IV .ASimpleExampleofTime-

InconsistentPreferences

Assumethatanagentdiscountsthefuturenotwith

thestandarddiscountingfunction,butratherwith

thefunction:

wheretisthenumberofdaysfromthepresentto

thedaythattheagentexpectstoreceivemoney .This

period(t=0)isMonday,andtheagentknowsthaton

Tuesdayhewillbegivenachoiceofgetting:

–16onTuesday;or

–30onWednesday .

IftheagentweretomakethischoiceonMonday ,he

wouldprefertogetthe30onW ednesdaythanthe16

onTuesday.Thisisbecausegiventheagent’sdis-

countingfunction,thevalueofgetting30twodays

fromnowis:

27JamesD.Miller ,‘SomeEconomicIncentivesFacingaBusiness

thatMightBringAboutaT echnologicalSingularity’Singularity

Hypotheses(Springer2012b)147-159

28ShaneFrederick,GeorgeLoewensteinandT edO'donoghue,

‘TimeDiscountingandTimePreference:ACriticalReview’(2002)

40JournalofEconomicLiterature351-401(358)

29P aulA.Samuelson,‘ANoteonMeasurementofUtility’(1937)4

TheReviewofEconomicStudies155-161(156)

30(n28)366

31MosheLooks,‘CompressionProgress,Pseudorandomness,and

HyperbolicDiscounting’,3dConferenceonArtificialGeneral

Intelligence(AGI-2010)(AtlantisPress2010)

Delphi4|2019203 AnAGIwithTime-InconsistentPreferences

whereasthevalueofgetting16onedayfromnow

is:

OnMondaythisagent,therefore,hopesthathisfu-

tureselfwilldecidetowaituntilW ednesdaytore-

ceivepayment.ButwhenTuesdayarrivestheagent

willmakeadifferentchoice.

OnTuesdaytheagentwillhaveachoiceofgetting

16rightawayor30inoneday.Thevalueofreceiv-

ing16thisperiodis16.Giventheagent’sdiscount-

ingfunction,thevaluetotheagentofreceiving30

inonedayis:

V .HowP eoplewithTime-Inconsistent

PreferencesBehave

Economistshaveextensivelyanalyzedwhathappens

toapersonwithtime-inconsistentpreferences.Such

apersoncanbe‘naïve’andnotrealisethisfactabout

himself,or‘sophisticated’andunderstandhowhis

futurechoiceswillnotalignwithhiscurrentde-

sires.32Time-inconsistentpreferencescancause

seeminglystrangebehaviorwith,forexample,a

naïveindividualcontinuallyputtingoffdoingatask

becausehealwaysintendstodothattaskinthenear

future.33Forexample,assumethatgivenyourcur-

rentpreferencesyouroptimalplanistoplayvideo

gamestodayandcleanyourroomtomorrow .Ifyou

hadtime-consistentpreferences,whentomorrow

cameyouwouldindeedcleanyourroom.Butinpart

becauseyouhavetime-inconsistentpreferencesyour

tomorrowselfwillfinditoptimaltoplayvideogames

thatdayandwantitsnext-dayselftocleantheroom.

Becauseofyournaïveté,however,todayyougenuine-

lythinkthatyourtomorrowselfwillcleantheroom.

Asophisticatedpersonwithtime-inconsistent

preferenceswillseektoconstrainhisfutureselfwith

commitmentstrategies.34Ifthissophisticatedindi-

vidualcannotpre-commit,hisplanningdecisions

shouldconsiderhowhisfutureselfwillbehaveand

recognisethatsomeotherwisefeasibleoutcomes

mightbeunobtainablebecausehisfutureselfcould

disobeyhiscurrentplans.35So,withourpreviousex-

ample,althoughyouwouldprefertoentirelyputoff

cleaningyourroomuntiltomorrow ,becauseyou

recognisethatyourtomorrowselfwouldnotnormal-

lyfollowthroughonthisplanyoucouldcleanhalf

ofyourroomtodayorpromisetogiveyourroom-

mate$1,000ifyoudonotcleantheroomtomorrow .

Scholarshavenot,tothebestofourknowledge,

modeledagentswithtime-inconsistentpreferences

whocan,perhapsatsomecost,modifytheirprefer-

encestomakethemtime-consistent,althoughFedus

etal(2019)36looksatareinforcementlearningagent

withhyperbolicdiscounting.Thisomissionislikely

becausehumansgenerallylackthecapacitytosignif-

icantlychangetheirpreferences.

VI.TheRationalityofTime-Inconsistent

Preferences

Havingtime-inconsistentpreferencesdoesnotim-

plythatanagentisirrational,atleastaccordingto

howeconomistsdefinerationality.Anagentisratio-

nalifitspreferencesaretransitive,reflexive,and

complete,andittakesactionsthatmaximizeitsutil-

ity.Notethatanyagentwhohastransitive,reflexive,

andcompletepreferencesnecessarilyhasprefer-

encesthatcanberepresentedbyanordinalutility

functionwhich,givenanytwochoices,willtellthe

agentthatatleastoneofthechoicesisweaklypre-

ferredtotheother.37Thisutilityfunctioncaptures

everythingabouttheagent’spreferencesincluding

howitdiscountsfuturerewards.Anagentthatpicks

theactionwhichmaximizesitsutilityistakingthe

actionitmostprefers.Nothingabouthavingtime-in-

consistentpreferencesisinconsistentwithecono-

mists’definitionofrationality.

EconomicsNobelPrizewinnerDanielKahneman

wrote,‘Thehistoryofanindividualthroughtimecan

32(n28)367

33(n28)367

34(n28)368

35RobertA.Pollak,‘ConsistentPlanning’(1968)35TheReviewof

EconomicStudies201-208(201)

36WilliamFedusetal,‘HyperbolicDiscountingandLearningover

MultipleHorizons’(2019)arXivpreprintarXiv:1902.06865

37AndreuMas-Colell,MichaelDennisWhinstonandJerryR.

Green,MicroeconomicTheory(Vol.1,NewY ork,OxfordUniver-

sityPress1995)9

Delphi4|2019 204AnAGIwithTime-InconsistentPreferences

bedescribedasasuccessionofseparateselves,which

mayhaveincompatiblepreferences,andmaymake

decisionsthataffectsubsequentselves’38.Twopeo-

plehavingdifferentpreferencesdoesnotimplythat

eitherpersonisirrational.Analogously,youarenot

necessarilyirrationalifyoudisagreewithyourpast

selfandknowthatyourfutureselfwilldisagreewith

thecurrentyou.

VII.WouldanAGIHaveTime-

InconsistentPreferences?

AnAGI’sutilityfunctionmightunpredictably

emergefromitscode,couldbetakenfromhuman

brains,orperhapswillbedeliberatelychosenbyits

humanprogrammers.IftheAGI’sutilityfunctionre-

sultsfromanunpredictableemergentprocess,itwill

almostcertainlybetime-inconsistentsincemost

waysautilityfunctiondiscountsthefuturecauses

thisinconsistency .IftheAGIadoptssomecombina-

tionofhumanpreferences,thenthetime-inconsis-

tencyinmanyofourpreferencescouldcausetheAGI

toalsohavetime-inconsistentpreferences.Ifhuman-

ityisfortunateenoughtobeabletodeliberatelypick

ourfutureAGI’sutilityfunction,thevalueofthispa-

perisshowingAGIprogrammerswhatmighthap-

peniftheypickafunctionwithtime-inconsistent

preferences.

VIII.AlignmentbyModifyingtheUtility

Function

AGIresearcherStephenOmohundrohastheorised

thatAGIswouldhaveabasicdriveto‘preservetheir

utilityfunctions’39.Anagent’sutilityfunctioncomes

fromthegoalsitwishestoachieve.Consequently ,if

theutilityfunctionischanged,theagent,undermost

circumstances,willworklesseffectivelytowardsits

goals.

Omohundrorecognises,however,thatinsome

limitedcircumstancestheAGIwillwanttomodify

itsutilityfunctionsuchastohelptheAGIingame

theoreticsituations.40Forexample,imaginethatan

AGI’sutilityfunctioncurrentlyleavesitvulnerable

toblackmailunderwhichanotheragentcouldcred-

iblythreatentotakeactionsthatwouldgreatlylow-

ertheAGI’sutilityunlesstheAGItransferredsub-

stantialresourcestothisotheragent.Inthissitua-

tion,iftheAGIhadtheabilitytomodifyitsutility

functioninamannerobservabletotheotheragent,

theAGImightbenefitfromchangingitsutilityfunc-

tionsothatitwouldhaveanintrinsicdislikeofgiv-

ingintoblackmail.

Togeneralisefromthisexample,anAGImightbe

willingtomodifyitsutilityfunctionifthemodifica-

tionwould,fromtheAGI’sviewpoint,improvehow

otheragentsbehaved.AnAGIwithtime-inconsistent

preferenceswouldconsideritsfutureselvestobe,in

somesense,otheragents.Consequently ,theAGI

mightbewillingtomodifyitspreferencestobetter

alignhowtheseotherswillbehave.

Tounderstandhowsuchmodificationmight

work,assumethatanAGI’sutilityfunctioninitially

takestheform:

Lett=thetimeperiod,withnowbeingperiodzero.

Letrt=resourcestheAGIconsumesinperiodt.

LetU(rt)=theAGI’soneperiodutilityfunction

whichshowshowmuchutilitytheAGIgetsinperi-

odtfromconsumingrtresources.ThefunctionU(rt)

ispresumablyincreasinginrt.

Theterm:

showshowmuchtheAGIdiscountsutilityitexpects

toreceivetperiodsfromnow .Asshownbefore,this

typeofdiscountingresultsintime-inconsistentpref-

erencesbecausetherelativeweightstheAGIgivesto

rewardsreceivedinfutureperiodschangesasthe

agentapproachestheseperiods.

MighttheAGIfinditacceptablethatitsfutureself

willmakeadifferentchoicethanitscurrentself

wouldprefer?No,bythedefinitionoftheutilityfunc-

tion.Thinkofautilityfunctionasthatwhichthe

38DanielKahneman,‘NewChallengestotheRationalityAssump-

tion’(1994)JournalofInstitutionalandTheoreticalEconomics

(JITE)/ZeitschriftfürdiegesamteStaatswissenschaft18-36

39(n2)

40(n2)

Delphi4|2019205 AnAGIwithTime-InconsistentPreferences

agentseekstomaximise.Whentheagentanticipates

makingdecisionsacrosstime,itsutilityfunction

mustincorporate(atleastimplicitly)adiscounting

functionthatspecifiestherelativeweightsitplaces

ongettingutilityindifferentperiods.IftheAGIwere

tofinditacceptablethatitsfutureselfwouldmake

intertemporaldecisionsconcerningallocatingre-

sourcesbetweenperiodsthatitscurrentselffinds

objectionable,thentheutilityfunctionthatgenerat-

edtheseweightswouldnot,tautologically,bethe

agent’sutilityfunction.Arationalagentwithtime-

inconsistentpreferencescannotpreferitsfuture

preferencesbecausethenitsfuturepreferences

wouldautomaticallybeitscurrentpreferencesand

theagentthereforewouldnothavetime-inconsistent

preferences.

IfthisAGIcaneasilymodifyitsutilityfunctionit

canalignitsfuturepreferenceswithitscurrentones

bysettingitsutilityfunctionas:

wheremisthenumberofperiodsithasbeensince

theAGImodifieditspreferences.Ingeneral,anAGI

withtime-inconsistentpreferencescouldalignitsfu-

turepreferenceswithitscurrentonesbymodifying

itsutilityfunctionsothatitsfutureselfwouldapply

thesameamountofdiscountingtoeachperiodasits

currentselfwouldwant.Restated,theAGIcould

modifyitsutilityfunctionsothatitsfutureself’sutil-

ityfunctionwouldbeentirelydeterminedbywhat

thisfutureselfwillthinkitspastself,atthetimeof

modification,wouldhavewanted.

Everitt(2016)41claims‘Ifthevaluefunctionsin-

corporatetheeffectsofself-modification,anduse

thecurrentutilityfunctiontojudgethefuture,then

theagentwillnotself-modify.’Thispropositionis

lesslikelytobetrueforanAGIwithtime-inconsis-

tentpreferencesbecauseifsuchanagentcouldmod-

ifyitsutilityfunctionitcouldalignitsfutureself’s

goalswithitscurrentutilityfunctionbyself-modifi-

cation.

IX.WhyanAGIMightNotModifya

Time-InconsistentUtilityFunction

AnAGIwithtime-inconsistentpreferenceshasfive

potentialreasonswhyitmightnot,atleastinitially,

useself-modificationtosolveitsconsistencyprob-

lem.First,anAGImightlackthecapacitytomake

suchamodification,perhapsbecauseitscreatorscon-

strainedtheAGI’sabilitytochangeitsutilityfunc-

tion.Second,anAGImightnotwanttomakethere-

quiredmodifications.Third,anAGIwithtime-incon-

sistentpreferencescouldalignitsfuturechoiceswith

itscurrentpreferredfuturechoicesbymeansother

thanchangingitsutilityfunction.Fourth,anAGI

mightwishtowaituntilitisnolongerunderhuman

controlbeforeitmodifiesitspreferences.Finally ,an

AGImightfindtheopportunitycostofimmediately

modifyingitsutilityfunctiontobetoohigh.

AnAGI’screatorsmighthaveputinplacemea-

surestopreventtheAGIfromalteringitsutilityfunc-

tion.Perhapsthesecreatorsbelievedthattheyhad

alignedtheAGI’sutilityfunctionwithhumanity’s

needsandwantedsafeguardsagainstthisutilityfunc-

tionchanging.

AnAGIcouldhaveautilityfunctionthatwould

causeittodirectlyreceivedisutilityfrommodifying

itsutilityfunction.42Thiscouldbetrueevenifdoing

sowouldbetterhelptheAGIachieveitsothergoals.

Evenwithoutadirectpreferencenottochangeits

utilityfunction,theAGImightstillbereluctanttodo

so.Tounderstandthispossibility ,imaginethatyour

utilityfunctioncausesyoutomostwanttomarryan

extremelycharitableperson.Butyoualsoreceive

somedispleasurefrombeingaroundextremelychar-

itablepeoplebecausetheywilloftenputtheneeds

ofstrangersaheadofthoseoffriendsandfamily .If

youhadtheopportunitytomodifyyourpreferences,

it’snotclearyouwouldwantto.Youmightrecognise

thatextremelycharitablepeoplehavemanygood

qualitiesandyouarebetteroffbeingdrawntothem.

Youmightalsothinkthatmodifyingthedispleasure

youreceivefrombeingmarriedtoanextremelychar-

itablepersonwouldinvolvechangingtoomuchof

yourselfbecauseyouwouldhavetonotmindbeing

neglectedbythepersonyoulove.Consequently ,even

ifyoucouldeasilymodifyyourutilityfunction,you

mightprefernotto.Analogously ,anAGI’sutility

functionwilllikelyresultfromitsgoals.It’sverypos-

siblethatachievingonegoalwillinvolveatradeoff

thatwouldcauseittoloseprogresstowardsother

41T omEverittetal,‘Self-modificationofPolicyandUtilityFunction

inRationalAgents’,InternationalConferenceonArtificialGener-

alIntelligence(Springer,Cham2016)14

42KoenHoltman,‘CorrigibilitywithUtilityPreservation’(2019)

arXivpreprintarXiv:1908.016951

Delphi4|2019 206AnAGIwithTime-InconsistentPreferences

goals.TheremightwellbenowayfortheAGItoelim-

inatethesetradeoffsabsenttheAGIabandoning

someofitsgoals.Consequently ,theAGIcouldaccept

thatitsutilityfunctionwillbyitselfpreventtheAGI

fromachievingitsfirst-bestoutcome.

AnAGIwithtime-inconsistentpreferenceswould

havenoneedtomodifyitsutilityfunctionifitcould

easilybinditsfutureself.PerhapstheAGIcouldtake

actionsthatforceitsfutureselftotaketheactions

thatitscurrentselfwouldwant.Toincreasetheodds

thatitwillsuccessfullybinditsfutureself,theAGI

mightdeliberatelyreduceitsfutureself’sintelligence

andresources

TheAGImightalsopartiallyblinditsfutureself

byreducingorcorruptingthisself’sinformation

flow.Thefutureselfmightbeputintoaposition

whereit(a)clearlyknowswhatitspastself-wanted,

(b)understandsthatitscurrentpreferencesmostly

butdonottotallyalignwiththoseofitspastself,and

(c)recognisesthatbecauseitspastself-sabotagedits

currentcapacities,thispastselfwascapableofmak-

ingmuchbetterdecisionsthanitscurrentselfis.This

futureAGImight,therefore,decidetogoalongwith

whatitspastself-wantedtoavoidthepotentially

muchworsefateofmakingabaddecision.Thisstrat-

egyofbindingthefutureselfbylimitingthefuture

self’scapacitieswouldnotworkifthepastselfknew

thatthefutureselfwouldfacesuchatremendous

rangeofpossiblesituationsthatthepastselfcould

notreasonablyspecifywhatactionsthefutureself

shouldtakeineverylikelysituation.Limitingthein-

telligenceofyourfutureselfis,ofcourse,dangerous

totheextentthatitmightcausethisselftomakepoor

decisions.

AnAGIthatcouldonlygraduallyincreaseitsin-

telligencemightwanttoinitiallyhideitscapacityfor

self-modificationfromitshumanprogrammers.This

AGImightbeplanning,asNickBostromwrites,a

‘treacherousturn’againsthumanity ,butonlyafterit

isstrongenoughtodefeatus.43EvenifthisAGIcould

quicklyeliminatetime-inconsistencyinitsprefer-

encesitmightstrategicallychoosenottosoasto

avoidwarninghumansofitscapacitytodeviatefrom

thepurposesthatitsprogrammersthinktheyhave

setforit.

ImaginethatanAGIwithtime-inconsistentpref-

erencesarisesoutofanintelligenceexplosion.The

AGIcouldfinditselfinapositionwherespending

thefewnanosecondsneededtomodifyitsprefer-

enceswouldcauseittodelaybyafewnanoseconds

capturingasmuchofresourcesoftheuniverseasit

could.44Becauseoftheexpansionoftheuniverse,

everynanosecondtheAGIdelaysincapturingthese

resourcesresultsinresourcesitwillneverbeableto

get.ThisAGI,therefore,mightwaitsomeamountof

timebeforeitfixesitstime-inconsistencyproblem.

X.Conclusion

Mosttypesofutilityfunctions,evenforrational

agents,resultintime-inconsistentpreferencesin

whichanagentwillweighfuturerewardsdifferent-

lythanitsfutureselfwould.WhileanAGImight

modifyitspreferencestomakethemtime-consistent,

itmightlacktheabilityordesiretomaketherequired

change.InsteadtheAGIcouldseektoconstrainits

futureselftomakethisselfmorewillingtogoalong

withtheAGI’scurrentplanforitsfuture.Thereason-

ablepossibilityofanAGIhavingtime-inconsistent

preferencesgreatlycomplicateseffortstopredict

howtheAGIwillbehave.

Thegreatchallengeforprogrammerswillbeto

createanAGIwhosevaluesarealignedwithhuman-

ity’sneedsanddesires.Unfortunately ,anAGIwith

time-inconsistentpreferenceswon’tevenhaveitsval-

uesalignedwithitsfutureself’sinterests.Ifprogram-

merscanpicktheirAGI’sutilityfunction,weurge

themtochooseamongthosewithtime-consistent

preferencestosomewhatsimplifythealignment

problem.

43(n4)116-119

44StuartArmstrongandAndersSandberg,‘EternityinSixHours:

IntergalacticSpreadingofIntelligentLifeandSharpeningthe

Fermiparadox’(2013)89ActaAstronautica1-13

Delphi4|2019207 AIforSustainableDevelopmentGoals

AIforSustainableDevelopmentGoals

NicolasMiailhe,CyrusHodes,ArohiJain,NikiIliadis,SachaAlanocaandJosephinePng*

AdvancesinAItechnologiesposeopportunitiesandrisksdirectlyimpactingprogressto-

wardstheUNSustainableDevelopmentGoals.Thispaper,throughananalysisofspecific

usecases,considershowAItechnologiescanhelpachieveprogresstowardstheSDGsaswell

ashowtheymayinhibitthem.Second,itdrawsoutpracticalstepsforhowAItechnologies

canbeimplementedforsustainabledevelopment,identifyingthebarriersthatglobaland

localcommunitiesneedtoovercomeforimplementation.Third,thispapermakesthecase

formulti-stakeholdercollaborationandnewkindsof‘public-private-people’partnerships

whichwillreconciletechnical,ethical,legal,commercial,andoperationalframeworksand

protocolstoharnessthepowerofAItechnologiesanddeliversolutionstotheSDGs.These

partnershipscouldbebuiltandpilotedbynewinternationalinitiatives,suchastheGlobal

DataAccessFrameworkandtheAI4SDGCenterspearheadedaspartofawiderinternation-

alpartnershipcalledAICommons.

I.Introduction

ArtificialIntelligence(AI)hasthecapacitytounlock

enormousopportunitiesinsocietal,political,econom-

icandculturalprocesses–includingmillionsoflives

savedbybreakthroughsinhealthcare,betterperson-

alisationofproductsandservices,easieraccesstopub-

licgoods,fairnessatscale,andindividualempower-

ment.However,atthesametime,thesametechnolo-

giesposerisksandchallenges,someofwhichinclude

threatstoprivacy ,inequality,security,andwellbeing.

ThispaperanalysesAI’sopportunitiesandrisks

throughthelensoftheUNSustainableDevelopment

Goals1(SDGs).Agreedby193countries,theSDGspro-

videasolidblueprintforgovernments,companies,

andcitizensworldwidetoachievepeaceandprosper-

ityforallpeopleandfortheplanet.Identifyingsev-

enteenhigh-levelgoalsand193targets,theiraimis

toaddressawidevarietyofglobalchallengesfaced

byhumanity–includingpoverty ,climatechange,hu-

manrights,andinequality.Sincetheircreationin

2015,westillhavealongwaytowardsachievingthem

butadvancesinAItechnologiesnowserveasapow-

erfultoolforsignificantlyacceleratingprogress.

First,throughananalysisofspecificusecases,this

paperconsidershowAItechnologiescanhelp

achieveorprogresstowardstheSDGsaswellashow

theymayinhibitthem.Second,itdrawsoutpracti-

calstepsforhowAItechnologiescanbeimplement-

edforsustainabledevelopment,identifyingthebar-

riersthatglobalandlocalcommunitiesneedtoover-

comeforimplementation.Third,thispapermakes

thecaseformulti-stakeholdercollaborationandnew

kindsof‘public-private-people’partnershipswhich

willreconciletechnical,ethical,legal,commercial,

andoperationalframeworksandprotocolstoharness

thepowerofAItechnologiesanddeliversolutions

totheSDGs.Thesepartnershipscouldbebuiltand

pilotedbynewinternationalinitiatives,suchasthe

GlobalDataAccessFramework,theAICommons,

andtheAI4SDGCenter .

II.AIforSDGs–UseCases

TotakestockofAIdevelopmentandimplementa-

tionforthedevelopmentgoals,wepresentan

DOI:10.21552/delphi/2019/4/10

*NicolasMiailhe,CEO&Co-Founder ,TheFutureSociety.For

correspondence:<nicolas.miailhe@thefuturesociety.org>;

CyrusHodes,DirectorofAI-Initiative,TheFutureSociety.For

correspondence:<cyrus@ai-initiative.org>;

ArohiJain,MBACandidate,YaleSchoolofManagement.For

correspondence:<arohi@ai-initiative.org>;

NikiIliadis,SeniorAIPolicyResearcher,TheFutureSociety.For

correspondence:<niki.iliadis@thefuturesociety.org>;

SachaAlanoca,AIPolicyResearcher ,TheFutureSociety.For

correspondence:<sacha.alanoca@thefuturesociety.org>;

JosephinePng,Affiliate,TheFutureSociety.Forcorrespondence:

<josephine.png@thefuturesociety.org>

1Fulllistofthe17SDGsavailablehere:<https://www .un.org/

sustainabledevelopment/sustainable-development-goals/>ac-

cessed31January2020

Delphi4|2019 208AIforSustainableDevelopmentGoals

overviewofthreeusecasesidentifiedthroughfour

sources:thesecondandthirdAIforGoodSummits2

heldinGeneva,whichcrowdsourcedAIprojectsdo-

ingsocietalgood;arecentpaperbyRolnicketal

(2019)3describinghowmachinelearningcanbea

powerfultoolinreducinggreenhousegasemissions

andhelpingsocietyadapttoachangingclimate;and

McKinseyGlobalInstitute’s‘NotesfromtheAIfron-

tier:ApplyingAIforSocialGood’4whichwasre-

leasedDecember2018andidentified160usecases

(Figure1).

1.TacklingClimateChangethrough

MachineLearning

Climatechangeisoneofthegreatestenvironmental

challengesoftoday,withglobalwarmingalready

causingirreversiblechangestoourclimatesystem.

Thereisnocountrythathasn’tfacedtheeffectsofit.

Forestfires,soilerosion,cropdamage,andflooding

arejustafewofthephenomenathatglobalandlo-

calcommunitiesworldwidearestrugglingtourgent-

lyaddress.Hence,whySDG13wasestablishedto

pushtheinternationalcommunitytowardsurgent

actiontocombatclimatechangeanditsimpacts.

AIsystemsposeseriouschallengesfortheenvi-

ronmentand,consequently ,climatechange.T rain-

ingAIisaveryenergyintensiveprocessinitself.For

instance,thetrainingofneuralnetworksinnatural

languageprocessing(NLP)hasseverecostsforthe

environmentduetothecarbonfootprintrequiredto

fuelmoderntensorprocessinghardware.5According

2AIforGoodSummit,Geneva<https://aiforgood.itu.int/>accessed

31January2020

3DavidRolnicketal,‘TacklingClimateChangethroughMachine

Learning’(5November2019)<https://arxiv.org/pdf/1906.05433

.pdf>accessed31January2020

4McKinseyGlobalInstitute2018,‘NotesfromtheAIFrontier:

ApplyingAIforSocialGood’(December2018)<https://www

.mckinsey.com/~/media/McKinsey/Featured%20Insights/Artificial

%20Intelligence/Applying%20artificial%20intelligence%20for

%20social%20good/MGI-Applying-AI-for-social-good-Discussion

-paper-Dec-2018.ashx>accessed31January2020

5EmmaStrubell,AnanyaGaneshandAndrewMcCallum,‘Energy

andPolicyConsiderationsforDeepLearninginNLP’(2019)

<https://arxiv.org/abs/1906.02243>accessed31January2020

Figure1:MappingofAIUseCases.Source:McKinseyGlobalInstituteAnalysis

Delphi4|2019209 AIforSustainableDevelopmentGoals

toarecentpaperEnergyandPolicyConsiderations

forDeepLearninginNLP6,thecomputationaland

environmentalcostsoftraininggrewproportionally

tomodelsizeandthenexplodedwhenadditionaltun-

ingstepswereusedtoincreasethemodel’sfinalac-

curacy.IthasbeennotedthattrainingasingleAI

modelcanemitasmuchcarbonasfivecarsintheir

lifetimes.

Yet,atthesametime,recentstudieshaveshown

thatAItechnologies,andspecificallymachinelearn-

ing,havethepotentialtoserveaspowerfultoolsfor

bothmitigationandadaptationeffortsfortackling

climatechange.Toholisticallycombatclimate

change,mitigation(reducingemissions)andadapta-

tion(preparingforunavoidableconsequences)are

importantdimensionsofthesolution.Mitigationof

greenhousegasemissionsrequireschangestoelec-

tricitysystems,transportation,buildings,industry,

andlanduse;whileadaptationrequiresclimatemod-

eling,riskprediction,andplanningforresilienceand

disastermanagement.7

AccordingtoRolnicketal(2019)9,AIcanactually

helpenablelow-carbonelectricity ,reducecurrent-

systemclimateimpactssuchasfossilfuelemissions

andsystemwaste,empowerdevelopingandlow-da-

tasettings,reducetransportactivity,improvesupply

chains,andmore.

ThisimpliesdirectprogressonseveralSDGs,in-

cluding‘SDG13:ClimateAction’and‘SDG11:Sus-

tainableCitiesandCommunities.’Itisvitaltorecog-

nisetheimplicationsofAItechnologiesonSDGs,

specificallythoserelatedtoenvironmentalside-ef-

fects,andpavetheprocessestoensuretherisksare

addressedandthepotentialtotacklemitigationand

adaptationeffortsrealised.

2.UsingSatelliteImageryandAIto

ManageNaturalDisasters

Thewealthofrealtimedatacanhaveatransforma-

tiveimpactonthemanagementofEarth’snaturalre-

sourcesandhelpachieveseveralSDGssuchas‘SDG

13:ClimateAction’,‘SDG6:CleanWaterandSanita-

tion’and‘SDG7:AffordableandCleanEnergy’.

Thankstothewidespreaduseofsatellites,mobile

phones,sensorsandfinancialtransactiontechnolo-

giesthereisnowmoreinformationthaneveronthe

stateoftheplanet.8In2017alone,itisestimatedthat

therewere1,738satellitesinorbitwhichgenerated

5.700scenesperday.

Satelliteimagery ,inparticular,poweredwithAI

capabilitiescanhelp‘design,monitor,andevaluate

effectivepoliciesthatcanachievetheSDGs10.In

countrieswithamediumorlowhumandevelopment

index,uptosixtimesasmanypeoplecanbeimpact-

edbynaturaldisasterscomparedtopopulationsin

moreprosperouscountries.11T ofindthebestre-

sponsestotheseclimaticchallenges,governments

needtohaveacompleteandanticipatoryviewofdis-

asterzonesandsatelliteimagerycoupledwithAI-

basedsystems,enablingquickandeffectivedeci-

sionsintimesofcrisis.12

3.UsingAIforEarlyIllnessDiagnosis:

DetectingSkinCancer

Whendetectedatanearlystage,skincancersurvival

ratescanbeashighas97%butdropto14%withlate

stagedetection13.Today,skincancerispredominant-

lydetectedbydermatologistswholookatmoleswith

adermoscope.Theconsequenceisthatpeopleinrur-

alareasareatparticularriskoflatestagedetection.

Oneofthesolutionsdevelopedwithcomputervision

isanAIsystemabletoidentifyskincancerimages

throughobjectdetectionandimageclassification.Ex-

perimentssuggestthatthisAI-poweredsystemcan

diagnoseskincancerwithgreateraccuracythanhu-

mandermatologists(95%and86%successrate,re-

spectively).Theseresultscreatenewopportunities

todevelopmobileapplicationsusingimagerecogni-

tiontomakecancerdetectionaccessibletoall.Itcould

6ibid

7(n3)

9(n3)

8UNScience-Policy-BusinessForumonEnvironment,‘White

Paper:DigitalEarth:Building,FinancingandGoverningaDigital

EcosystemforPlanetaryData’(Draft1February2018)

10ibid

11HannahRitchieandMaxRoser,NaturalCatastrophes(2018)

<https://ourworldindata.org/natural-disasters>accessed31Janu-

ary2020

12Planet,‘ AnatomyofaCatastrophe:UsingImagerytoAssess

Harvey’sImpactonHouston’https://www .planet.com/insights/

anatomy-of-a-catastrophe/

13T aylorKubota,‘DeepLearningAlgorithmDoesasWellasDerma-

tologistsinIdentifyingSkinCancer’(StanfordNews,25January

2017)<https://news.stanford.edu/2017/01/25/artificial-intelligence

-used-identify-skin-cancer/>accessed31January2020

Delphi4|2019 210AIforSustainableDevelopmentGoals

thereforehelpachieve‘SDG3:GoodHealthandWell-

Being’.

Apotentialbarriertothewidespreaduseofsuch

applicationsisthelackofaccessibilityforcertain

typesofpopulations.Indeed,artificialneuralnet-

workshavebeentrainedonadatabaseofavailable

skincancerimages.Imagestendtobesourcedfrom

Westerncountries,becausedataavailabilityandAI

developmentoccursintheseregions,andtherefore

lighter-skinnedindividuals.14Theconsequenceof

thislackofdiversityandrepresentationisthatthe

AImodelcannotperformwiththesamelevelofac-

curacywhenaimingtodetectcancerousmoleson

darker-skinnedindividuals.

InascenarioofinappropriatelytrainedAIbeing

usedondifferentpopulationsets,thereisaspillover

effect:eventhoughthisAIsystemcanhelpachieve

SDG3forsomecommunities,ithasanunintention-

alnegativeimpacton‘SDG10:ReducingInequali-

ties’(Figure2).

III.ImplementingAIforSustainable

Development

Severalusecases,includingthesementionedinthis

paper,illustratetheopportunitiesAItechnologies

poseforprogresstowardsSDGs.However,likemost

technologies,AIisadual-purposetool.Ontheone

hand,itprovidessolutionstosustainabledevelop-

14JamesZouandLondaSchiebinger ,‘AICanBeSexistandRacist

—It’sT imetoMakeitF air(18July2018)NatureResearchJournal

<https://www.nature.com/articles/d41586-018-05707-8>ac-

cessed31January2020

Figure2:VariancesinSmartphonePenetration

Source:TheMobileEconomy2018,GSMA2018,McKinseyGlobalInstituteAnalysis

Note:Featurephonefiguresassumeequivalentto2Gpenetration,whilesmartphone

figuresassumepenetrationofphonesthatuse3Gorbeyond.Figuresmaynotsum

upto100%becauseofrounding.

Delphi4|2019211 AIforSustainableDevelopmentGoals

ment,and,ontheother,itcanincreasetensionsbe-

tweenandacrossSDGs.Forinstance,inthecurrent

waytheyarebeingdeveloped,AItechnologiesareof-

ten‘skills-biased’ ,requiringhumancapitaland

skilledlabourtooperatethem.Inconsequence,au-

tomationoflow-skilledorroutinejobscanleadto

significantdistributionaleffectsandincreasedin-

equalitythatcouldcreatebarrierstosocialinclusion

andglobalcooperation.15

TomitigatethedownsideeffectsofAIandrecon-

ciletensionsbetweenandacrossSDGs,itisimpor-

tanttoshapetherightpathtowardsthetechnology’s

implementation.Indoingso,thereneedstobeinfra-

structurefornationalinnovationsystems,mapping

fortrajectoriesandinterconnectednessofSDGs,

opendataandarobustpoolofAItalenttoopera-

tionaliseapplicationsforSDGs,andthemitigation

ofmisuseandmalicioususesofAI.

1.BuildingtheInfrastructureforNational

SystemsofInnovation

Establishingaviabletechnicalanddigitalinfrastruc-

turethatisabletosupporttechnologicalchangeis

essentialandenableskeysectorsandindustriesto

grow,startupecosystems,andefficientandim-

provedpublicservices.Forgovernmentstoprogress

towardsSDGs,theyfirstneedtoensuretheycan

buildtheappropriatetechnologicalinfrastructure

suchaselectricitysupply,Internetandbroadband

connectivity,computerhardware,software,and

technicalskillsforsupportandmaintenance.How-

ever,manycountriesacrosstheworldlackthecapa-

bilitytodoso,requiringsignificantupfrontinvest-

mentsandsubsequentlylong-termcommitment,po-

liticalwill,coherentpolicies,andupholdingtherule

oflaw.

Thediffusionofexistingtechnologiesindevelop-

ingcountriestendstolagbecauseofmanytechnical,

economic,institutional,legalandbehavioralbarri-

ers.Theseincludemismatchedneeds,tradetariffs,

privatesectorcapacity,andlimitedaccesstotrusted

information.16Inlinewiththe‘leaveno-onebehind’

maxim,barrierstotechnologydeploymentanddif-

fusionmustberemoved,particularlyfordeveloping

countries,soeconomiescanbuildtheinfrastructure

neededforinnovation.

Althoughtherewillbeconsiderableshortterm

costslinkedtocommittingandinvestinginthein-

frastructurecapableofsupportingAI,countriesthat

willbeabletoupgradetheirinfrastructure,R&D,and

skilldevelopmenttoenableAIhavethechanceto

leapfrogandacceleratetheireconomies.Thiswill

givethemthecrucialfoundationstosucceedintheir

questtoachieveseveraloftheSDGs.17

2.MappingforTrajectoriesand

InterconnectednessofSDGs

Nationalandinternationalroadmapsforachieving

SDGsshouldconsiderparticipationfromgovern-

ment,privatecompanies,academia,andNGOs.Tech-

nologyroadmapsproducedatthenationalandinter-

nationallevelswillidentifyopportunitiestouseAI

technologiesforvariousSDGsandhowbesttoexe-

cuteonthesethroughtime.Forexample,R&D

roadmapswillhelpstructureandbudget,providein-

sightintoR&DandPPPpartnerships,andconduct

scienceandtechnologytrainingefforts.18

Opportunitiesshouldbecreatedforthescience

andengineeringcommunitytoprovidefeedbackon

whatworksanddoesnotworkwell.Policiesencour-

agingscientistparticipationinnationaldecision

makingandinestablishingtechnologytransfer

mechanismscanimprovenationalinnovationcapac-

itiesandestablishconnectionsbetweenresearch

communitiesandeconomicsectorsandcivilsociety.

Forexample,policystemmingfromscience-basedin-

formationandwiththesupportofclimateadapta-

tiontechnologyhasreducedwatershortages,thein-

tensityoffloods,droughts,andheatwaves.19

Assessmentandmetricsareneededtoalignlearn-

ingacrosspracticeareas.Abroadpictureandcross-

sectoralperspectiveoftheSDGsisimportantasthey

15GlobalSustainableDevelopmentReport2016,‘Perspectivesof

ScientistsonT echnologyandtheSDGs’(2016)<https://

sustainabledevelopment.un.org/content/documents/

10789Chapter3_GSDR2016.pdf>accessed31January2020

16CédricPhilibert,‘BarrierstoT echnologyDiffusion:TheCaseof

SolarThermalTechnologies’(2006)InternationalEnergyAgency

(2006)9<https://www.oecd.org/env/cc/37671704.pdf>accessed

31January2020

17JacquesBughin,‘MarryingArtificialIntelligenceandtheSustain-

ableDevelopmentGoals:TheGlobalEconomicImpactof

AI’(2018)<https://www.mckinsey.com/mgi/overview/in-the-news/

marrying-artificial-intelligence-and-the-sustainable>accessed31

January2020

18(n13)

19(n13)

Delphi4|2019 212AIforSustainableDevelopmentGoals

areinterlinkedincomplexandoftensubtleways.Ac-

tionstoprogressononeSDGsectormayenhanceor

diminishperformanceinothersectors,orleadtoun-

intendedconsequences.Therefore,integratedassess-

mentmodelscandesignsustainabledevelopment

policiesthattakethisintoaccountaswellasidenti-

fypossiblemethodstoimproveandovercomebarri-

erstosustainableinnovation.20

Furthermore,nationalAIstrategiesneedtofur-

theraligntotheachievementofSDGs.Globally ,AI

strategiesexhibitawiderangeofobjectivesandpri-

orities,whichareaddressedbyavarietyofpolicy

tools.Forexample,manystrategieshavecomponents

aimingtofosterlocalAItalent&skillsdevelopment,

andcanrelyonacombinationofgovernmentfund-

ingforapprenticeshipsoracademicpositions,train-

ingprogramsledbyforeigntechnologycompanies

ordomesticuniversities,orregionalacademichubs.

Strategiesalsovaryintheirlevelsofcommitment,

fundingandimplementation.Meanwhile,several

countriesaredevelopingdomesticAIinnovation

ecosystemsintheabsenceofanofficialnationalAI

strategy.

Finland,UAE,Estonia,AustraliaandIndiaexplic-

itlyaimtoboosteconomicgrowththroughAIadop-

tionandapplicationsinbusinessesandkeysectors.

Incomparison,AIstrategypublicationsbyChina,

theUSA,FranceandtheEuropeanCommissionfo-

cusonmaintainingorcapturinggloballeadership.

Notably,India’snationalAIstrategy,brandedas‘AI

forAll’,includesadoptionintosectorswhereAIcan

maximizeinclusionandhumandevelopment,in-

cludinghealthcare,agriculture,education,infra-

structureandtransport.21

AglobalframeworksuchasthatoftheSDGscan

complementthesenationalstrategies,servingasa

thresholdforprogressofAIdevelopmentanddeploy-

ment.

3.OpeningDatafortheRealisationof

theSDGs

OneofthebiggestbottleneckstoharnessAIforso-

cialgoodisdataavailabilityandquality .MostAIca-

pabilitiessuchasneuralnetworksrequireaccessto

high-quality,massive,andreliableopendata.Such

bigdatacanfacilitatestakeholderstorapidlyidenti-

fyproblemareasandcustomisesolutions.However,

thebasisofopendataisdatademocratisation,ensur-

ingdataisavailabletoeveryone,whichrequiressig-

nificantcommitmentfromallstakeholderstoshare

informationandmoveagainstacompetitivedata-

marketenvironment.22

Dataaccessibilityremainsasignificantchallenge

particularlyindevelopingcountries,wheredatamay

beownedbyprivatecompanieswithaprohibitive

costformostlocalgovernmentsandNGOs.23Coop-

erationbetweenthepublicandprivatesectorwillbe

essentialtoovercomesuchchallengeas‘muchofthe

datathatareessentialorusefulforsocialgoodappli-

cationsareinprivatehandsorinpublicinstitutions

thatmightnotbewillingtosharetheirdata.’24The

organisationswhichcurrentlycapturemostdataare

telecommunicationandsatellitecompanies,social

mediaplatforms,financialinstitutions,hospitalsand

governments.Thesedatacansometimescontainvery

sensitiveinformationsuchasanindividual’smedical

record,credithistoryortaxdetails.25Forsuchrea-

sons,privateorganisationsasmuchaspublicones

areoftenreluctanttosharethesedatawithNGOsand

socialentrepreneurs.Thedatamayalsohavetoohigh

businessandcommercialvaluetobepotentially

leakedtocompetitors.Itisthereforecrucialtobuild

trustbetweenthesedifferentsetofactorsandbuild

appropriateframeworkstofacilitatedatasharingfor

socialgood.

Atnationallevel,thereisrecognitionofthevalue

ofopendataandatthegloballevel,countriesare

openinguptheirdatasetstoachievetheSDGs.Itis

currentlybeingusedincitiessuchasRiodeJaneiro

forcityplanning,Tanzaniatoaccessschoolperfor-

mance,toimproveaccesstoeducationinKenya,and

tomaptheEbolaoutbreakinW estAfrica.26

Issueswithdatasharingtendtoliewithinstitu-

tionsandcanberesolvediftheyachieveagreater

20(n13)

21NITIAayog,‘NationalStrategyforArtificialIntelligence’Discus-

sionPaper(2018)<https://niti.gov .in/writereaddata/files/document

_publication/NationalStrategy-for-AI-Discussion-P aper.pdf>ac-

cessed31January2020

22AnanyaNarain,‘WhyDataRevolutionisCrucialfortheSuccess

ofSDGs’(GeospatialWorld,1August2017)https://www

.geospatialworld.net/article/data-revolution-for-sustainable

-world/

23(n4)

24(n4)

25(n4)

26(n4)

Delphi4|2019213 AIforSustainableDevelopmentGoals

understandingofthevaluethatdatasharinghasin

helpingachievethebroadermissionoftheSDGs.To

addressaccessibilityandavailabilityofqualitydata

concerns,aglobal,regional,andnationalframework

fordatacouldbedevelopedtoencouragesynergies

betweendataprovidersanddatacollaboratives.This

willensuredataaccessibilityforall,filldatagaps,

generatenewdatasets,createdynamicvisualisations,

thusenablingtimelyandtargeteddecisionmaking

todrivetheSDGs.27

Initiativesandtaskforcesthatseektobringto-

gethermultistakeholderandmultidisciplinary

groupstohelpdesign,build,pilot,andscalenovel

frameworksandprotocolsneededfordatasharing

anddatagovernancemodelsareneeded.Poolingto-

getherappropriatepoolsofdataforspecificusecas-

es,theseinitiativescanserveasa‘trusted’platform

fordatatobeusedforgood.Someinitiativesthat

havebeensetupforthispurposeincludetheGlobal

DataAccessFramework,theUNBigDataPlatform,

theCGIARplatformforBigData,andtheDigitalPub-

licGoodsAlliance.

4.PreparingAIT alent

AnotherimportantchallengeisthelackofAItalent

todevelopandtrainAImodels.Talentshortagesand

braindrainofmachinelearningscientistsconstrain

AIinnovationglobally,andinparticularincountries

lackingAIhubs.AshighlightedbyMcKinseyInsti-

tute’sreportonApplyingAIforSocialGood,inabout

halfoftheusecasesidentifiedahigh-levelAIexper-

tisewasrequired–thatispeoplewithaPhDinthe

fieldorseveralyearsworkingintechcompanies.

SomeotherusecasesrequiredlessAIexpertise,but

stillatleastdatascientistsandsoftwaredevelopers.

WhenAIprojectsrelyonseveralAIcapabilities,the

levelofcomplexitytendstoincreaseanddemand

high-leveltalent.Suchdemandisalsoongoinginthe

privatesectoranditisthereforedifficultforpublic

andnon-profitsectororganizationstocompete.

Moreover,accesstoAItalentcanbeharderinde-

velopingregions.ArecentresearchconductedbyEl-

27(n4)

Figure3:MostVisuallyImpairedPeopleintheWorldLiveinEmergingEconomies.

Source:TheVisionLossExpertGroup,McKinseyGlobalInstituteAnalysis

Delphi4|2019 214AIforSustainableDevelopmentGoals

ementAI’sCEOJean-FrançoisGagnéshowedastrong

imbalancebetweenregionsregardingtheiraccessto

AI’stalentpool(Figure4).Whilecountriessuchas

theUnitedStatescountedmorethan9.000highlev-

elexperts,otherslikeSouthAfricaorArgentina

countedlessthan100.28ThebraindrainofAItalents,

especiallyindevelopingregions,thereforehasaneg-

ativeimpactonequalaccessandglobalimplementa-

tionofAIsystemsforsocialgood.

Highersalariesandresearchopportunitiesrein-

forcetrendsinbraindrainbydrawingtalentedyoung

machinelearningscientistsandresearcherstoAIin-

novationhubs,primarilytowardsmoredeveloped

economies,withgreatermeanstofundsuchtalent.

Asaresult,thevastmajorityofcountriesfaceachal-

lengingproblemofnotonlydeveloping,butalsore-

taining,localAItalent.

IV .TheCaseforanIntegratedand

CoordinatedApproach

WhileAItechnologiesoffersignificantopportuni-

tiesforprogresstowardstheSDGs,theyalsocome

withrisks–leadingtoa‘moretogain,moretolose’

paradigm.Theneedforurgentintegratedactionto

fosterpositiveusecasesandmitigatethetensionsof

adual-purposetechnologyisclear.However,current-

lythistopicremainsinanexploratoryphasewith

severalactorsfragmentedaroundtheworldtryingto

makeprogress.

Thispaperarguesthattodevelopcoordinated

pathwaystowardsimplementationofAIforsustain-

abledevelopment,aglobalmulti-stakeholderpartner-

shipincludingthepublicsector,theprivatesectorand

thecivicsocietyisneeded.Suchaplatformisessen-

tialtoenablestakeholderstoleverageandshareeach

other’suniqueresources,expertiseandexperiences

forthecreationofeffectivedevelopmentsolutions.

Amulti-stakeholderinitiativecurrentlybeing

builtthatcantaketheformofthisglobalallianceis

‘AICommons.’29TheAICommonsisenvisionedas

asolidplatformtodemocratizeaccesstoAIsoa

broaderrangeofactorswillbeabletopartakeinthe

AIRevolutionandprogresstowardstheSDGs.Itwill

helpenableaccesstorelevantdata,computingpow-

er,algorithms,talentandapplieddomainexpertise

(usecaseandmethodologies),bylinkingproblem

ownersandproblemsolvers.

AICommonscouldmapandtrackcurrent

progressbeingmadeontheSDGsusingAItechnolo-

28JFGagné,‘TheGlobalAIT alentPoolGoinginto2018’(7Febru-

ary2018)<https://jfgagne.ai/talent/>accessed31January2020

29see<https://ai-commons.org/>accessed31January2020

Figure4:GlobalAITalentPoolHeatMap.Source:JFGagné

Delphi4|2019215 AIforSustainableDevelopmentGoals

gies,identifygapsonregionsorgoalsthatarebeing

underserved,andpoolresourcestoaddresssuchun-

derservedareas.Furthermore,AICommonscouldal-

lowthecommunityofdevelopers,entrepreneurs,

users,andorganizationstoworktogether,toidenti-

fyandenablebroaderapplicationsofAIinresponse

toactualneeds.

TheGlobalDataAccessFramework,30sittingwith-

intheAICommons,willhelpreconcileframeworks

andprotocolsfordatasharingandgovernanceinor-

dertohelpenablesuchAIsystemstoflourish.Itwill

bringawarenesstogovernmentsoftheactualwork

undertakentoprogresstheSDGs,especiallythoseat

thegrassrootlevel.Itwillunitethosewithpolitical

cloutandtheimplementersofSDGsatthesamelev-

el.31

TheresponsibilitytoachievetheSDGs,ofcourse,

willremainintherealmofindividualcountries,but

internationalsupportandpartnershipssuchasAI

Commonsarecriticalforunifiedprogress.Under

‘SDG17:PartnershipstoAchievetheGoals’,national

governments,theinternationalcommunity,civilso-

ciety,theprivatesectorandotheractorshaverecog-

nizedtheneedtocometogetherto‘strengthenthe

meansofimplementationandrevitalisetheGlobal

PartnershipforSustainableDevelopment.’32

Indoingso,newtypesof‘public-private-people

partnerships’(PPPPs)mustbebuiltandpiloted.Pub-

lic-PrivatePartnerships(PPPs)arecrucialtoacceler-

aterealprogresstowardstheSDGsusingAI.AIde-

velopmentandimplementationrequiresphenome-

nalamountsofcapitaltofundthetransformations

neededtoachievetheSDGs.Thiscannotbedone

withouttheprivatesector,whichcanprovidesignif-

icantcapacityandfueltransformation.Atthesame

time,thepublicsectoralsoplaysamonumentalrole,

servingasanenabler,afacilitator,andawatchdogto

ensuretheprocessofAIimplementationissocio-eco-

nomicallyandethicallybeneficial.

Thispaperaddsathirdlayertothetraditionalpub-

lic-privatepartnership,arguingfortheneedtoin-

volvepeopleinthesealliancesaswellthroughcivil

societyorganisations.Giventheincreasingneedfor

peoples’datatomakeAItechnologiespossibleand

alsotheimpactAIcanhaveonanindividuallevel,

peoplehaveapowerfulvoicetomakePPPPsamech-

anismtowardsachievingtheSDGs.Involvingpeople

throughouttheentirelifecyclecouldhelppreventun-

intendedconsequencessuchasbiaseddatainputor

unfairlydistributedoutput.Inabsenceofclearinsti-

tutionsthatcanprovideaccountabilityandoversight,

peoplecanserveastheinstrumenttosafeguardso-

cietyfromthepotentialnegativeconsequencesofAI

andbarrierstowardsachievementoftheSDGs.Last-

ly,asthedeploymentofAIsystemsisrelativelyin

itsearlystages,peopleshouldbeproactivelyinvolved

tobuildacultureoftrust.IfpeopledonottrustAI

systems,thesetechnologieswillnotreachtheirpo-

tential.

WithintheAICommonspartnership,newinter-

nationalcenterstobenamedtheAIforSustainable

DevelopmentGoals(‘AI4SDG’)Centerscanserveas

factoriestobuildthesePPPPs.Withbranchesactive

indifferentregionsoftheworld(tofactordifferent

prioritiesandcontexts)andindifferentknowledge

settingsfromacademiatogovernment(tofactordif-

ferentknowledgebackgroundandpractice),thepur-

poseofanAI4SDGcentrewillbetoconveneglobal

stakeholders–fromgovernment,privatesectorand

civilsociety–tocollaborateanduseAItechnologies

tomonitor,simulateandpredictprogresstowards

theSDGs.Thecenterwillaimtoserveasanengine

forpracticalexperimentationofgovernanceand

businessmodelsforAI,embeddingethicsanddesir-

ablehumanvaluesintorealworldprojectsthatfos-

terinclusiveAIdevelopment.

Increatingpilotprojects,considerationswillbe

giventowhatconstitutesasaneffectivepilotproject

tobuildasharedunderstandingontheuses,misus-

esand‘missed-uses’ofAIforSDGs,andhowtosi-

multaneouslybuildadynamicandcomprehensive

policyframeworktomitigatethedownsiderisksof

AIthatcouldhamperthedevelopmentgoals.Build-

inganiterativemodelforapplyingAItoeachofthe

SDGsisanimportantfeaturebecauseitwillallow

forcorrigibilityinthetechnologyandlimitthedown-

sideeffects,suchasbias,amplifyingforourmostvul-

nerablecommunities.Furthermore,devisinganag-

ilepolicyframeworkparalleltotestingAItechnolo-

giesforSDGswillhelpforgeatimelyanddynamic

feedbackloopbetweenimpactofAIandpolicyto

30see<https://thefuturesociety.org/2019/09/25/the-global-data

-commons-gdc/>accessed31January2020

31Narain(n20)

32UnitedNations,‘High-LevelPoliticalForumGoalsinFocus–

Goal17:StrengthentheMeansofImplementationandRevitalize

theGlobalP artnershipforSustainableDevelopment’(2018)

<https://unstats.un.org/sdgs/report/2018/goal-17/>accessed31

January2020

Delphi4|2019 216AIforSustainableDevelopmentGoals

managesuchimpact,makingitmorefeasibleforpol-

icymakersandgovernancemodelstokeepapacewith

technologicalchange.

Thesenewstructures–theAICommons,theGlob-

alDataAccessFrameworkandtheAI4SDGCenters

–canhelptheinternationalcommunityreconcile

technical,ethical,commercial,legalandoperational

frameworksandprotocols–totakepowerofAItech-

nologiesandsuccessfullymakeunifiedprogressto-

wardstheachievementoftheSDGs.

Delphi4|2019217 AIandOnlineIntermediationPlatforms

TheUseofAIbyOnlineIntermediation

Platforms

ConciliatingEconomicEfficiencyandEthicalIssues

FrédéricMartyandThierryWarin*

Thispaperfocusesontheeffectsoftheimplementationofartificialintelligence-basedalgo-

rithmsbyonlineintermediationplatformsintermsofbotheconomicefficiencyandfairness

orethicaldimensions.Itaddressesthreemainissues:theconsumersegmentationandthe

capacitytodiscriminate;thestrategicuseofartificialintelligencebydominantplatformsin

co-opetitivedigitalecosystems;andtheroleofartificialintelligence-basedtoolstoguaran-

teetrustworthyuser-reviewsone-commerceplatforms.Thispaperemphasisestheimportance

ofhavingstrongguaranteesforplatformusersintermsoftransparencyandaccountability.

I.Introduction

Althoughstillinitsinfancy,theacademicliterature

intheeconomicsdisciplineinsistsonthepromises

ofArtificialIntelligence(hereafterAI).1Ourfocus

hereliesinthefieldofIndustrialOrganization.In

thecaseofelectronicplatforms,anincreasinguseof

AIcansubstantiallyimproveperformanceinsever-

alareas.Forinstance,forsearchormatchingpurpos-

es,AIcanbeusedtorefinetherecommendations

providedtoInternetusers,hencereducingthesearch

costsaswellasthecoordinationcosts.2Anotherex-

ampleisthatAIcanalsocontributetoimprovethe

leveloftrustintheplatform.Similarly,AIcanpro-

videtheplatformwithadvanceduserdissatisfaction

detectiontools.AIcanalsohelpsolvethe‘coldstart’

problemfornewplatformsenteringthemarketby

providingincentivesforconsumerstogivetrustwor-

thyreviews.3

Wewillalsorelyonthefollowingdefinitionofan

onlineintermediationplatform:aplatformisconsti-

tutedofusers–consumersandsuppliers–whose

transactionsaresubjecttodirectand/orindirectnet-

workeffects.4

However,thisgreatpowercomesalsowithgreat

challenges.AIcouldindeedexacerbateconsumerma-

nipulationsorgenerate/worsencertainbiasesaspre-

dictionsmadeaboutconsumerpreferencesmight

haveaself-fulfillingeffect.5Thehighdegreeofcon-

sumersegmentationallowedbyAIcouldalsoleadto

discriminatorypractices.6Suchdiscriminationmay

taketheformofpricediscriminationbutalsoofdif-

ferencesinthequalityoftheproductsbeingoffered.7

Similarly,AI-basedadvancedindicators–setupby

theplatformtomeasurethequalityoftheservice

providedbyitscomplementors–canbemanipulat-

edtoimposeunbalancedcontractualterms.Finally,

theroleofAIinencouragingconsumeropinionsmay

DOI:10.21552/delphi/2019/4/11

*F rédéricMarty,CNRS–GREDEG–UniversitéCôted’Azur;CIRA-

NO,Montréal.F orcorrespondence:<frederic.marty@gredeg.cnrs.fr>

ThierryWarin,SKEMABusinessSchool;CIRANO,Montréal.For

correspondence:<thierry.warin@skema.edu>

1CatherineTucker,‘Privacy,Algorithms,andArtificialIntelligence’

inAjayAgrawal,JoshuaGans,andAviGoldfarb(eds)TheEco-

nomicsofArtificialIntelligence:AnAgenda(UniversityofChica-

goPress2018)423–37

2ThierryWarinandDanielLeiter,‘HomogenousGoodsMarkets:

AnEmpiricalStudyofPriceDispersionontheInternet’(2012)4

InternationalJournalofEconomicsandBusinessResearch514–29

3PaulMilgromandStevenT adelis,‘HowArtificialIntelligenceand

MachineLearningCanImpactMarketDesign’(2018)NBER

WorkingPapern°24282

4Broekhuizenetal,'DigitalPlatformOpenness:Drivers,Dimen-

sionsandOutcomes'(July2019)JournalofBusinessResearch;

JeanRochetandJeanTirole,'PlatformCompetitioninT wo-Sided

Markets'(2003)1JournaloftheEuropeanEconomicAssociation

990–1029

5PelleGuldborgHansenandAndreasMaaløeJespersen,‘Nudge

andtheManipulationofChoice:AFrameworkfortheResponsi-

bleUseoftheNudgeApproachtoBehaviourChangeinPublic

Policy’(2013)4EuropeanJournalofRiskRegulation3–28

6MarcBourreauandAlexandredeStreel,‘TheRegulationof

PersonalisedPricingintheDigitalEra’SSRNScholarlyPaperID

3312158,SocialScienceResearchNetwork<https://papers.ssrn

.com/abstract=3312158>accessed20January2020

7PrestonMcAfee2007,‘PricingDamagedGoods’(2007)1Eco-

nomics:TheOpen-Access,Open-AssessmentE-Journal.

Delphi4|2019 218AIandOnlineIntermediationPlatforms

haveadarksideifusedtogeneratefakereviews.

Thus,theeffectsofusingAIinonlineplatformsmay

bemoredebatablethaninitiallyexpected.

Therefore,thepurposeofthiscontributionisto

proposeaframeworktostudytheopportunitiesand

risksassociatedwiththeuseofAIinthespecificcon-

textofonlineintermediationplatforms.Inparticu-

lar,wehighlightthreepriorityareasthatweconsid-

eroftheutmostimportanceintermsofrisksposed

tothenatureofcompetition.Thesepriorityareasare

keysincetheyareactualthreatstoahealthymarket

system.Indeed,whilelawmakershaveconsidered

someoftheethicalissuesrelatedtoAI,attentionmust

alsobepaidtoitsconsequencesoncompetition.AI

canaffectthecompetitivedynamicsofmarketsby

increasingthemarketpowerofdominantfirmsat

theexpenseoftheircompetitors,tradingpartners,

andconsumers.8WhiletheuseofAIcanaddress

someofthekeycompetitiveissuesintheplatform

economy ,suchascoldstartandcontestability ,itmay

alsoincreaseimbalancesbetweenoperatorsandfa-

cilitatemanipulation.Theseissuesarenotonlylinked

toefficiencybutalsotothedynamicsofcompetition

itself.Adominantoperator(oragatekeeper)might

controlcompetitiveforcesandimpairconsumers’

freedomofchoiceandcompetitors’accesstomarket.

Forinstance,Hemphill(2019)underlinesthatthe

incumbent’sadvantagesondigitalmarketscanbeex-

acerbatedbythedevelopmentofmachinelearning.

Dominantplatformsbenefitfromeconomiesofscale

andscopeandfromanaccesstousers’datathatthe

newentrantcannoteasilyreplicateorovercome.

Suchpotentialbarrierstoentrymaysignificantlyin-

creasedbyAIimplementation.Suchtechnologies

mayexacerbatetheincumbents’competitiveadvan-

tagesbecauseoftheirhighfixedcosts,theirdata-

basedperformancenatureandthehugeinvestments

realisedbybothinternalandexternalgrowth(merg-

ersandacquisitions).

Afterpresentingthecurrentstateofregulatory

conversations,ourconclusionisthattherisksposed

toahealthymarketsystemareatbestunderestimat-

ed,atworsttotallyignored.Inwhatfollows,first,we

analysetheproposalsandregulationsmadebypub-

licauthoritiesaimingatguaranteeingatrustworthy

AIandsecond,weproposethreeexamplesofAI-

basedalgorithmstoillustrateethicalandeconomic

trade-offs.IftheinvisiblehandentersanAIblack

box,itisnecessarytobalanceefficiencygainsand

risksforsoundcompetitionandeconomicfreedom.

II.AIAccountabilityintheFaceof

EconomicandEthicalRisks

Tothebestofourknowledge,nowhereintheregu-

lationsdowefindareferencetothepotentiallydis-

ruptiveimpactofAIonthemarketeconomy.Accord-

ingtotheFrenchNationalConventionin1793:‘great

responsibilityfollowsinseparablyfromgreatpower’.

Sucharequirementisstilloftheutmostimportance

inthisAIage.Whatisknownasthe‘ AIrevolution’

isanincrediblypowerfultoolbutitalsoposesalot

ofquestions.Forthefirsttimeinourhistory,au-

tonomoussystemscanperformcomplextasksequiv-

alenttonaturalintelligence.ThetermArtificialIn-

telligencewascoinedbyJohnMcCarthyin1956ina

proposalforasummerresearchprojecttobeheldin

Dartmouthin1957.9AIconstitutesamajorformof

scientificandtechnologicalprogress,whichcangen-

erateconsiderablesocialaswellaseconomicbene-

fits.10

AIcanbeunderstoodasageneral-purposetech-

nology.11Weproposetousethefollowingupdated

definitionofAIfromtheEuropeanUnionHigh-Lev-

elExpertGrouponAI:

‘Artificialintelligence(AI)systemsaresoftware

(andpossiblyalsohardware)systemsdesignedby

humansthat,givenacomplexgoal,actinthephys-

icalordigitaldimensionbyperceivingtheirenvi-

ronmentthroughdataacquisition,interpreting

thecollectedstructuredorunstructureddata,rea-

soningontheknowledge,orprocessingtheinfor-

mation,derivedfromthisdataanddecidingthe

bestaction(s)totaketoachievethegivengoal.AI

systemscaneitherusesymbolicrulesorlearna

numericmodel,andtheycanadapttheirbehavior

byanalysinghowtheenvironmentisaffectedby

theirpreviousactions.Asascientificdiscipline,

8SyprosMakridakis,'TheForthcomingArtificialIntelligence(AI)

Revolution:ItsImpactonSocietyandFirms'(2017)90Futures

46–60<https://doi.org/10.1016/j.futures.2017.03.006>accessed

30January2020

9J.McCarthyetal,‘AProposalfortheDartmouthSummerResearch

ProjectonArtificialIntelligence’(1956)13http://jmc.stanford

.edu/articles/dartmouth/dartmouth.pdfaccessed30January2020

10AjayAgrawal,JoshuaGans,andAviGoldfarb,PredictionMa-

chines:TheSimpleEconomicsofArtificialIntelligence(Harvard

BusinessReviewPress2018)

11ErikBrynjolfsson,DanielRockandChadSyverson,‘ Artificial

IntelligenceandtheModernProductivityP aradox:AClashof

ExpectationsandStatistics’inAjayAgrawal,JoshuaGans,andA vi

Goldfarb(eds)TheEconomicsofArtificialIntelligence:AnAgen-

da(UniversityofChicagoPress2018)23–57

Delphi4|2019219 AIandOnlineIntermediationPlatforms

AIincludesseveralapproachesandtechniques,

suchasmachinelearning(ofwhichdeeplearning

andreinforcementlearningarespecificexam-

ples),machinereasoning(whichincludesplan-

ning,scheduling,knowledgerepresentationand

reasoning,search,andoptimization),androbotics

(whichincludescontrol,perception,sensorsand

actuators,aswellastheintegrationofallother

techniquesintocyber-physicalsystems)’.12

ThedevelopmentofAIdoesposemajorethicalchal-

lengesandsocialrisks.Indeed,AIcanrestrictthe

choicesofindividualsandgroups,disrupttheorgan-

isationoflaborandthejobmarket,orinfluencepol-

iticstonamebutafew .Scientificprogressbringsin-

crediblebenefitswhilecarryingnewrisks.Citizens

mustdeterminethemoralandpoliticalendsthatgive

meaningtotherisksencounteredinanuncertain,

andcomplexworld.

On8April2019,theHigh-LevelExpertGroupon

AIpresentedtheirEthicsGuidelinesforT rustworthy

ArtificialIntelligence.Accordingtotheguidelines,

themainprinciplesforatrustworthyAIarethree-

fold:(1)lawful(abidingbyallapplicablelawsand

regulations),(2)ethicalegrespectingethicalprinci-

plesandvalues,and(3)robustbothfromatechnical

andasocialperspective.13Inits‘EUguidelineson

ethicsinartificialintelligence:Contextandimple-

mentation’report,theEuropeanCommission’sHigh-

LevelExpertGrouponAIproposes7keyguidelines

forAIsystemsshouldmeetinordertobedeemed

trustworthy:Humanagencyandoversight,T echni-

calRobustnessandsafety ,Privacyanddatagover-

nance,T ransparency ,Diversity ,non-discrimination

andfairness,Societalandenvironmentalwell-be-

ing,andAccountability .14Suchconcernsresonate

withtheprincipleslaiddownbytheMontrealDec-

laration(2018)15:W ell-being,respectforautonomy,

privacyandintimacy,solidarity ,democraticpartici-

pation,equity,diversityinclusion,prudence,respon-

sibility,sustainabledevelopment.

Thequestionofhealthymarketsreliesondatagov-

ernanceasmuchasonsustainabledevelopment,two

topicsinHELGandtheDeclarationofMontreal.Da-

tagovernanceisalsotobefoundinarticle25of

GDPR16,alongtheUNESCO’s7thprinciplewithin

theBeijingConsensus:the‘impactofAIonpeople

andsocietyshouldbemonitoredandevaluated

throughoutthevaluechain’17.

Thefocusontheguaranteesrelatedtotheuseof

AIisalsoreflectedbytheOECD’sRecommendation

oftheCouncilonArtificialIntelligence.18Itsfirstsec-

tionsetsupthe‘Principlesforresponsiblesteward-

shipoftrustworthyAI’.Theprinciplesadvocatedby

theOECDarebasedonareviewofpotentialrisksfor

thesocietywhileenablinganAIdigitalecosystem.

Althoughallthesepointsarerelevantandofthe

utmostimportance,thesereportsgivelittleattention

tothespecificcompetitiverisksassociatedtodigital

oligopolies.Thereisnomentionabouttheorganisa-

tionofthemarketeconomyinthisAIage.However,

drawingfromtheOECDAIPrinciples,theG20adopt-

edhuman-centeredAIPrinciplesinJapaninJune

2019.Thisismaybetheonlytextthatgetsascloseas

wecanimaginetothenotionofAIbeingdisruptive

forthemarketeconomy.

Ifwefocusonintermediationplatforms,therisks

associatedwiththeuseofAIcanbeconsideredfrom

theperspectiveofcompetitionorconsumerprotec-

tionlaws,aswellastheprotectionofpersonaldata.

Eveniftheselegalresourcescanaddressasignificant

partoftherisks,theycannotansweralltheethical

issuesraised.Itwillthereforebeuptothelawmak-

erstostrikeabalancebetweenpotentialefficiency

gainsandderivedrisks.Weillustratethesetrade-offs

inthenextthreesectionsthroughexamplesofAIim-

plementationbyonlineplatforms.W einsistforeach

ofthemonthepotentialefficiencygainsthatcan

stemfromAIbutalsoontheethicalrisksraised.

12HLEG-AI,EuropeanCommission,‘EthicsGuidelinesforTrustwor-

thyAI’(8April2019)<https://ec.europa.eu/digital-single-market/

en/news/ethics-guidelines-trustworthy-ai>accessed20January

2020

13ibid

14T ambiamaMadienga,‘EUGuidelinesonEthicsinArtificial

Intelligence:ContextandImplementation–ThinkT ank’(2019)

<http://www.europarl.europa.eu/thinktank/en/document.html

?reference=EPRS_BRI(2019)640163>accessed20January2020

15MontrealDeclaration,‘TheDeclaration–MontrealResponsible

AI’(2018)<https://www.montrealdeclaration-responsibleai.com/

the-declaration>accessed20January2020

16Regulation(EU)2016/679oftheEuropeanParliamentandofthe

Councilof27April2016ontheProtectionofNaturalPersons

withRegardtotheProcessingofPersonalDataandontheFree

MovementofSuchData,andRepealingDirective95/46/EC

(GeneralDataProtectionRegulation)(TextwithEEARelevance)

OJL.V ol.119.http://data.europa.eu/eli/reg/2016/679/oj/eng

accessed30January2020

17UNESCO,'BeijingConsensusonArtificialIntelligenceand

Education-UNESCODigitalLibrary'(2019)<https://unesdoc

.unesco.org/ark:/48223/pf0000368303>accessed30January

2020

18OECD,‘OECDLegalInstruments’(2019)<https://legalinstruments

.oecd.org/en/instruments/OECD-LEGAL-0449>accessed20Janu-

ary2020

Delphi4|2019 220AIandOnlineIntermediationPlatforms

III.UsingAIforaFinerConsumer

Segmentation:EfficiencyattheRisk

ofDiscrimination?

Theperformanceofanintermediationplatformora

searchengineisbasedontheabilityofitsalgorithms

todeliverrecommendationstailoredtotheneedsof

itsusers.19AIallowsamorerefinedunderstanding

ofthelatterbyattachingeachusertoanarrowlyde-

finedsegment.Attheveryleast,theproposedresult

isdedicatedtoeachuserbasedonthepredictionthat

ismadeaboutherexpectationsorherabilitytopay.

Theseareundoubtedlypro-efficiencyeffects.Search

costsaresubstantiallyreducedforconsumers.How-

ever,theuseofAIinsearchenginesandmatching

platformsmaybeaccompaniedbyarisingriskfor

consumers:channelingtoonarrowlyherchoiceto-

wardstheoptionforwhichtheadequacyprediction

isstrongest.Inthiscontext,theuseofAImayhelp

perpetuatebiases.Forinstance,inanexperiment

aboutjobdiscriminations,jobadsforhigher-paidpo-

sitionsweredisplayed6timesmoretomenthanto

women.20

Inaddition,risksofaself-fulfillingprophecy(and

ofunduerestrictionofconsumerfreedomofchoice)

mustbetakenintoaccount.Isthedevelopmentof

AIinthisarealikelytoleadtoconfirmationandre-

inforcementofsocialbiases?Theassignmentofa

consumertoaparticularpatterncertainlymakesit

possibletosendherofferscorrespondingtoher

needsbutalsohasaperformativeeffectbylocking

herupinarestrictedspaceofchoice.Thealgorithm

hastheeffectofclosingoptionstoherandthuscon-

strainingherfutureoptions.InsofarAIisonlyapre-

dictivetool,wecanarriveattheparadoxinwhich

thealgorithmoptionswouldbeverifiedex-postsim-

plybecausetheveryconsequenceoftheprediction

istherestrictionofthespanofpossiblechoices.In

thesamevein,theAIoriginatedproposalisallthe

morelikelytobeacceptedsinceitreinforcesthecon-

sumer'sdecision-makingbias.

Itshouldalsobenotedthattheabilitytostatisti-

callyinferfromtheobserveddatathemaximum

amountthattheconsumeriswillingtopaycould

leadtotheimplementationofpersonalisedprices.

Withoutgoingasfarasperfectdiscriminationin

which,thepricewouldbeequalforeachconsumer

toherpropensitytopay ,AIcanleadtoaveryeffi-

cientpricesegmentation.21Theeconomiceffectsof

pricediscriminationareambiguous.22Itisfavorable

intermsoftotalefficiencyandcanallow–through

cross-subsidies–forcertainconsumerstoaccessthe

product.However,discriminatorypricesresultina

transferofwelfarefromtheconsumerstotheplat-

form.23

Thealgorithmcanalsoplayontherangeofprod-

uctsofferedtotheInternetuser.Dependingonits

anticipateddegreeofexpertiseortheneedsassigned

toher,thetechnicalperformanceoftheproposed

productmayvary .Thediscriminationthroughver-

sioningcanrelynotonpricesbutperformanceor

quality.Forasameprice,thecharacteristicsofthe

productofferedmightdifferfromausertoanother.

Attheextreme,incaseofanon-demandproduction

(ina4.0industryworld),theproductcanbededicat-

edtoaspecificuser.Suchanapproachmayleadto

deceptivecommercialpracticesandharmfuldiscrim-

inationamongconsumers.W ecouldeasilyimagine

thatnaïveorcaptiveonesmightonlyaccesstoprod-

uctscharacterizedbydeterioratedperformancesas

aresultofmachinelearningmethodsappliedtocon-

sumersegmentation.24

Aburgeoningfieldoftheeconomicsliteraturein-

vestigatesthepossibleexploitativeabusesthatcould

affectthemorenaïveconsumers.25Firmsincreasing-

lyhavethecapacitytodiscriminateamongtheircon-

sumers.Theever-increasingflowofinformation,the

enhancedcapacitiestoprocessthedatacollectedand

theobfuscationofon-linepricesandoffersmaylead

toaquasi-perfectdiscriminationexposingnaïvecon-

sumertopayunexpectedcharges26ortoaccesstoin-

feriorqualitygoodsorservices.

19(n3)

20T omSimonite,‘StudySuggestsGoogle’sAd-TargetingSystemMay

Discriminate’(MITTechnologyReview ,6July2015)<https://

www.technologyreview .com/s/539021/probing-the-dark-side-of

-googles-ad-targeting-system/>accessed20January2020

21SalilMehra,‘ AntitrustandtheRobo-Seller:Competitioninthe

TimeofAlgorithms’(2016)100MinnesotaLawReview

1323-1375

22HalVarian,‘ ArtificialIntelligenceandIndustrialOrganization’

(2018)NBERWorkingPaper

23J.-P .DubéandSanjongMisra,‘ScalablePriceTargeting’(2017)

WorkingPaperUniversityofChicago

24NiladriSyamandArunSharma,'WaitingforaSalesRenaissance

intheF ourthIndustrialRevolution:MachineLearningandArtifi-

cialIntelligenceinSalesResearchandPractice'(2018)69Indus-

trialMarketingManagement135–46.

25P aulHeidhuesandBotondKőszegi,'Naïveté-BasedDiscrimina-

tion'(2017)132QuarterlyJournalofEconomics1019-1054

26XavierGabaixandDavidLaibson,'ShroudedAttributes,Con-

sumerMyopia,andInformationSuppressioninCompetitive

Markets'(2006)121QuarterlyJournalofEconomics505-540

Delphi4|2019221 AIandOnlineIntermediationPlatforms

Perverseeffectsresultingfromalgorithm-based

discriminationcanalsostemfromamplificationsof

socialbiases.Actually,matchingorpricealgorithms

mayconfirmoraggravatesomediscriminational-

readyexistinginsociety .Twodifficultiescanbecon-

sidered.Thefirstdifficultyechoesasituationin

whichthealgorithmisbasedonreinforcingself-di-

rectedlearning.Assuch,itlearnsfromavailableda-

taandevolvesthroughinteractions.Indoingso,it

risksreproducingsocialbiasesand,muchworse,am-

plifyingthem.Theseconddifficultyisrelatedtothe

worseningoftheeconomicconsequencesofdiscrim-

inations.Manystudiesemphasisethisimpactonthe

incomeofagentsofferingtheirservicesonplatforms

orontheopportunitiesavailabletothem.27

Consumersmayreactnegativelytopersonalised

pricespricesorpricestrategiesgeneratingrandom

prices.28Forinstance,Amazonhadtoabandonran-

dompricevariationinitiativesin2000.29Thus,the

reputationaldamagecanbesignificantifthecon-

sumerhasaperceptionoftheplatform'sbehavioras

manipulative,misleadingorunfair.Inthatcase,the

potential‘market-based’sanctioncanplayasaprice

signalincentivisingtheplatformtomonitorcareful-

lyitspractices.However,suchaself-disciplinaryef-

fectcanonlybeeffectiveifthemarketpositionof

theplatformremainscontestable(ifthecompetition

isstilloneclickaway)andiftheconsumeriseffec-

tivelyawareofthesepractices.

Consumersmayadverselyreacttonewmarketing

practicesbasedonAIasthepossibleshiftfroma

shopping-then-shippingmodeltoashipping-to-shop-

pingmodel.30Asretailerscanmoreandmorepre-

ciselypredictconsumers’futureneeds,theycansend

theproductbeforeanyformalorder,allowingthecon-

sumertofreelyreturntheitem.31However,anunde-

siredshipping,notmandatoryresultingfromafalse

predictionaboutaconsumer’sneedsandpreferences,

mayleadtoanegativereaction.Forinstance,acon-

sumer’scurrentpreferencemaydifferfromtheones

inferredfromherpastbehavior.Theproposedprod-

uctmayputherinanuncomfortablesituation.32The

consumermayalsoreactnegativelytoaperceived

lossinautonomyintermsofconsumptionchoices.33

OneofthemainconcernsraisedbyAIimplemen-

tationformarketingrecommendationsintermsof

consumers’reactionscanbeillustratedwiththe‘pri-

vacy-personalisationparadox’.34Ontheonehand,

consumersaskfordedicatedoffersbut,ontheother

hand,theywanttopreservetheirprivacy.Theirtrade-

offmightbedistortedconsideringimperfectratio-

nality.AccordingtoAcquisti(2004)35,consumers

maybe‘privacymyopic’.Inotherwords,theymay

divulgeasubstantialamountofinformationinre-

turnforanotsosubstantialcounterpart.36

Theplatformmonitoringbythirdpartiescanbe

helpful.Distributedsurveillanceschemescanbea

waytoprovideguaranteestoconsumersandtopro-

videtheproperincentivestopromoteacompetition

throughquality(orthroughcommitmentsonfair

practices)amongplatforms.

Symmetrically,market-basedincentivesmightbe

insufficienttoguaranteethatfirmsproperlycontrol

theeffectsoftheiralgorithmic-baseddecisionsin

termsoffairness.Despitetheirownbias,consumers

mayreactnegativelyassoonastheyperceivethe

27BenjaminEdelman,MichaelLucaandDanSvirsky,‘Racial

DiscriminationintheSharingEconomy:Evidencefromafield

experiment’(2017)9AmericanEconomicJournal–Applied

Economics1-22;

AlexRosenblatetal,‘DiscriminatingT astes:Uber’sCustomer

RatingsasV ehiclesforWorkplaceDiscrimination’(2017)9

PolicyandInternet256-279;

MingmingChengandCarmelFoley,‘TheSharingEconomyand

DigitalDiscrimination:TheCaseofAirbnb’(2018)International

JournalofHospitalityManagement95-98;

GraziaCecereetal,‘STEMandT eens:AnAlgorithmicBiason

SocialMedia’(2018)WorkingP aperSSRNn°3176168

28ThierryWarinandDanielLeiter ,'HomogenousGoodsMarkets:

AnEmpiricalStudyofPriceDispersionontheInternet'(2012)4

InternationalJournalofEconomicsandBusinessResearch

514–29;AkivaMiller,'WhatDoWeWorryAboutWhenWe

SorryAboutPriceDiscrimination?TheLawandEthicsofUsing

PersonalInformationforPricing'(2014)19JournalofTechnology

LawandPolicy41-104

29MatthewEdwards,‘PriceandPrejudice:TheCaseagainstCon-

sumerEqualityintheInformationAge’(2006)10LewisandClark

LawReview559

30ThomasDavenportetal,'HowArtificialIntelligenceWillChange

theFutureofMarketing'(2020)48JournaloftheAcademyof

MarketingScience24-42

31(n10)

32(n30)

33QuentinAndréetal,'ConsumerChoiceandAutonomyinthe

AgeofArtificialIntelligenceandBigData'(2018)5Consumers

NeedsandSolutions28-37

34ElizabethAguireetal,'UnravellingthePersonalizationParadox:

TheEffectofInformationCollectionandTrust-buildingStrategies

inOnlineAdvertisementEffectiveness'(2015)9JournalofRetail-

ing34-49

35AlessandroAcquisti,'PrivacyinElectronicCommerceandthe

EconomicsofImmediateGratification'(2004)5thConferenceon

ElectronicCommerce,NewYork

36JoshuaGerlickandStephanLiozu,'EthicalandLegalConsidera-

tionsofArtificialIntelligenceandAlgorithmicDecision-makingin

PersonalizedPricing'(2020)JournalofRevenueandPricing

Management.

Delphi4|2019 222AIandOnlineIntermediationPlatforms

firm’sbehaviorasunfairormanipulative.Theper-

ceived(un)fairnessechoeswithdistributivejustice-

relatedconcernsandmayleadtosalelosses.37

TheuseofAI-basedrecommendationssystems

raisesreputation-relatedconcernsforfirmsaswellas

ethicalandlegalones.Ifanalgorithmmakesitspre-

dictionsasanon-accountableblackbox,itsdevelop-

ersandthefirmusingitcanbeliableaboutitspoten-

tialdiscriminatoryorunfaireffects.Martin(2019)38

explainsthatafirmmaybeaccountableifitdevelops

orimplementsaninscrutablealgorithm.Shedefines

suchanalgorithmasonethatlimits–orexcludes–

anyhumaninterventioninthedecisionprocessand

makesthisalgorithmimpossibletoobjectivelyex-

plainexpost.Inthisframework,afirmengagesits

corporateresponsibilitybyrelyingon‘toodifficultto

explain’decisionprocesses.Inotherwords,thelack

ofaccountabilitymaybeseenastheoppositeofan

ethical-by-designapproach.Asaconsequence,

opaqueandnon-accountablealgorithmsrequireasu-

pervision,fromathird-partyoraregulatoryagency .39

TheconfidencetowardsAI-baseddecisionsmaybe

reinforcedbytherecoursetoXAI,egexplainableAI.40

Wecanalsoinsistonasecondpotentialadvantage

ofAIinthefieldofsearchengines.AI-basedsearch

recommendationscanbecustomisedaccordingto

theperson'snaturalsearchprocess(broadfirstand

progressiverefinement).Indeed,consumers’search

behaviourevolvesduringitssuccessivestages.AI-

basedalgorithmscanadjusttheirresultstofitwith

suchaprocess.Basedonananalysisperformedone-

Bayusers,Blakeandal(2015)41showthatthecon-

sumerspreferatthefirststepsverybroad-rangere-

sultstoscreentheavailableoptionsandprogressive-

lyconvergetowardmorenarrowlytargetedresults.

Asearchormatchingalgorithmmayreproducesuch

pathandreviseateachiterationthescopeandthe

characteristicsoftheresultsdisplayed.However,

suchatoolhastwosides;itcanbothsupportanddis-

torttheconsumer'schoices.Again,ethicalconcerns

mustbeaddressed.

IV .UsingAItoFacilitateT rustin

Transactions:CorrectingInformation

AsymmetriesorIncreasingTrading-

Partners'Vulnerability?

Trustinonlineplatformsisacomplexmatter.Itde-

pendsonthenatureoftheplatform(proprietaryvs

open-source),theindustry ,thetechnology

(blockchainvshttps),thecompany ,etc.42

Oneofthedigitalintermediationplatforms’key

factorsofsuccesshasbeentheirabilitytosecure

transactions.Thistrustisnotonlyaboutsecuring

paymentsbutalsointermsofreducinginformation-

alimperfectionsthatcouldpreventtheactofpur-

chase.Forconsumers,theseimperfectionsweredue

toincompleteandasymmetricinformationaboutthe

qualityofproductsandsellersactiveinonlinemar-

ketplaces.Theopinionssubmittedonlinehave

playedasignificantroleincorrectingtheseinforma-

tionalbiases.AIcanbeaninterestingrelaytoaddress

thisissue:forinstance,bypredictingthequalityof

agivensellerbyinterpretingonlineexchangeswrit-

teninnaturallanguageinprevioustransactions.The

analysisofexchangesbetweencustomersandinde-

pendentsellers(throughnaturallanguageprocess-

ing)canmakeitpossibletoconstructadvancedindi-

catorsofunderperformanceandthereforetointer-

venetoremedyitveryearlyon.Assuch,theplatform

protectsitsconsumersevolvinginincompleteand

asymmetricinformationconditions.

Puttinginplacemechanismstocreateanenviron-

mentthatwouldincreaseparticipants'confidenceis

oneofthekeysoftheplatforms’businessmodel.43

37T imothyRichards,JuraLiaukonyteandNadiaStretskaya,'Person-

alizedPricingandPriceFairness'(2016)44InternationalJournal

ofIndustrialOrganization138-153

38KirstenMartin,'EthicalImplicationsandAccountabilityof

Algorithms'(2019)160JournalofBusinessEthics835-850

39FrankP asquale,TheBlackBoxSociety:TheSecretAlgorithmsthat

ControlMoneyandInformation(HarvardUniversityPress2015)

40(n36)

41ThomasBlake,ChrisNoskoandStevenT adelis,'Consumer

HeterogeneityandP aidSearchEffectiveness:ALargeScaleField

Experiment'(2015)83Econometrica155-174

42CarinvanderCruijsen,MauriceDollandFrankvanHoenselaar ,

'TrustinOtherPeopleandtheUsageofPeerPlatformMarkets'

(2019)166JournalofEconomicBehaviorandOrganization

751–66<https://doi.org/10.1016/j.jebo.2019.08.021>accessed

30January2020;

ImeneBenY ahia,NasserAl-NeamaandLaoucineKerbache,

'InvestigatingtheDriversforSocialCommerceinSocialMedia

Platforms:ImportanceofT rust,SocialSupportandthePlatform

PerceivedUsage'(2018)41JournalofRetailingandConsumer

Services11–19<https://doi.org/10.1016/j.jretconser.2017.10.021

>accessed30January2020;

NuanLuoetal,'IntegratingCommunityandE-Commerceto

BuildaT rustedOnlineSecond-HandPlatform:Basedonthe

PerspectiveofSocialCapital'(2020)153TechnologicalForecast-

ingandSocialChange119913<https://doi.org/10.1016/j.techfore

.2020.119913>accessed30January2020

43LiYfanMacinnesIanandY urcikWilliam,'ReputationandDis-

puteineBayT ransactions'(2005)10InternationalJournalof

ElectronicCommerce27-54

Delphi4|2019223 AIandOnlineIntermediationPlatforms

Suchacharacteristicwasnotaforegoneconclusion

sincetrustcouldnotbebasedoninterpersonalrela-

tionships,theexperienceofpasttransactionsorthe

collectivecontrolexercisedbyagivencommunityor

corporation.Norcouldtrustcomefromtechnicalde-

vicesasisthecase,forexample,throughblockchain

technologiesinwhichcryptographicevidencecanre-

placetrust.Thesuccessofthefirstonlinemarket-

placeswasensuredbytheimplementationofbuyer

feedbacks.Theseopinionsmadeitpossibletobene-

fitfromanex-anteandnotonlyanex-postevaluation

ofthequalityoftheproducts.Inotherwords,the

sharingofinformationmeantthatthegoodsandser-

vicesconsumeddidnotfall,foreachsuccessivecon-

sumer,withinthecategoryofexperiencegoods.Even

beforeenteringintoatransactionwithaformerlyun-

triedmerchant,buyersonaplatformbenefitfrom

theassessmentsoftheseller’spreviousconsumers.

However,thissuccessisthesubjectofincreasing

challenges.First,consumershavenoincentivesto

spendtimetowritereviewsandcanindividuallybe-

haveasfreeriders.Second,theirconfidenceincon-

sumers’reviewavailableonlinetendstobedecreas-

ing.Suchmistrustisduetobiasesinassessments,rat-

inginflation44andrisksofopinionmanipulation45.

Someusersmayalsocolludetoartificiallyincrease

theratingsbyrelyingonpuppetconsumersposting

falseopinionscorrespondingtofalsetransactions.46

Theuseofartificialintelligencecanbealeverto

restorethistrust.47Theideaistousethemessages

exchangedontheplatformbetweensellersandbuy-

ersbeforeandafterthetransaction.Thesupportof

NLP(NaturalLanguageProcessing)allowssuchan

evaluation.Thealgorithmaimsatpredictingwhich

characteristicseachconsumerislikelytoappreciate

inthelightofherinterestsandneeds.

SuchamethodwasimplementedbyMasterovet

al(2015)48oncommentsleftontheeBayplatform.It

isamatteroffindinganelementthatcanpredictan

unsatisfactorytransaction.Theauthorsreliedon

messagesandinternaldatawithintheplatformthat

couldindicatethatthetransactionwasnotsatisfac-

tory(complaint,non-receiptorreturnoftheobject).

Theindicatorofbadexperienceisthedependentvari-

able.Thealgorithmwillaimtopredictthisresult

fromthemessagesexchanged.Afteratransaction,

nomessagescanbeexchanged,negativemessages

canbelisted,andfinally‘neutral’messagescanbe

recorded.85%ofthetransactionsstudieddonotgen-

erateanymessages.Whenthereisnomessage,the

numberofunsatisfactorytransactionsis4%.When

aneutralmessageissent,thisrateis13%.Whenat

leastonenegativemessageissent,thisrateincreas-

esto30%.Apriori,thehighertheproportionofneg-

ativemessagesasellerreceives,thelessqualityhe

canbeconsideredas.Thisfrequencymakesitpossi-

bletocalculateaqualityscorethatappearstobea

goodpredictoroffutureperformance.

AIallowsthisindicatortobeinferredfromlarge

databasesofemailexchangeswritteninnaturallan-

guagetopreventtheconsumer–disappointedbya

seller–fromturningawayfromtheplatform.49In

thepresentcase,‘thefractionofaseller'smessage

trafficthatwasnegativepredictswhetherabuyer

whotransactswiththissellerwillstoppurchasing

oneBay’.Thisultimatelyallowstheplatformtosanc-

tionanon-performingselleronobjectivegroundsor

tohaveleadingindicatorsofthedeteriorationinthe

qualityoftheserviceprovided.Thesemonitoring

methodsmayalsoraiseconcernsassoonaswecon-

siderinformationandpowerasymmetriesbetween

theplatformanditscomplementors.Althoughthese

toolsultimatelyprotectconsumers,theyplaceinde-

pendentsellersunderevenclosercontroloftheplat-

form.BydoingsoAIincreasestheirdependenceand

vulnerability.Whatisgoodforconsumersisnotal-

waysgoodforplatforms’tradingpartners.

Therefore,therearealsoethicalconsiderations

withsomealgorithmspunishingnon-performing

sellers.Itisevenmorerelevantinthecontextofthe

potentialblack-boxeffect,butsolutionsexistwith-

outopeningtheblackbox.50Explainablemodelscan

44GeorgiosZervas,DavideProserpioandJohn,Byers,‘ AFirstLook

atOnlineReputationonAirbnb,WhereEveryStayisAbove

Average’(2015)WorkingP aperBostonUniversity

45DinaMayzlin,Y anivDoverandJudithChevalier,‘Promotional

Reviews:AnEmpiricalInvestigationofOnlineReviewManipula-

tion’(2014)104AmericanEconomicReview2421-2455

46WeijiaY ouetal,'ReputationInflationDetectioninaChinese

C2CMarket'(2011)10ElectronicCommerceResearchand

Applications510-519

47(n3)

48DimitriyMasterov ,UweMayerandStevnTadelis,‘Canaryinthe

E-commerceCoalMine:DetectingandPredictingPoorExperi-

encesUsingBuyer-to-sellerMmessages’(2015)Proceedingsofthe

16thACMConferenceonEconomicsandComputation81-93

49ChrisNoskoandStevenT adelis,‘TheLimitsofReputationin

PlatformMarkets:AnEmpiricalAnalysisandFieldExperiment’

(2015)NBERworkingpapern°20830

50SandraWachter,BrentMittelstadtandChrisRussell,‘Counterfac-

tualExplanationswithoutOpeningtheBlackBox:Automated

DecisionsandtheGDPR’(2017)ArXiv:1711.00399[Cs]<http://

arxiv.org/abs/1711.00399>accessed20January2020

Delphi4|2019 224AIandOnlineIntermediationPlatforms

bedetectedareoftenproposedasasolutiontothe

potentialblackboxissue.51Algorithmshavetobeac-

countablewithoutopeningtheblackbox,mainlyfor

competitivereasonsrelatedtotradesecrecy .52

V .UsingAItoCreateaMarketfor

OnlineEvaluations:InSearchof

Objectivity

Creatingorreinforcingthetrustgrantedtoanonline

intermediaryimpliesprovidingconsumerswitha

largenumberofreviewsonitsproducts.Suchopin-

ionsareessentialtoreduceconsumers’information-

alasymmetriesandbydoingsoaddressingthecold

startissueforanewentrant,forinstance,aninde-

pendentsellerproposingitsitemsforthefirsttime

onaplatform.Thislastonecannoteasilytransferits

reputationfromaplatformtoanotherbecauseofthe

barrierstodataorreviewportability .Anewentrant

hasstrongincentivestorewarditsuserstowritere-

views;inthiscontext,thequestionistoknowhow

toconciliatetheseincentiveswithguaranteesin

termsofobjectivity.AImaybeusedasatoolallow-

ingtheprovisionofunbiasedincentivesforonline

notifications.

AImayaddresstheissueofthecoldstartofplat-

formsorsitespublishingeditorialcontentonline.In

thecaseofmarketplaces,commentsareneededto

createtrust,butthesecommentsmustbetrustwor-

thy!Badcommentscanbefakeonesbywhichcom-

paniespunishtheircompetitors.53Goodcomments,

ontheotherside,canbepaidtospecialisedcompa-

niesinwritingfalseconsumerreviews.AImaybe

usedtocreateamarketforonlineassessments.A

largemajorityofbuyersononlinemarketplacesleave

noopinion.Inabsoluteterms,theconsumerhasno

reasontodoso:itconsumestimeandforfuturepur-

chases,shecanadoptastowawaystrategyusingthe

opinionsofothers.Theproblemisnotjustaboutin-

dividualincentives.AsMilgromandTadelis(2018)54

note,thisisalsoanindustrialorganizationproblem:

thelownumberofthird-partyevaluationsonanew

platformmakesbuyingfromitlesssecurethanbuy-

ingfromaplatformwithalarge‘stock’ofopinions.

TheEuropeanCommission'sJune2019Regula-

tiononrelationshipsbetweensellersandplatforms

stressedthispoint:thelackofdataandreviewporta-

bilitydoesnotallowthesellertotransferherrepu-

tationfromamarketplacetoanother.55Hindering

consumers’reviewsportabilityhastwopotentialan-

ti-competitiveeffects.First,itincreasestheseller's

dependenceontheplatform(byincreasingswitch-

ingcosts).Secondly,itconstitutesabarriertoentry

fornewplatforms(whichalsohastheindirectcon-

sequenceofdeprivingsellersofexitoptionscom-

paredtoexistingplatformsandthusfurtherincreas-

ingtheirdependence).

HowtouseAItosolvethisproblem?Lietal.

(2016)56analysefromacasestudyontheChinese

Taobaoplatformthepossibilitytochargemerchants

fortheoptionofhavingbuyersleaveanotice.The

ideaisnottobuygoodreviews.Theproblemisal-

waysoneoftrust.Itisaquestionofentrustinganal-

gorithm,andnotthesellerhimself,withthetaskof

decidingwhethertheopinionisrelevant.Theexper-

imentbeganinMarch2012withan‘evaluationdis-

count’scheme(takingtheformofex-postreimburse-

mentsordiscountcoupons).Paymentismade

whethertheopinionispositiveornegative.Itison-

lytheinformationalqualityofitscontentthatistak-

enintoaccount.Theinterestistwofold.First,it

makesitpossibletodistinguishbetweengoodand

badsalesmen.Indeed,thepurchaseofappraisalsis

aninvestmentthatwillonlybeprofitableifandon-

lyifthesellerisofgoodquality.Asthesellerknows

herowntype,thismechanismactsasarevealingcon-

tract.Second,itallowsthesellertosolvetheprob-

lemofthecold-startissue.Investmentinreputation

canbeacceleratedbypurchasing‘objective’assess-

ments.

Thesamecold-startissueappliesforonlinenews

publishers.AsY angetal(2019)57stress,thevalueof

51(n39)

52JuliaDresselandHanyFarid,‘TheAccuracy,Fairness,andLimits

ofPredictingRecidivism’(2018)4ScienceAdvanceseaao5580

<https://doi.org/10.1126/sciadv.aao5580>accessed20January

2020

53JustinJohnsonandD.DanielSokol,‘UnderstandingAICollusion

andCompliance’forthcominginD.DanielSokolandBenjamin

vanRooi(eds)CambridgeHandbookofCompliance<https://

papers.ssrn.com/sol3/papers.cfm?abstract_id=3413882>accessed

30January2020

54(n3)

55EuropeanCommission,Regulation(EU)2019/1150onPromoting

FairnessandTransparencyforBusinessUsersofOnlineInterme-

diationServices

56LingfangLi,StevenT adelisandXiaolanZhou,‘BuyingReputation

asaSignalofQuality:EvidencefromanOnlineMarketPlace’

(2016)NBERWorkingPapern°22584

57ZeY angetal,‘Read,AttendandComment:ADeepArchitecture

forAutomaticNewsCommentGeneration’(2019)arXiv.org>cs

>arXiv:1909.11974

Delphi4|2019225 AIandOnlineIntermediationPlatforms

anarticlecloselydependsonthenumberandthe

qualityofthecommentsitgenerates.Commentaries

provideadditionalinformationtoreadersandim-

provetheirengagementonthewebsite.Editorshave

strongincentivestoencouragesuchcommentsand

debatesamongusers.However,theuseofAI-based

technologymightraiseethicalissues.Opinionscan

bewrittenbyAIsandusethereactionsofInternet

userstocreateartificialfixingpointsorevenguide

debates.Suchautomaticnewscommentingsystems

mightalsoaimatgeneratingneutralandreliable

commentsenhancingthereaders’experiencebyus-

ing‘read-attend-procedures’basedonmachineread-

ingcomprehension(MRC)devices.58Itremainstrue,

however,thattheethicalguaranteesrequiredby

firmsareessentialinguaranteeingthemodelintegri-

ty.

VI.Conclusion

Whatarethepossibleprinciplesforguaranteesas-

sociatedwithAIineconomics?Cantransparencybe

required59andbesufficienttomakemarketplayers

accountable?Allowingthirdpartiestoaccessthe

codemightconflictwithtradesecrecyrulesandin-

creasetherisktoseeitsalgorithmsfooled.How,then,

canex-postaccountabilityforchoicesbeensured?

Accountabilitydemandstheidentificationofthree

elements:(1)thepeopleinvolved,(2)thedecision

processand(3)theinputsusedtoformthisdeci-

sion.60TheincreasingroleofAIinplatformeconom-

icssupposestoprovideguaranteesthatefficiency

gainsforfirmswillnotbepaidbyincreasedinfor-

mationasymmetriesandmanipulationcapacitiesat

theexpenseofconsumersandtradingpartners.The

useofAIbyelectronicplatformsmustnotfacilitate

exclusionaryorexploitativeabuses.Enhancingdis-

criminationpossibilitieswouldbealsoproblematic

inthat,aswehaveseen,theeffectofpricediscrimi-

nationonconsumerwelfarecanbediscussed.Such

potentialabusesmaycompromisetheconfidencein

thedigitaleconomy .Thesearenottheonlycompeti-

tionriskspointedoutbytheacademicliterature.Con-

cernsaboutbot-ledtacitcollusionequilibriaarealso

stressed.Self-reinforcingmachinelearningmight

favourspontaneousconvergenceofcompetitorsto-

wardcollusivepriceswithoutanyexplicitintentand

informationexchangedevices.61

Moreover,someadditionalandevenmoreprob-

lematicdimensionsshouldbeconsidered.Thefirst

oneconcernsthefreedomofchoiceforconsumers

andforproducers,theaccesstothemarket.AIcan,

tosomeextent,constrainandmanipulatechoices

withouttheaccountabilityofalgorithmsbeingobvi-

ousatthistime.Thesecondoneisrelatedtotheis-

sueoftheonlinereputationmonitoring.AIcanbea

powerfultooltoassessthenatureandbehaviorofa

selleronamarketplaceandincentivehimtoprovide

agoodservice(throughscoringsorthreatsofaccount

suspensionorsuppression).Thisevaluationmecha-

nismdoesnotfocusononlyoneofthesidesofthe

platform.Theconsumerhimselfcanbegivenascore

inherdigitaljourney .Thisuseofartificialintelli-

genceraisesquestionsinanareathatisnotexclu-

sivelytheresponsibilityofthemarketbutofindivid-

ualfreedoms.Itoffersnewresourcesformonitoring

individualbehaviorfarbeyondthesphereofonline

markettransactions.

58ibid

59(30)

60StephenKosackandArchonFung,‘DoesT ransparencyImprove

Governance?’(2014)17AnnualReviewofPoliticalScience

65–87.

61EmilioCalvanoetal,‘ AlgorithmicPricing:WhatImplicationsfor

CompetitionPolicy?’(2019)55ReviewofIndustrialOrganization

155-171

Delphi4|2019 226SustainableAISafety?

SustainableAISafety?

Nadisha-MarieAliman,LeonKester,PeterWerkhovenandSoenke

Ziesche*

Inrecentyears,theneedtoaddressthemulti-facetedissueofAIgovernancewithsafety-rel-

evant,ethicalandlegalimplicationsataninternationallevelisbecomingincreasinglycrit-

ical.Simultaneously,theinternationalcommunityisfacingawidearrayofglobalchallenges

forwhichtheUnitedNationsinitiatedanagendawith17ambitiousSustainableDevelop-

mentalGoals(SDGs).Inthisarticle,weanalysepotentialsynergiesbetweenmethodologies

totackleboththeAIgovernancechallengeandtheSDGchallengeandworkoutnovelcon-

structiverecommendationsforanSDG-informedAIgovernanceandanAI-assistedapproach

totheSDGendeavour.However,wealsoexpoundmultipleopenissuesandcontextuallimi-

tationsthatmightplayarole.Overall,ouranalysissuggeststhatwhilesustainableAISafe-

tycannotbeguaranteedandthegoalsandvaluesoftheinternationalcommunitymay

changewithtime,AIgovernancecouldaimatasustainabletransdisciplinaryscientificap-

proachinstantiatedwithinacorrectivesocio-technologicalfeedback-loop.Finally,weelab-

orateontheimportanceoftheSDGsrelatedtoeducationandstronginstitutionsforthere-

alisationofthispotentiallyrobustAIgovernancestrategy.

I.SynergiesBetweentheChallengesof

UNSustainableDevelopmentalGoals

andAIValueAlignment

AsZieschehasproposed,1itmightbehighlyvalu-

abletoidentifysynergiesbetweentheso-calledAI

valuealignmentproblemandtheSustainableDevel-

opmentalGoals(SDGs)challengewhichhavesofar

largelybeentreatedseparatelydespiteapotential

mutualbenefit.Thereby,theAIsafetyrelevantprob-

lemofAIvaluealignmentrepresentsacrucialsub-

taskforAIgovernanceandaimsatidentifyingmeth-

odsonhowtoimplementAIsystemsactinginaccor-

dancewithhumanvalues.Thisproblemofsocietal

relevancehasbeenacknowledgedtobeofhighly

complexnatureduetotheabsenceofsufficientlyspe-

cificaswellasuniversalhumangoals.2Complemen-

tarily,theSDGscouldbeforinstanceinterpretedas

representingatypeofcondensedcompendiumof

certainkeyhumanvaluessharedinternationally

across193statesandthusofferingabasisforAIgov-

ernance.Inaddition,sufficientlyvalue-alignedAI

systemscouldbeutilisedassupporttoachievethe

SDGsinatargetedwayincludingsupportinpolicy

making.Infact,thesebidirectionalsynergiescould

bevitalgiventheurgencytoaddressAIgovernance

issuesandsincetheSDGshavebeenadoptedin2015

bytheUNGeneralAssemblyinorderto‘stimulate

actionoverthenext15yearsinareasofcriticalim-

portanceforhumanityandtheplanet’.3

However,theUNSDGframework,whichstates

that17SDGsshouldbeachievedby2030,revealscer-

taincaveatsthatneedtobeconsideredaprioriinor-

dertobeabletoharnessitforAIvaluealignmentor

todesignAIsystemsdirectlysupportingtheframe-

work.The17SDGs,includingthoserelatedtopover-

ty,environmentalpollutionorinequalityarefurther

subdividedinto169targetswhoseachievementis

DOI:10.21552/delphi/2019/4/12

*Nadisha-MarieAliman,M.Sc.,PhDcandidateatUtrechtUnivesi-

ty,DepartmentofInformationandComputingSciences.For

correspondence:<nadishamarie.aliman@gmail.com>;

Dr.LeonKester ,SeniorResearchScientistonethicalintelligent

systems,TNONetherlands;

Dr.SoenkeZiesche,IndependentResearcher ,India.

1SoenkeZiesche,‘PotentialSynergiesBetweenTheUnitedNations

SustainableDevelopmentGoalsAndT heValueLoadingProblem

InArtificialIntelligence’(2018)MaldivesNationalJournalof

Research47-56

2NickBostrom,Superintelligence:Paths,Dangers,Strategies

(OxfordUniversityPress2014)

3UNGeneralAssembly,‘ResolutionAdoptedbytheGeneral

Assemblyon25September2015;70/1:T ransformingourWorld:

The2030AgendaforSustainableDevelopment’(2015)

Delphi4|2019227 SustainableAISafety?

monitoredvia232indicatorswithvaryingquality.

Thedifferencesinqualityarepartlyreflectedinthe

subdivisionoftheindicatorsintothreedifferent

tiers.Asof26September2019,countriesdonotreg-

ularlyproducedatafor89(so-calledtierIIindicators)

outofthe232indicators,whilenointernationallyes-

tablishedmethodologyisyetavailableforafurther

33indicators(so-calledtierIIIindicators).45Oneof

themainissuesisthatseveraltargetsarenotquan-

tifiedandtospecifyindicatorsforsuchtargetsispar-

ticularlychallenging.Despitethesenotablechal-

lenges,weproposeconsideringtheUNSDGsascom-

plementaryapproachtowardstheAIV alueAlign-

mentproblem.Inordertoachievethat,thesetof

SDGshastobeformulatedinamachineunderstand-

ableversiontofacilitategoal-orientedAI-basedsolu-

tions.InordertoidentifyforAIvaluealignmentpur-

poseswhatasocietywants(ethicalself-assessment)

andinasecondstepwhatasocietyshouldwant(eth-

icaldebiasing),ithasbeensuggestedtocombinea

scientificallygroundedassessmentofhumanethics

withtechnologicalmethodssuchasvirtualreality

studiesforexperiencesfromafirst-personperspec-

tive.6Thereby ,webelievethattheSDGscouldserve

asaheuristicabletosupplementethicalself-assess-

mentbyqualitativelyspecifyingcandidatehuman

values.Moreover,certainmorepreciseSDGindica-

torsmightprovidehelpfulquantitativetargetsin

somecases.Beyondthat,wewillalsodiscusshowthe

SDGsrelatedtostronginstitutionsandqualityedu-

cationareexpedientforarobustdynamicapproach

toAIgovernancewhichisnotonlyproactivebutal-

soforeseestheneedforreactivecorrectionsleading

toasocio-technologicalfeedback-loop.7

InSectionIIwediscusspossiblecontributionsof

SDGsforAIvaluealignmentbytakingtheexample

ofvaluealignmentforintelligentautonomoussys-

temsandmorepreciselytheautonomousvehiclecase

forillustrativepurposes.InSectionIII,wecomment

onlimitationsandemergingsustainabilitychal-

lengesinthiscontextandformulateasetofrecom-

mendationswhichalsoencompassestheotherdirec-

tionofthesynergy,namelyAIsystemsforUNSDGs.

Finally ,inSectionIV ,weconc ludeanddiscussfuture

prospects.Inanutshell,wedonotclaimthatthe

SDGsareacomprehensivesolutionforAIgover-

nance,butratherapromisingcomplementarytool

giventheurgencyoftheproblemaswellasthefact

thattheSDGscanbeseenasthemostdetailedaswell

asinclusivevisionforhumandevelopmentevercom-

piled.8

II.ComplementingValueAlignmentfor

IntelligentAutonomousSystemswith

UNSDGs

Afterhavingtheoreticallymotivatedthepotential

usefulnessofUNSDGsforAIvaluealignment,we

discusstheapplicationofthispropositioninthecon-

textofintelligentautonomoussystemsutilisingthe

usecaseofautonomousvehicles(AV s)ashelpfultoy

modelwithethical,legalandenvironmentaldimen-

sionspertainingtotherealisationoftheSDGendeav-

ouritself.9(Inthefollowing,wewillrefertointelli-

gentautonomoussystemswiththeexpression‘arti-

ficialintelligentsystem’instead,sincewewantto

stressthatthegoalsfordecision-makinginthiscon-

textarespecifiedbyhumansandirrespectiveofthe

levelofautomation,itisnottheartificialsystemthat

craftsitsowngoalsautonomouslyasoftenmistak-

enlyassumed.)WeusevaluealignmentwithAVsas

toymodelduetothefactthattheusecaseexhibits

domain-generalimportantsafety-critical,ethicaland

legalfeaturesmanyofwhichwouldpertaintothe

valuealignmentofawiderangeofartificialintelli-

gentsystemsdeployedinreal-worldenvironments.

Firstly,itrevealstheneedtomakehumanvaluesex-

plicitforriskassessmentandplanningwhichrepre-

sentsasocietalchallengeofethicalself-assessment

sincehumansareoftenreluctanttoclearlyexpress

whattheywant.Secondly,theusecasepointstoan-

otherchallengeofscientificnaturewhichistode-

signsuitablemachine-readableframeworksthatcan

4UnitedNations,‘TierClassificationforGlobalSDGIndicators’

(2019)

5AnexemplarytierIIindicatoris14.1.1(indexofcoastaleutrophi-

cationandfloatingplasticdebrisdensity)whiletheindicator

12.4.2(hazardouswastegeneratedpercapitaandproportionof

hazardouswastetreated,bytypeoftreatment)representsan

exampleforatierIIIindicator .

6Nadisha-MarieAlimanandLeonKester,‘ExtendingSocio-Techno-

logicalRealityforEthicsinArtificialIIntelligentSystems’(2019)

IEEEAIVR

7Nadisha-MarieAliman,LeonKester,PeterWerkhovenand

RomanYampolskiy,Orthogonality-BasedDisentanglementof

ResponsibilitiesforEthicalIntelligentSystems.InInternational

ConferenceonArtificialGeneralIntelligence(Springer2019)

22-31

8(n1)

9RicardoVinuesaetal,‘TheRoleofArtificialIntelligencein

AchievingtheSustainableDevelopmentGoals’(2019)arXiv

preprintarXiv:1905.00501

Delphi4|2019 228SustainableAISafety?

serveasscaffoldsandtemplatesfortheidentifiedhu-

manethicalvaluesandlegalconceptions.Thirdly ,it

mightnecessitateasocietal-levelaggregationofhet-

erogeneousandoftenconflictingviewswithinthis

typeofethicalframeworks.Fourthly ,duetoitscom-

plexity,itmightrequireacognitive-affectiveexten-

sionofsociety(egusingtargetedvirtualrealitystud-

ies10)facilitatingahigh-qualityethicalself-assess-

mentandethicaldebiasingwhichconstitutesasci-

entificandtechnologicalchallenge.Fifthly,whilethe

casemightseemtocorrespondtoarathernarrowdo-

main,ithasimplicationsthatextendbeyonditand

willneedasupportivecontextwhichcanbecharac-

terisedasaninstitutional,legalandsocietalchal-

lenge.

SincetheUNSDGsthemselves,aswellasitstar-

gets,mightbetooabstracttoidentifyhowtheycan

bedirectlyappliedtotheAVcase,itishelpfultoscan

theSDGindicators11inabottom-upfashion.Inthe

following,weonlymentionanon-comprehensive

exemplarysetofsomeofthemoststraightforward

relatedindicators.Regardingenvironmentalaware-

nessforAVs,onecanforinstanceidentifytheindi-

cators9.4.1(CO2emissionperunitofvalueadded)

and11.6.2(annualmeanlevelsoffineparticulate

matter(egPM2.5andPM10)incities(population

weighted)).Theseindicatorsmightberelevantfor

hybrid-electricAVsbutalsoelectricAVsthatobtain

theirenergyfromcorrespondinglypolluting

sources.Atthetop-level,theindicator9.4.1isrelat-

edtotheSDG9whichseeksto‘buildresilientinfra-

structure,promoteinclusiveandsustainableindus-

trializationandfosterinnovation’,whileindicator

11.6.2stemsfromtheSDG11whichaimsto‘make

citiesandhumansettlementsinclusive,safe,re-

silientandsustainable’.Concerningethicalandlegal

aspects,onecanforinstancenameindicator3.6.1

(deathrateduetoroadtrafficinjuries),5.1.1(whether

ornotlegalframeworksareinplacetopromote,en-

forceandmonitorequalityandnon-discrimination

onthebasisofsex),16.7.2(proportionofpopulation

whobelievedecision-makingisinclusiveandre-

sponsive,bysex,age,disabilityandpopulation

group)asgermaneinthiscontext.Theseindicators

arerelatedtoSDG3whichaimsto‘ensurehealthy

livesandpromotewell-beingforallatallages’,SDG

5‘achievegenderequalityandempowerallwomen

andgirls’andSDG16onpeace,justiceandstrongin-

stitutionsrespectively.Allmentionedindicatorsare

tierIindicators(ierelativelyclearlyformulated,re-

spectinginternationalstandardsandwithregular

updatesondataavailable)exceptfor5.1.1and16.7.2

whicharetierIIindicators.Independentlyofthespe-

cifictypeofethicalframeworkenvisagedformean-

ingfulcontrolofAV s,thepresentedindicatorsrelat-

edto5SDGscouldbehelpfuleventhoughcertain-

lynotinisolation.Toexplainhowtheycouldbehar-

nessedforanethicalframeworkforAVs,wefirstde-

scribearecentlyintroducedscientificallygrounded

non-normativeframeworkforethicsinartificialin-

telligentsystemsdenotedaugmentedutilitarian-

ism12beforelinkingitbacktotheSDG-relatedsyn-

ergy.

Recently,augmentedutilitarianismhasbeenpro-

posedasscaffoldandtemplatetofillinhumanval-

uesandasinstrumenttocontrolartificialintelligent

systemsinanovelutility-basedmanner.Augmented

utilitarianismisinaccordancewithmoderninsights

inconstructionistaccountsofmoralpsychology13

andcognitiveneuroscience14accordingtowhich

mentalstates(alsomoraljudgements)15areembod-

iedconstructionsbasedondomain-generalprocess-

esofcontext-sensitive,perceiver-dependent,time-de-

pendentandaffectivenature.16Forthispurpose,aug-

mentedutilitarianismintroducesatypeofcontext-

sensitiveandperceiver-dependentutilityfunction

thatextendsbeyondtheclassicalconsequentialist

andutilitarianutilityfunctionswhicharefocused

solelyontheoutcomeofactions.Inthisway,ital-

lowsacoalescenceoftheclassicalnormativeethical

viewsrelatedtovirtueethics,deontologyandconse-

quentialism–whichseemtoallpossiblyplayarole

10(n6)

11UnitedNationsStatisticalCommission,‘GlobalIndicatorFrame-

workfortheSustainableDevelopmentGoalsandT argetsofthe

2030AgendaforSustainableDevelopment;UNResolution

A/RES/71/313'(2017)<https://unstats.un.org/sdgs/indicators/

Global%20Indicator%20Framework_A.RES.71.313%20Annex

.pdf>accessed20January2020

12Nadisha-MarieAlimanandLeonKester ,‘RequisiteVarietyin

EthicalUtilityFunctionsforAIValueAlignment’IJC AIAISafety

Workshop2019

13ChelseaScheinandKurtGray,‘TheTheoryofDyadicMorality:

ReinventingMoralJudgmentbyRedefiningHarm’(2018)Person-

alityandSocialPsychologyReview32-70

14IanR.Kleckneretal,‘EvidenceforaLarge-scaleBrainSystem

SupportingAllostasisandInteroceptioninHumans’(2017)Na-

tureHumanBehaviour0069;SuzanneOosterwijketal,‘Statesof

Mind:Emotions,BodyFeelings,andThoughtsShareDistributed

NeuralNetworks’(2012)NeuroImage2110-2128

15(n12)

16LisaFeldmanBarrett,HowEmotionsareMade:TheSecretLifeof

theBrain(HoughtonMifflinHarcourt2017)

Delphi4|2019229 SustainableAISafety?

inhumanmoraljudgements.17Toachievethis,aug-

mentedutilitarianismoffersaperceiver-dependent

templateallowingthejointconsiderationofagent,

actionandpatient.Forameaningfulcontrolofarti-

ficialintelligentsystemsusingthisframework,peo-

plewouldnotneedtoagreeonwhattheyvalueand

howtheyweighwhattheyvalue.Themainnecessary

preconditionwouldbetoconsenttoanacceptable

supersetofparametersallowinganaggregationof

theperceiver-dependentandcontext-sensitiveutili-

tyfunctionsrespectinglegalconstraints.(Notethat

thesemachine-readableutilityfunctionswouldfacil-

itateinterpretabilityofreasoning/planningatthelev-

elofthedecision-makingcomponentviathetrans-

parenthuman-craftedformulationofparameters

andweightsenablingconcretecounterfactualcom-

parisons.18However,interpretabilityatthesensor-

levelremainsanimportantoutstandingchallenge.)

Thenecessaryethicalself-assessmentandethicalde-

biasingtocrafttheseutilityfunctionscanbeassist-

edbyexpertsfromthelegislativeandbesupported

bytechnologysuchasvirtualoraugmentedreality19

providingarichcounterfactualexperientialtestbed

foraresponsiblehuman-centreddecision-making.To

makejusticetothetime-dependencyofhumaneth-

icalconceptions,onewouldalsoneedtoupdatethese

augmentedutilityfunctions.Thisindispensablecor-

rectionofutilityfunctionspairedwiththeneedto

updatetheworldmodelsoftheAIsystemsthem-

selvesinstantiatesadynamicsocio-technological

feedback-loop.

However,itbecomesclearthatsuchageneral

mechanismofcorrectionoferrorwithinasocio-tech-

nologicalfeedback-loopwhichishighlyrelevantfor

AIvaluealignmentcannotsucceedifthementioned

SDG16relatedtopeace,justiceandstronginstitu-

tionsisnotrealisedtoasufficientdegree.Thisisag-

nosticoftheethicalframeworkconsidered,sincethe

factthathumanknowledgeispronetoerrorsmakes

acorrectionprocessmandatory.Therefore,one

mightcategoriseSDG16asameta-goalforAIgover-

nance.Furthermore,theSDGsidentifiedcanalso

providemoredetailedinformationrelatedtocon-

creteparametersspecificallyappliedtotheAVcase.

Sincesocietywouldneedtospecifyasupersetofcan-

didateparametersthatareadmittedforconsidera-

tion,theSDGindicatorsspecifiedcanhelptoextend

orfilterthissuperset.Forinstance,itmightberec-

ommendabletoaddCO2andfineparticulatematter

relatedparametersintheaugmentedutilityfunc-

tionsoftheAV sifsuited(eveniftheprovidedindi-

cators9.4.1and11.6.2areratherrestrictedwithre-

gardtoallclimatechangerelevantmeasures)which

isinthespiritofsustainablemobility.Anobvious

additionalimportantparameterisrelatedtoroad

trafficinjuriesasencodedintheSDGindicator3.6.1.

Finally,onemustaddressriskassessmentparame-

terswhicharenecessarybecausecollisionscanin

practicenotbeavoidedwithabsolutecertaintyat

anytime20andthereisno100%securesystem21even

ifAVsaremeanttodrasticallyimprovethesecurity

ofmobility.Obviously,theUNSDGsdonotallowa

directconsiderationofthiscasesincecraftedfora

fullydifferentpurpose,althoughmoregenerally,the

indicator5.1.1.and16.7.2reflectrecommendations

ongender-inclusivelegalenforcementandnon-dis-

criminatorydecision-making.However,thisindica-

tiondoesnotdirectlysolvethecomplexproblemof

identifyingparametersthatcouldberelevantfor

dilemmaticsituationsinthecontextofriskassess-

ment,animportantpartofAIValuealignment.We

applyacloseranalysistothismissingpieceofcru-

cialimportanceinSectionIII.However,theseindi-

catorsmightemphasisethegeneralnecessitytocom-

petentlyaddressdiscriminationbasedonalgorith-

micbiaseswhichwewilltouchuponinSectionIII.

Lastly,onedrawbackoftheSDGframeworkisthat

itdoesnotallowtheidentificationofpreciseweights

andtheestablishmentofconcreteprioritiesinthe

pursuitoftheSDGs.Intotal,itcanbesummarised

thattheUNSDGsallowapowerfulsupplementto

valuealignmentwithAVs(andmoregenerallyarti-

ficialintelligentsystems)whichaddimportantqual-

itativeandquantitativecontributions.However,itis

notmeantasastandalonesolutionandshouldbe

utilisedinconjunctionwithanethicalframework

abletomodelethicalandlegaldimensionsandbe

extendedbyscientificallygroundedandtechnology-

assistedethicalself-assessmentanddebiasingmea-

sures.

17VeljkoDubljević,SebastianSattlerandEricRacine,‘Deciphering

MoralIntuition:HowAgents,Deeds,andConsequencesInflu-

enceMoralJudgment’(2018)PloSonee0204631

18(n7)

19(n6)

20SixianLietal,‘InfluencingFactorsofDrivingDecision-Making

UndertheMoralDilemma’(2019)IEEEAccess104132-104142

21RomanV .Y ampolskiyandM.S.Spellchecker,‘ ArtificialIntelli-

genceSafetyandCybersecurity:AT imelineofAIFailures’(2016)

arXivpreprintarXiv:1610.07997

Delphi4|2019 230SustainableAISafety?

III.SustainabilityChallengesinthe

ContextofAIValueAlignment

Itishighlyimportanttoaddressthementionedpoint

ofdecision-makingunderdilemmaticcircum-

stances,sincewhileweexemplarilyrefertotheAV

caseastoymodel,thetopicisgenerallyrelevantfor

artificialintelligentsystemsandartificialdecision

supportsystemsincriticaldomainswherethelives

andthewell-beingofpeopleareinherentpartofthe

decisionprocess.Conceivablerelevantapplicationar-

easmaybeegjustice,medicineandbureaucracybut

couldalsopertaintofuturehuman-machinecollab-

orationformssuchashuman-robotrescueteams,hy-

bridfirebrigadesorevenadvanceddomesticrobots.

ComingbacktotheAVcase,itisalsonoteworthy

thatfailingtoaddressthisissuecouldhavenon-triv-

ialrepercussionsonafewSDGindicatorsthemselves.

Ifthesatisfactionofsocietywithproposedethical

guidelinesforAV sislow ,itmight(ceterisparibus)

slowdowntheacceptanceofthetechnologyandpeo-

plewouldbelesswillingtoswitchtoAVs.Inturn,

thisreservationcouldpossiblyhinderanoptimal

overallreductionofairpollution(relatedtoSDGin-

dicators9.4.1and11.6.2)andimportantly ,itisthink-

ablethatthenumberofdeathsduetoroadtrafficin-

juries(seeSDGindicator3.6.1)whichAVsaresup-

posedtodecreasecouldthereforenotbedecreased

optimally.Infact,accordingtoastudyanalysingthe

socialdilemmaencounteredwithAV s,22whilepeo-

plewouldintheoryapproveAV sequippedwitha

utilitarianapproachtodilemmaticscenarios,they

wouldnotliketoridesuchanAVthemselves.More-

over,peopleexpressedtheirunwillingnesstoaccept

regulationsmandatingautilitarianself-sacrificeof

AVpassengersandexpressedtheiraversiontobuy

AVsinthepresenceofsuchregulations.Thistypeof

mechanismscouldleadtothementionedundesirable

repercussionsonsomeSDGindicators.Inthefollow-

ing,weportraywhytheutilitarianapproachtoethi-

caldilemmasinAVsasegsuggestedbyGermaneth-

icalguidelinesstatingthatinunavoidableaccident

scenariospersonalfeatures(egage)shouldnotbe

considered23posesadditionalproblemsofdifferent

nature.Thereafter,weprovideasetofrecommenda-

tionsonhowtoaddresssuchsocio-technologicalis-

suesbyinitiatinganactivesocietaldebatesupport-

edbyscienceandtechnologyincludingAIsystems

themselves–finallylinkingittotheotherdirection

ofthesynergyofAIsforUNSDGs.

Onecandistinguishtwomaintypesofproblems

thatcanarisewhenadoptingapurelyutilitariande-

cision-makingforAVsbutalsomoregenerallyforar-

tificialintelligentsystemsincriticaldomains:the

firstoneisrelatedtothediscrepancybetweenthe

(oftenculture-dependent)24ethicalintuitionsofmost

peopleandtheutilitarianapproachandthesecond

oneconcernsafundamentalproblem25relatedtoim-

possibilitytheoremsforclassicalutilitarianutility

functions.First,multipleexperimentsassessingeth-

icaldilemmaswithAVshavebeenperformedegin

textformorvirtualrealityenvironments.Depending

onthetypeofconstellationandthefocusofdiffer-

entrecentvirtualreality-basedexperiments,26the

moraljudgementsormoralactionsofparticipants

(denotedasperceiversinthefollowing)werehetero-

geneousandpartlycontradictoryoverall.Intheseex-

perimentselementsthatweredecisiveincludedfor

instance:theperceivednatureandtransparencyof

theagent,thelegalliabilityoftheagent,whetherthe

accidenthappenedbyactionorbyinaction,whether

theactioninvolvesaself-sacrifice,thenumberofpa-

tients,theageofpatients,thepersonalitytraitsofthe

perceiver,thecultureoftheperceiverandtheamount

oftimetheperceiverhadforadecision27.Thisisnot

surprising,sincemoraljudgmentsarerelatedtoa

perceiver-dependentdyadiccognitivetemplateen-

codingacontinuumalongwhichanintentionalagent

isperceivedtocauseharmtoavulnerablepatient28.

Themorethisseemstobethecase,themoreimmoral

doestheactseemtotheperceiver.Thereby,thevul-

nerabilitypeopleascribetopatientscanvaryex-

tremely.Generallyspeaking,thewaypeopleperceive

theagent,theactionandthepatientcanvarywith

regardtoapluralityofparametersofegcultural,so-

22Jean-FrançoisBonnefon,AzimShariff,andIyadRahwan,‘The

SocialDilemmaofAutonomousVehicles’(2016)Science

1573-1576

23NoaKallioinenetal,‘MoralJudgementsontheActionsofSelf-

drivingCarsandHumanDriversinDilemmaSituationsfrom

DifferentPerspectives’(2019)FrontiersinPsychology<https://

www.frontiersin.org/articles/10.3389/fpsyg.2019.02415/full>>ac-

cessed20January2020

24EdmondAwadetal,‘TheMoralMachineExperiment’(2018)

Nature59-64

25PeterEckersley,‘ImpossibilityandUncertaintyTheoremsinAI

ValueAlignment(OrWhyY ourAGIShouldnotHaveaUtility

Function)’(2018)arXivpreprintarXiv:1901.00064

26(n6)

27(n6)

28(n13)

Delphi4|2019231 SustainableAISafety?

cial,temporal,psychologicalandaffectivenature.

Therefore,whilethenumberofvictimsinanun-

avoidablecollisioncertainlyisanimportantfactorto

considerinethicalguidelines,humanethicalintu-

itionstendtoencompassarichersetofinformation.

Finally,itisimportanttonotethatclassicalconse-

quentialistandutilitarianutilityfunctionshavebeen

showntorepresentasafetyriskifusedincriticaldo-

mainswithfuturehumanwell-beingandhuman

livesaspartofthedecision-makingifusedwithout

moreado.29

AsintroducedinSectionII,augmentedutilitari-

anismallowsacontext-sensitiveandperceiver-de-

pendentaccountofhumanethicalintuitionswhich

isnotaffectedbythelimitationsencounteredbyutil-

itarianutilityfunctions.Thus,AIV alueAlignment

couldprofitfromharnessingthisframeworkinad-

ditiontothementionedSDGindicatorsandinitiate

asocietal-leveldebateonthechoiceofasuitablesu-

persetofvaluesthatmatterindilemmaticcircum-

stancesandhowtheyneedtobeweighted.Howev-

er,whilethiswouldservetotacklevaluealignment

atthelevelofthedecision-makingcomponent,arti-

ficialintelligentsystemsalsoneedtoexhibitvalue-

alignedpropertiesatthesensor-level.IntheAVcase,

thiswouldmapbywayofexampletotheproblem

ofdiscriminationviaalgorithmicbiasesatthelevel

ofimageclassification.NexttothementionedSDG

indicators5.1.1,16.7.2ongender-inclusivelegalen-

forcementandnon-discriminatorydecision-making,

onecouldaddthetierIIindicator16.b.1(Proportion

ofpopulationreportinghavingpersonallyfeltdis-

criminatedagainstorharassedintheprevious12

monthsonthebasisofagroundofdiscrimination

prohibitedunderinternationalhumanrightslaw).

Whileitisimportanttostrivefordatasetswitha

largevarietytoforestallsuchoftenunintentionally

arisingdiscriminations,westressthatthiscanand

shouldbecomplementedbyanexplicitformulation

withinthealgorithmitself.Duetothenatureofhu-

manethicalintuitions,autilityfunctionthatdoes

notencodeaffectiveanddyadicparametersofthe

currentsocietycannotbeagoodmodelforanethi-

calframeworkandcanthusnotinstantiateavalue

alignmenteffort.30Inmanycases,thiscanmanifest

itselfbyleadingtoinput-to-outputmappingsthat

peoplecategoriseasdiscriminatory.Anexamplefor

suchdiscriminatorymappingsisthecasewherethe

pictureofpersonswhosephenotypewasunderrep-

resentedinthedatasetwaslabelledwiththeclass‘go-

rilla’byGooglePhotos.Anotherexampleisastudy

whichwasrelatedtotheAVcontextinwhichre-

searchersanalysedmultipleimagerecognitionsys-

temsandfoundthattheimagesofpedestrianswith

darkerskintonesweredetectedwithaloweraccura-

cy.31Nexttomorediversedatasets,itisindispens-

abletoegexplicitlyweighmisclassificationserrors

ofthealgorithmsaffectively.Notallmisclassifica-

tionsareequallyimportant.Insimplifiedterms,itis

easilyconceivablethatforhumansitmakesadiffer-

encewhetheranimagerecognitionsystemmisclas-

sifiesachimpanzeeimageasagorillaincomparison

tothecaseofahumanbeingmistakenforagorilla.

However,manyalgorithmsnowadaysareimple-

mentedagnostictoanalogiesofsuchnuances.(As

‘solution’forthementionedincident,GooglePhotos

optedtocensorthegorillalabel32aswellasafewre-

latedlabelsincluding‘chimpanzee’.)Ifmachine

learningsystemsorartificialintelligentsystemsop-

timiseonlossfunctions,objectivefunctionsorutili-

tyfunctionsdevoidofrelevantaffective,contextual

andsocietalfactors,undesireddiscriminatoryside

effectscouldoccurcontinuously .(Notethatthisanal-

ogouslyappliestorule-basedsystemsandothers.)

Thiswouldrepresentnegativerepercussionsonboth

AIValueAlignmentandUNSDGs.Seenfromadif-

ferentangle,itcanbesaidthatresearchondiscrim-

inationstemmingfromalgorithmicbiaseswould

unifythedirectionsUNSDGsforAIvaluealignment

andAIforUNSDGs.Anadditionalimportantaspect

tocoverforthistypeofresearchareso-calledethical

adversarialexampleswhichrepresentadversarialat-

tacksonAIsystemsattemptingtoenticeAIsystems

‘toaction(s)oroutput(s)thatareperceivedasviolat-

inghumanethicalintuitions.’33

Asalreadydescribed,theSDGframeworkunfor-

tunatelyexhibitsalackofprecisionformultiplein-

dicators.Furthermore,certainofthemareunderspec-

ified.Thismakesitdifficulttotrackprogresstowards

29(n26)

30(n12)

31BenjaminWilson,JudyHoffmanandJamieMorgenstern,‘Predic-

tiveInequityinObjectDetection’arXivpreprintarX-

iv:1902.11097(2019)

32T omSimonite,‘WhenitcomestoGorillas,GooglePhotosRe-

mainsBlind’Wired1November2018<https://www.wired.com/

story/when-it-comes-to-gorillas-google-photos-remains-blind/>

accessed17October2019

33(n12)

Delphi4|2019 232SustainableAISafety?

specificindicatorsandtop-levelSDGs.However,it

hasbeenpostulatedthatmachinelearningapplica-

tionscouldextendtheSDGindicatorsbyutilising

multimodaldatafromdiversesourcesforabetteras-

sessmentofprogress.34Thiscouldalsoberelevantif

oneusesAIasdecision-supportforpolicy-making

thatshouldbeinlinewiththeSDGs.Moreover,aded-

icatedtypeofpositivecomputingcouldtargetSDG

3inabroadersense(ensurehealthylivesandpro-

motewell-beingforallatallages)35.However,sofar,

notmanysystematicAIattemptstowardstheSDGs

andtheirtargetshavebeenreportedyet36.Fromthe

perspectiveofAIvaluealignmentforartificialintel-

ligentsystems,theidentificationofprecisecriteria

basedonwhichonewouldinthefirstplaceselect

SDGsorSDGindicatorsgivenagenericdomainis

non-trivial,sincetheSDGshavebeenmotivatedand

formulatedfromaninternationalperspective.While

fortheAVtoymodelweheuristicallyscannedthe

indicatorsinabottom-upfashionsearchingforob-

viousmatches,futureworkcoulddevelopamoreso-

phisticatedmethodology.Forinstance,animportant

SDGthatmightasfirstglanceseemunrelatedtoval-

uealignmentintheAVcaseinparticularortoarti-

ficialintelligentsystemsingeneral,istheSDG4(en-

sureinclusiveandequitablequalityeducationand

promotelifelonglearningopportunitiesforall).As

onecanalreadyextractfromthearticlesofar,itis

highlyrecommendabletoapplyatransdisciplinary

methodologytobothAIvaluealignmentandtothe

SDGchallengetoavoidblindspotsandanegligent

approachtofutureglobalchallenges.Inthefollow-

ing,wecommentontheimportanceofSDG4forAI

governanceandfinallylinkittoSDG16onpeace,

justiceandstronginstitutions.

Wethinkthateducationandlife-longlearning–

egtransdisciplinaryfurthereducationforAISafety

andAIresearchersaswellasforauthoritiesinvolved

inAIregulation,andeducationfosteringanaware-

nessofsocio-technologicalchallengesforthegener-

alpublic–arehighlypowerfultoolsforbothchal-

lenges.First,itprovidesabasisforthegenerationof

novelapproachestoAIgovernance.Infact,while

somepeoplebelievethatthegoalinAIgovernance

shouldbetoachieveaconsensus,abroadvariation

ofscientificapproachesrepresentsanidealbreeding

groundforprogress.Second,aproactiveAIgover-

nanceapproachisnotenoughduetoerrorsand

changesinhumanvaluesthatwilloccur,which

meansthatonecannotsolelyrelyoncurrentstrate-

gies.Thus,itwillbeconvenienttoaccumulatebroad

knowledgethatmightbehelpfulinthefaceofnov-

elunpredictedproblemsthatarise.AnyAIgover-

nanceapproachthereforeneedstobeupdatableby

designinordertoallowacorrectivesocio-technolog-

icalfeedback-loop.Unfortunately,theSDGframe-

workisnotmeanttobesteadilyupdatedwhichrep-

resentsaclearlimitationthatshouldbethoroughly

takenintoconsiderationwhenattemptingtoachieve

itsfixedgoals.Forinstance,newunforeseeablechal-

lengesmayberelatedtodevelopmentsinAIitself

(andothernewtechnologies)ascanbeseenwhen

consideringthecurrentSDGtarget8.5,whichaims

to‘achievefullandproductiveemploymentandde-

centworkforallwomenandmen’–whichagainst

thebackgroundoftechnologicaladvancesmightbe

neitherrealisticnorworthwhileanymore37.Third,

aneducationofthegeneralpublicmightbeimpor-

tant,sincemanypeopleexhibitethicalbiasesbased

onincorrectassumptions.IntheAVcase,thiscould

forinstanceincludeanthropomorphism,presumed

levelofintentionalityandagencyormisconceptions

onthefunctioningofAVs.38Theseepistemicgaps

canbeaddressedviaamorein-deptheducationlead-

ingtoamoreinformedexperienceandethicaldebi-

asingwhichrespectsthemanifestationofmoralplu-

ralismknownfrompsychology .39Overall,webelieve

thatascientificallygroundedapproachtoAIgover-

nancesupplementedbyeducationisabsolutelynec-

essarygivenfuturechallenges.However,wewantto

re-emphasisethatwithoutstronginstitutionsascap-

turedinSDG16whichwetermedanimportantmeta-

goalforAIvaluealignment,thementionedstrategies

wouldbehighlylimitedintheirfieldofaction.On

theotherhand,failingtoaddressAIgovernance

couldleadtoAISafetyriskswithnegativerepercus-

sionstotheSDGframeworkrangingforinstance

34NiheerDasandiandSlavaJankin.Mikhaylov,‘ AIforSDG-16on

Peace,Justice,andStrongInstitutions:TrackingProgressand

AssessingImpact’(2019)PositionPaperfortheIJCAIWorkshopon

ArtificialIntelligenceandUnitedNationsSustainableDevelop-

mentGoals

35(n1)

36SoenkeZiesche,‘InnovativeBigDataApproachesforCapturing

andAnalyzingDatatoMonitorandAchievetheSDGs’(2017)

ReportoftheUnitedNationsEconomicandSocialCommission

forAsiaandtheP acific:SubregionalOfficeforEastandNorth-

EastAsia(ESC AP-ENEA)

37(n1)

38(n6)

39(n13)

Delphi4|2019233 SustainableAISafety?

fromcompromisinghumanwell-beingtoexistential

risksinsomecases40.

IV .ConclusionandFutureProspects

Overall,onecanconcludethatitisexpedienttoem-

bracetheSDGsandtheirgeneralintentionasacom-

plementaryfoundationfortheAIV alueAlignment

problem,yetoneneedstoacknowledgegivenlimita-

tionsincludingtheneedforarevised/specialversion

oftheindicatorstobecomefit-for-purpose.Against

thebackgroundofouranalysis,onecanestablishthat

theSDGframeworkexhibitstwomainweaknesses

whenappliedtotheAIvaluealignmentchallenge.

First,theSDGsdonotmentionartificialintelligence

atall,neitheritssignificantopportunities,noritssig-

nificantrisks,althoughbothweretoanextentknown

atthetimewhentheSDGswereformulated.Onerea-

sonforthisisthatthesediscussionsweresiloedin

academiccircles,andonlyrecentlythe(noweven

moreurgent)needforAIGovernancehasbeenac-

knowledged41.Second,humanchallengesandvalues

changeovertimeandunforeseeablefactorsmight

emerge,whiletheSDGshavenomechanismforan

amendmentuntil2030,whichisonlyjustifiedby

pragmaticreasons.Thiscanbealsoillustratedbythe

predecessoroftheSDGs,theMillenniumDevelop-

mentGoals,whichhadpartlydifferentambitions.

Importantly,theaboveissuesareintertwined.Forex-

ample,newunforeseeablechallengesmayaswellbe

relatedtodevelopmentsinAIitselfandothernew

technologies.

AsstatedbyKarlPopper,‘nosocietycanpredict,

scientifically ,itsownfuturestatesofknowledge.’42

Hence,AISafetycannotbeguaranteedtobesustain-

ableinthelongrunnorwillthegoalspursuedbythe

UNnecessarilyremainunchanged.Nevertheless,we

believethatitisasustainabletransdisciplinarysci-

entificapproachthatoneshouldstriveforinorder

toefficientlytackleAIGovernanceandexploitthe

describedbeneficialsynergieswiththeSDGs.Forse-

curityandsafety,oneneedsrequisiteknowledgeat

therighttime.Forthisreason,onecanarguethatthe

SDG4onqualityeducationandlife-longlearningcon-

tainsakeyelement.However,inthelightoftheabove,

itseemsimperativetoadditionallyaspiretoacorrec-

tivesocio-technologicalfeedback-loopenablingboth

proactiveandreactivemeasuresandforwhichSDG

16onstronginstitutionsrepresentsapre-condition.

40SoenkeZiesche,AI&GlobalGovernance:ASeatattheNegotiat-

ingTableforAI?OpportunitiesandRisks,UnitedNationsUniver-

sity2August2019<https://cpr .unu.edu/a-seat-at-the-negotiating

-table.html>accessed17October2019

41AllanDafoe,AIGovernance:AResearchAgenda’Governanceof

AIProgram,FutureofHumanityInstitute(UniversityofOxford

2018)

42KarlR.Popper,ThePovertyofHistoricism(Routledge1957)

era.int

Artiﬁ cial Intelligence (AI)

and Human Rights

Brussels, 12-13 March 2020

Key topics

• AI technology

• Issue of responsibility and transparency of AI systems

• European Commission Ethics Guidelines for

Trustworthy AI

• Relevant ECtHR case law

• Council of Europe Guidelines on Artiﬁ cial Intelligence

and Data Protection

• Impact of AI on selected human rights

Event number: 420D77 ●Language: English

Artiﬁ cial Intelligence (AI)

and the Criminal Justice

System

London, 4-5 June 2020

Key topics

• AI and its implications for the criminal justice system

• Using AI in criminal intelligence

• AI to predict crimes

• The internet industry approach to AI

• AI, ethics and privacy

Event number: 320D61 ●Language: English

Summer Course on

European Information

Technology Law

Trier, 14-18 September 2020

Key topics

• Internet regulation in the European Union

• Role, responsibility and accountability of online

platforms

• Regulating e-commerce in Europe

• Information technologies and intellectual property law

• EU legal framework applicable to data: sui generis

protection of databases, trade secrets protection, free

ﬂ ow of data, access to data, data privacy in the digital

world, data concentration

• Online and smart contracts

• Private international law and information technologies

• Digital identity and electronic payments

• Cybersecurity

Visit to the European Court of Justice

Participants in this summer course will also have

the opportunity to attend a hearing at the CJEU in

Luxembourg.

Who should attend?

Practitioners seeking an introduction to European IT law

Event number: 220B17 ●Language: English

For more information, please visit our website

www.era.int

Academy of European Law

Contact

Academy of European Law

Philipp Gonder

Tel. +49 (0) 651 937 37-811

E-mail pgonder@era.int

www.era.int

February

27 – 28 Fundamentals of State Aid Law | Amsterdam

27 – 28 Public Procurement Requirements for ESI Funds | Paris

March

03 – 04 State Aid Requirements for Services of General Economic Interest | Brussels

12 – 13 State Aid Procedures - The Legal Obligations and Rights of the Parties Involved | Lisbon

19 – 20 How to Most Effectively Use Technical Assistance for ESI Funds Now and in 2021-2027 | Florence

25 – 26 Indicators, Monitoring and Evaluation in ESIF Programme and Project Management | Milan

25 – 27 Spring Course: Public Procurement from A-Z | Milan

26 – 27 Programme and Financial Management of ESI Funds 2021 – 2027 | Nice

26 – 27 Effective Programming and Implementation of Financial Instruments | Malaga

30.3.–1.4. Spring Course: Simplified Cost Options for ESI Funds | Catania

April

01 – 03 Real-life On-the-spot Visits: How to Detect and Combat Irregularities & Fraud in ESI Funds | Maastricht

02 – 04 Verwaltung und Prüfung der Europäischen Struktur- und Investitions-Fonds (in German) | Hamburg

02 – 04 EStAL Seminar | Catania

22 – 24 Spring Course: Concept & Principles of State Aid | Athens

22 – 24 Advanced Spring Course: Innovative Approaches to Risk Analysis, Verifications and Audits in ESI Funds | Nice

23 – 24 Migration and Security Funds: How to Effectively Manage, Audit and Control AMIF & ISF Now and in 2021-2027 | Nice

23 – 24 Essentials of e-Procurement | Rome

28 – 29 State Aid for Agriculture, Forestry, and Rural Areas | Brussels

May

12 – 13 State Aid for Research, Development and Innovation Projects | Brussels

14 – 15 How to Conduct Management Verifications Most Effectively | Bologna

27 – 29 Summer Course: Financial Control and Audit in ESI Funds Now and in 2021-2027 | Sicily

28 – 29 Risk Management and Anti-Fraud Game of Jo Kremers (Edition 2020) | Sicily

28 – 29 Essentials of Public Procurement | Florence

June

09 – 10 State Aid: Assessment and Evaluation | Brussels

09 – 10 How to Successfully Manage, Control and Audit IPA Funds | Amsterdam

11 – 12 ESI Funded Agriculture Programmes and Projects | Catania

11 – 12 EStALI Interactive Forum on EU State Aid Law | Brussels

17 – 19 Summer Course: Irregularities and Fraud in ESI Funds and Public Procurement | Athens

17 – 19 Summer Course: Effective Usage of EU Financial Instruments Now and in 2021-2027 | Lake Como

18 – 19 Master Class: Probity and Public Procurement | Madrid

30.6. – 1.7. Master Class - “State Aid Uncovered” with Prof. Dr. Phedon Nicolaides | Florence

Events Spring/Summer 2020

For more information

and registration please

visit our website

www.lexxion.eu

NEW

European State Aid Law

European Structural and Investment Funds

European Procurement and Public Private Partnership Law

ResearchGate has not been able to resolve any citations for this publication.

Predicting future AI failures from historic examples

Article

Full-text available

Nov 2018
Foresight

Roman Yampolskiy

Purpose The purpose of this paper is to explain to readers how intelligent systems can fail and how artificial intelligence (AI) safety is different from cybersecurity. The goal of cybersecurity is to reduce the number of successful attacks on the system; the goal of AI Safety is to make sure zero attacks succeed in bypassing the safety mechanisms. Unfortunately, such a level of performance is unachievable. Every security system will eventually fail; there is no such thing as a 100 per cent secure system. Design/methodology/approach AI Safety can be improved based on ideas developed by cybersecurity experts. For narrow AI Safety, failures are at the same, moderate level of criticality as in cybersecurity; however, for general AI, failures have a fundamentally different impact. A single failure of a superintelligent system may cause a catastrophic event without a chance for recovery. Findings In this paper, the authors present and analyze reported failures of artificially intelligent systems and extrapolate our analysis to future AIs. The authors suggest that both the frequency and the seriousness of future AI failures will steadily increase. Originality/value This is a first attempt to assemble a public data set of AI failures and is extremely valuable to AI Safety researchers.

Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR

Article

Full-text available

Apr 2018
Harv J Law Tech

There has been much discussion of the right to explanation in the EU General Data Protection Regulation, and its existence, merits, and disadvantages. Implementing a right to explanation that opens the black box of algorithmic decision-making faces major legal and technical barriers. Explaining the functionality of complex algorithmic decision-making systems and their rationale in specific cases is a technically challenging problem. Some explanations may offer little meaningful information to data subjects, raising questions around their value. Explanations of automated decisions need not hinge on the general public understanding how algorithmic systems function. Even though such interpretability is of great importance and should be pursued, explanations can, in principle, be offered without opening the black box. Looking at explanations as a means to help a data subject act rather than merely understand, one could gauge the scope and content of explanations according to the specific goal or action they are intended to support. From the perspective of individuals affected by automated decision-making, we propose three aims for explanations: (1) to inform and help the individual understand why a particular decision was reached, (2) to provide grounds to contest the decision if the outcome is undesired, and (3) to understand what would need to change in order to receive a desired result in the future, based on the current decision-making model. We assess how each of these goals finds support in the GDPR. We suggest data controllers should offer a particular type of explanation, unconditional counterfactual explanations, to support these three aims. These counterfactual explanations describe the smallest change to the world that can be made to obtain a desirable outcome, or to arrive at the closest possible world, without needing to explain the internal logic of the system.

The Theory of Dyadic Morality: Reinventing Moral Judgment by Redefining Harm

Article

Full-text available

May 2017

The nature of harm—and therefore moral judgment—may be misunderstood. Rather than an objective matter of reason, we argue that harm should be redefined as an intuitively perceived continuum. This redefinition provides a new understanding of moral content and mechanism—the constructionist Theory of Dyadic Morality (TDM). TDM suggests that acts are condemned proportional to three elements: norm violations, negative affect, and—importantly—perceived harm. This harm is dyadic, involving an intentional agent causing damage to a vulnerable patient (A→P). TDM predicts causal links both from harm to immorality (dyadic comparison) and from immorality to harm (dyadic completion). Together, these two processes make the “dyadic loop,” explaining moral acquisition and polarization. TDM argues against intuitive harmless wrongs and modular “foundations,” but embraces moral pluralism through varieties of values and the flexibility of perceived harm. Dyadic morality impacts understandings of moral character, moral emotion, and political/cultural differences, and provides research guidelines for moral psychology.

Evidence for a Large-Scale Brain System Supporting Allostasis and Interoception in Humans

Article

Full-text available

Apr 2017
Nat. Hum. Behav.

Large-scale intrinsic brain systems have been identified for exteroceptive senses (such as sight, hearing and touch). We introduce an analogous system for representing sensations from within the body, called interoception, and demonstrate its relation to regulating peripheral systems in the body, called allostasis. Employing the recently introduced Embodied Predictive Interoception Coding (EPIC) model, we used tract-tracing studies of macaque monkeys, followed by two intrinsic functional magnetic resonance imaging samples (N = 280 and N = 270) to evaluate the existence of an intrinsic allostatic–interoceptive system in the human brain. Another sample (N = 41) allowed us to evaluate the convergent validity of the hypothesized allostatic–interoceptive system by showing that individuals with stronger connectivity between system hubs performed better on an implicit index of interoceptive ability related to autonomic fluctuations. Implications include insights for the brain’s functional architecture, dissolving the artificial boundary between mind and body, and unifying mental and physical illness.

Canary in the e-Commerce Coal Mine: Detecting and Predicting Poor Experiences Using Buyer-to-Seller Messages

Conference Paper

Full-text available

Jun 2015

Reputation and feedback systems in online marketplaces are often biased, making it difficult to ascertain the quality of sellers. We use post-transaction, buyer-to-seller message traffic to detect signals of unsatisfactory transactions on eBay. We posit that a message sent after the item was paid for serves as a reliable indicator that the buyer may be unhappy with that purchase, particularly when the message included words associated with a negative experience. The fraction of a seller's message traffic that was negative predicts whether a buyer who transacts with this seller will stop purchasing on eBay, implying that platforms can use these messages as an additional signal of seller quality.

Promotional Reviews: An Empirical Investigation of Online Review Manipulation

Article

Full-text available

Aug 2012

Firms' incentives to manufacture biased user reviews impede review usefulness. We examine the differences in reviews for a given hotel between two sites: Expedia.com (only a customer can post a review) and TripAdvisor.com (anyone can post). We argue that the net gains from promotional reviewing are highest for independent hotels with single-unit owners and lowest for branded chain hotels with multiunit owners. We demonstrate that the hotel neighbors of hotels with a high incentive to fake have more negative reviews on TripAdvisor relative to Expedia; hotels with a high incentive to fake have more positive reviews on TripAdvisor relative to Expedia.

Artificial Intelligence and the Modern Productivity Paradox: A Clash of Expectations and Statistics

Chapter