ArticlePDF Available

Abstract

Visual neuroscience benefits from high-quality datasets with neuronal responses to many images. Several neuroimaging datasets were published in recent years, but no comparable dataset with spiking activity exists. Here, we introduce the THINGS Ventral-stream Spiking Dataset (TVSD). We extensively sampled neuronal activity in response to >25,000 natural images from the THINGS database in macaques, using high-channel-count implants in three key cortical regions: V1, V4, and the inferotemporal cortex. We showcase the utility of TVSD by using an artificial neural network to visualize the tuning of neurons. We also characterize the correlated fluctuations in activity within and between areas and demonstrate that these noise correlations are strongest between neurons with similar tuning. The TVSD allows researchers to answer many questions about neuronal tuning, the interactions within and between cortical regions and to compare spiking activity in monkeys to human neuroimaging data.
An extensive dataset of spiking ac2vity to reveal the syntax of the
!
"
ventral stream
#
"
"
$
"
%&'('"%&)&(*1,*,#+",*-."/&-.1+"0&12*3"/4"5*(61 &-7"%8*9*:";4";'*(6<*=&1,2,3,4,*"
>
"
?
"
1. Department of Vision & Cognition, Netherlands Institute for Neuroscience (KNAW), 1105 BA
@
"
Amsterdam, Netherlands.
A
"
2. Department of Integrative Neurophysiology, VU University, De Boelelaan 1085, 1081 HV
B
"
Amsterdam, Netherlands.
C
"
3. Department of Neurosurgery, Academic Medical Centre, Postbus 22660, 1100 DD Amsterdam,
!D
"
Netherlands.
!!
"
4. Laboratory of Visual Brain Therapy, Sorbonne Université, INSERM, CNRS, Institut de la Vision, 17
!#
"
rue Moreau, F-75012 Paris, France,
!$
"
"
!>
"
E"F'::*<)'-7*-F*"<2'G(7"H*"&77:*<<*7"9'I"p.roelfsema@nin.knaw.nl, p.papale@nin.knaw.nl
!?
"
J4"K*&7"F'-9&F9"
!@
"
!A
"
!B
"
LG92':"M*:<8'-"
"
#"
Summary
!C
"
N8<G&("-*G:'<F8*-F*"H*-*689<"6:'="28.2OPG&(89Q"7&9&<*9<"3892"-*G:'-&(":*<)'-<*<"9'"=&-Q"
#D
"
8=&.*<4"5*M*:&("-*G:'8=&.8-."7&9&<*9<"3*:*")GH(8<2*7"8-":*F*-9"Q*&:<+"HG9"-'"F'=)&:&H(*"
#!
"
7&9&<*9"3892"<)8R8-."&F98M89Q"*S8<9<4"T*:*+"3*"8-9:'7GF*"92*"UTVWX5"N*-9:&(O<9:*&="5)8R8-."
##
"
Y&9&<*9"ZUN5Y[4"/*"*S9*-<8M*(Q"<&=)(*7"-*G:'-&("&F98M89Q"8-":*<)'-<*"9'"\#?+DDD"-&9G:&("
#$
"
8=&.*<"6:'="92*"UTVWX5"7&9&H&<*"8-"=&F&PG*<+"G<8-."28.2OF2&--*(OF'G-9"8=)(&-9<"8-"92:**"
#>
"
R*Q"F':98F&(":*.8'-<I"N!+"N>+"&-7"92*"8-6*:'9*=)':&("F':9*S4"/*"<2'3F&<*"92*"G98(89Q"'6"UN5Y"
#?
"
HQ"G<8-."&-"&:9868F8&("-*G:&("-*93':R"9'"M8<G&(8]*"92*"9G-8-."'6"-*G:'-<4"/*"&(<'"F2&:&F9*:8]*"
#@
"
92*"F'::*(&9*7"6(GF9G&98'-<"8-"&F98M89Q"38928-"&-7"H*93**-"&:*&<"&-7"7*='-<9:&9*"92&9"92*<*"
#A
"
-'8<*"F'::*(&98'-<"&:*"<9:'-.*<9"H*93**-"-*G:'-<"3892"<8=8(&:"9G-8-.4"U2*"UN5Y"&(('3<"
#B
"
:*<*&:F2*:<"9'"&-<3*:"=&-Q"PG*<98'-<"&H'G9"-*G:'-&("9G-8-.+"92*"8-9*:&F98'-<"38928-"&-7"
#C
"
H*93**-"F':98F&(":*.8'-<"&-7"9'"F'=)&:*"<)8R8-."&F98M89Q"8-"='-R*Q<"9'"2G=&-"
$D
"
-*G:'8=&.8-."7&9&4"
$!
"
"
$#
"
Keywords
$$
"
^)*-"Y&9&+"N8<8'-+"5)8R8-."&F98M89Q+"N*-9:&("<9:*&=+"W&9G:&("8=&.*<"
$>
"
LG92':"M*:<8'-"
"
$"
Introduction
$?
"
0&R8-."<*-<*"'6"&"M8<G&("<F*-*"8<"&"F'=)(*S"&-7"F'=)G9&98'-&((Q"8-9*-<8M*"):'F*<<+"
$@
"
&-7"89<"-*G:'-&("6'G-7&98'-<"8-"'G:"H:&8-<":*=&8-"6&:"6:'="H*8-."G-7*:<9''74"W*G:'<F8*-98<9<"
$A
"
2&M*"9:&7898'-&((Q"&)):'&F2*7"928<"):'H(*="HQ"8-M*<98.&98-."2'3"-*G:'-<"8-"M&:8'G<"F':98F&("
$B
"
:*.8'-<"'6"92*"M*-9:&("M8<G&("<9:*&=+"8-M'(M*7"8-"'H_*F9":*F'.-898'-+":*<)'-7"9'"6*&9G:*<"'6"92*"
$C
"
M8<G&("3':(7"Z*4.4"F'(':"&-7"':8*-9&98'-[+"G<8-."F'-9:'((*7"<98=G(81,24"0':*":*F*-9(Q+"2'3*M*:+"
>D
"
<F8*-98<9<"2&M*"&(<'"<9&:9*7"9'"8-F':)':&9*"-'M*(+"7&9&O7:8M*-"&)):'&F2*<"9'"F&)9G:*"-*G:'-&("
>!
"
:*):*<*-9&98'-<" '6" 92*" M*-9:&(" <9:*&=+" =*&<G:8-." H:&8-" :*<)'-<*<" 9'" (&:.*" F'((*F98'-<" '6"
>#
"
*F'('.8F&("<98=G(834"U28<"&)):'&F2" .&8-*7" )')G(&:89Q" 7G*"9'"92*"&<9'G-78-." )*:6':=&-F*" '6"
>$
"
F'=)G9&98'-&("='7*(<+" 7*M*(')*7" 8-" 92*" &:9868F8&("8-9*((8.*-F*" ZLV[" 68*(7+" 8-" ):*78F98-."H:&8-"
>>
"
&F98M89Q4–74"U2*<*"-*3"F'=)G9&98'-&("='7*(<"7*)*-7"'-"28.2OPG&(89Q"&-7"(&:.*O<F&(*"7&9&<*9<"
>?
"
3892" -*G:'-&(" :*<)'-<*<" 9'" 92'G<&-7<" '6" 8=&.*<2,8124" V-" :*F*-9" Q*&:<+" <*M*:&(" (&:.*O<F&(*"
>@
"
-*G:'8=&.8-."7&9&<*9<"2&M*"H**-":*F':7*7"&-7")GH(8F(Q"<2&:*7+"&(('38-.":*<*&:F2*:<"&:'G-7"
>A
"
92*"3':(7"9'"&77:*<<"92*8:":*<*&:F2"PG*<98'-<"G<8-."7&9&"'6"28.2"PG&(89Q8–114
>B
"
" U2*" *S8<98-." -*G:'8=&.8-." 7&9&<*9<" '66*:" *S9*-<8M*" H:&8-" F'M*:&.*" &-7" <98=G(G<"
>C
"
M&:8&H8(89Q+"HG9"9'"7**)*-"'G:"G-7*:<9&-78-."'6"92*"<Q-9&S"'6"92*"M*-9:&("<9:*&=+"89"8<"G<*6G("9'"
?D
"
8-M*<98.&9*" 92*" <)8R8-." &F98M89Q" &-7" 9G-8-." '6" -*G:'-<" G<8-." 8-9:&F':98F&(" *(*F9:'7*<+" 328F2"
?!
"
):'M87*" &" 28.2*:" <)&98'9*=)':&(" :*<'(G98'-" 92&-" -*G:'8=&.8-." 9*F2-8PG*<4" 5GF2" &" 7&9&<*9"
?#
"
7'*<"-'9"Q*9"*S8<9+"):*<G=&H(Q"H*F&G<*"89"8<"F2&((*-.8-."9'"<&=)(*"-*G:'-&(":*<)'-<*<"&F:'<<"
?$
"
*-'G.2"8=&.*<"&-7"-*G:'-<+"8-"=G(98)(*"H:&8-":*.8'-<"&9"'-F*+"&F:'<<"=G(98)(*"7&Q<4"T*:*"3*"
?>
"
'M*:F&=*" 92*<*" F2&((*-.*<" HQ" F2:'-8F&((Q" )(&F8-." =&-Q" `9&2" &::&Q<" 8-" R*Q" :*.8'-<" '6" 92*"
??
"
M*-9:&("<9:*&="G<8-."FG<9'=O=&7*"8=)(&-9<4"
?@
"
LG92':"M*:<8'-"
"
>"
/*"):*<*-9"92*"UTVWX5"N*-9:&("<9:*&="5)8R8-."Y&9&<*9"ZUN5Y[+"H&<*7"'-"(&:.*O<F&(*"
?A
"
*(*F9:')2Q<8'('.Q"*S)*:8=*-9<+"328F2"):'M87*<")GH(8F"&FF*<<"9'"<)8R8-.":*<)'-<*<"6:'="M8<G&("
?B
"
-*G:'-<"*(8F89*7"HQ"=&-Q"8=&.*<4"U2*"UTVWX5"7&9&H&<*"8<"&-"*S9*-<8M*"&-7"F&:*6G((Q"FG:&9*7"
?C
"
<*9"'6"-&9G:&(8<aF"8=&.*<+"6':" 328F2" 2G=&-" _G7.=*-9<"'6"F&9*.':Q"=*=H*:<28)"2&M*"H**-"
@D
"
&-&(Q]*7"<Q<9*=&aF&((Q124"/*"<&=)(*7"92*"=G(98OG-89"<)8R8-."&F98M89Q"Z0`L["8-"93'"=&F&PG*<"
@!
"
<**8-."=':*"92&-"#?+DDD"-&9G:&("8=&.*<"6:'="92*"UTVWX5"7&9&H&<*"G<8-."28.2OF2&--*(OF'G-9"
@#
"
8=)(&-9<"3892"!+D#>"=8F:'*(*F9:'7*<"8-"92:**"R*Q":*.8'-<"'6"92*"M*-9:&("<9:*&=I"92*"):8=&:Q"
@$
"
M8<G&(" F':9*S" ZN![+" &:*&" N>+" &-7" 92*" 8-6*:'9*=)':&(" F':9*S" ZVU[4" /*" ):'M87*" 92*" :&3" 7&9&"
@>
"
<&=)(*7"&9"$DRT]"&-7"92*"):*O):'F*<<*7"0`L+"328F2"8<"*&<Q"9'"G<*"HQ":*<*&:F2*:<"3892"-'"
@?
"
):8':"R-'3(*7.*"'6"8-9:&F':98F&(":*F':78-.<4"U'"7*='-<9:&9*"92*")'9*-98&("'6"92*"7&9&"<*9+"3*"
@@
"
<2'3F&<*"&"='7*(8-."&)):'&F2"H&<*7"'-"92*"):*O):'F*<<*7"0`L"&-7"&-&(Q]*"92*")&99*:-<"'6"
@A
"
F'::*(&9*7"6(GF9G&98'-<"8-"92*"68:8-.":&9*<"&F:'<<"N!+"N>"&-7"VU4"
@B
"
/*"*S)*F9"92&9"92*"UN5Y"38(("&(('3"<F8*-98<9<"9'"9*<9"92*':8*<"'-"92*"=*F2&-8<=<"6':"
@C
"
-&9G:&(" M8<8'-+" .'8-." H*Q'-7" 32&9" 2&<" H**-" )'<<8H(*" 3892" 92*" *S8<98-." 7&9&<*9<4" b*F&G<*"
AD
"
UTVWX5"3&<" G<*7"8-"<*M*:&("2G=&-"H*2&M8':&("&-7"-*G:'8=&.8-.")GH(8F"7&9&<*9<+"92*"UN5Y"
A!
"
38((" *-&H(*" G-):*F*7*-9*7" F'=)&:8<'-<" H*93**-" <)8R8-." 7&9&" 'H9&8-*7" 8-" ='-R*Q<" &-7"
A#
"
-*G:'8=&.8-."&-7"H*2&M8':&("7&9&"8-"2G=&-<4 ,8-&((Q+"92*"<8]*"'6"92*"UN5Y"F'G(7"G-('FR"92*"
A$
"
)'<<8H8(89Q"'6"9:&8-8-."LV"='7*(<"78:*F9(Q"'-"H:&8-"7&9&4"U2*"6G(("7&9&<*9"8<"&M&8(&H(*"'-"XOW'7*"
A>
"
XVW"&9"92*"(8-RI"299)<Icc7'84':.c!D4!#A?!c.O-'7*42FA](M4"
A?
"
A@
"
AA
"
Results
AB
"
LG92':"M*:<8'-"
"
?"
High-channel-count neuronal responses in the ventral stream to thousands of images"
AC
"
/*"8=)(&-9*7"!?"':"!@"`9&2"&::&Q<"8-"93'"=&F&PG*"='-R*Q<13,144"d&F2"&::&Q"2&7"*8.29"
BD
"
:'3<"&-7"F'(G=-<"'6"!4?O==O('-."<2&-R<"Z,8.G:*"!L+",8.G:*"5![4"/*"8=)(&-9*7"&"9'9&("'6"!?"
B!
"
&::&Q<"8-"N!"ZA"8-"='-R*Q"W"&-7"B"8-"='-R*Q",e"C@D"*(*F9:'7*<"8-"9'9&([+"A"&::&Q<"8-"N>"Z>"8-"
B#
"
='-R*Q"W"&-7"$"8-"='-R*Q",e">>B"*(*F9:'7*<"8-"9'9&(["&-7"C"&::&Q<"8-"VU"Z>"8-"='-R*Q"W"&-7"?"
B$
"
8-"='-R*Q",e"?A@"*(*F9:'7*<"8-"9'9&([4""
B>
"
/*"):*<*-9*7"#?+#>B"-&9G:&("8=&.*<"6:'="92*"UTVWX5"7&9&H&<*12"*-<G:8-."92&9"92*Q"
B?
"
F'M*:*7"92*":*F*)98M*"68*(7<"Z;,<["'6"92*"-*G:'-<+"7G:8-."&"9&<R"8-"328F2"92*"='-R*Q<"78:*F9*7"
B@
"
92*8:".&]*"9'"&"68S&98'-")'8-94"U2*"8=&.*<"3*:*"78M87*7"8-9'"&"9:&8-8-."&-7"&"9*<9"<*9+"HG9"92*8:"
BA
"
):*<*-9&98'-"3&<":&-7'=8]*7"&-7"8-9*:(*&M*74"U2*"9:&8-8-."<*9"<)&--*7"!+B?>"<Q<9*=&98F&((Q"
BB
"
FG:&9*7" <*=&-98F" F&9*.':8*<+" 3892" !#" 8=&.*<" )*:" F&9*.':Q+" 328F2" 3*:*" <2'3-" '-F*4"U2*"
BC
"
9:&8-8-."<*9"38((" &(('3"G<*:<"9'"<9G7Q" 92*"-*G:'-&(":*<)'-<*<"9'" &"387*":&-.*"'6" <a=G(8"&-7"
CD
"
F'=)&:*" 92*=" 9'" 92'<*" ):*78F9*7" HQ" ='7*(<" '6" F':aF&(" ):'F*<<8-.+" 8-F(G78-." 7**)" -*G:&("
C!
"
-*93':R<4"U2*"9*<9"<*9"F'=):8<*7"!DD"8=&.*<"92&9"3*:*"<2'3-"$D"98=*<"*&F2"Z,8.G:*"!b[4"U2*"
C#
"
)G:)'<*"'6"92*"9*<9"<*9"3&<"8["9'"F'=)&:*":*<)'-<*<"*(8F89*7"HQ"92*"<&=*"8=&.*<"&F:'<<"7&Q<7"
C$
"
Z*4.4"89"8<")'<<8H(*"9'"-':=&(8]*":*<)'-<*<"9'"92*<*"8=&.*<"&F:'<<"<*<<8'-<[+"88["9'"8-F:*&<*"92*"
C>
"
<8.-&(O9'O-'8<*":&98'"Z5W;["&-7"888["9'"*-&H(*"F:'<<OM&(87&98'-14"
C?
"
L"9:8&("<9&:9*7"3892"92*"):*<*-9&98'-"'6"&"68S&98'-")'8-9+"&":*7"7'9"'-"&"G-86':=".:&Q"
C@
"
H&FR.:'G-74"^-F*"92*"='-R*Q<"&99&8-*7"68S&98'-"6':"$DD=<+"3*"):*<*-9*7"&"<*PG*-F*"'6"6'G:"
CA
"
8=&.*<+" ):*<*-9*7" 6':" #DD=<" &-7" 6'(('3*7" HQ" #DD=<" 3892" &" .:&Q" <F:**-" Z,8.G:*" !f[4"U2*"
CB
"
='-R*Q<"3*:*":*3&:7*7" 3892" &"7:')"'6" _G8F*" 86"92*Q" =&8-9&8-*7" 68S&98'-" 7G:8-."92*" 32'(*"
CC
"
<*PG*-F*4"/*":*F':7*7"6'G:"<*<<8'-<"8-"*&F2"='-R*Q4"U2*"<*<<8'-<"<)&--*7"A"7&Q<"8-"='-R*Q"
!DD
"
,"3892"@+C?#"9:8&(<"&-7"#?+#>B"8=&.*"):*<*-9&98'-<+"&-7">"F'-<*FG98M*"7&Q<"8-"='-R*Q"W"3892"
!D!
"
@+AA!"9:8&(<"&-7"#?+#>B" 8=&.*" ):*<*-9&98'-<4" ,8.G:*"!Y"<2'3<"92*" :&3" *(*F9:')2Q<8'('.8F&("
!D#
"
LG92':"M*:<8'-"
"
@"
<8.-&(+" <&=)(*7" &9" $DRT]+" 6:'=" &-" *S&=)(*" 9:8&(" &9" &" :*):*<*-9&98M*" *(*F9:'7*" 8-" N!" 6:'="
!D$
"
='-R*Q"W4",':"'G:"&-&(Q<8<"3*"38(("2*:*"6'FG<"'-"0`L+"328F2":*):*<*-9<"92*"<)8R8-."&F98M89Q"
!D>
"
'6"<*M*:&("-*G:'-<"8-"92*"M8F8-89Q"'6"92*"*(*F9:'7*"Z0*92'7<[4""
!D?
"
0`L")')G(&a'-":*<)'-<*<"&:*"M*:Q"<8=8(&:"9'"92'<*"'H9&8-*7"HQ")''(8-."&F:'<<"<8-.(*"
!D@
"
G-89<"&-7"2&M*"H**-"G<*7"8-"=&-Q"):*M8'G<"<9G78*<15184"U:&G9=&-"*9"&(4"Z:*6419["7*='-<9:&9*7"
!DA
"
92&9"-*G:'-&("7Q-&=8F<" &:*" <8=8(&:" 32*-"&-&(Q<8-." =G(aG-89" 7&9&"&-7"<8-.(*" -*G:'-<+" &-7"
!DB
"
92&9"<8=8(&:"<F8*-agF"F'-F(G<8'-<" F&-" H*" :*&F2*7"&H'G9")')G(&a'-O(*M*("&FaM89Q4"U2*" 0`L"
!DC
"
3&<"<9&H(*"38928-" &-7"&F:'<<"92*":*F':78-."<*<<8'-<+" H*F&G<*"92*"&M*:&.*"&F98M89Q"8-" N!+"N>"
!!D
"
&-7"VU"*(8F89*7"HQ"&(("8=&.*<"8-"H('FR<"'6"h#D"=8-"&F:'<<"&(("<*<<8'-<"3&<"<8=8(&:"Z,8.G:*"!d+,[4"
!!!
"
W*M*:92*(*<<+"G<*:<"'6"92*"UN5Y"F&-"&))(Q"'92*:"68(9*:<"9'"*4.4"*S9:&F9"92*"('F&("68*(7")'9*-98&("
!!#
"
6:'="92*" H:'&7OH&-7" <8.-&(4" U2*Q" F&-" &(<'" G<*"<)8R*O<':98-." &(.':892=<" 8<'(&9*"<8-.(*" G-89<"
!!$
"
Z*S&=)(*<"'6"<)8R*<"<':9*7"3892"/&M*F(G<$20"&:*"):*<*-9*7"8-",8.G:*"5#[4"
!!>
"
"
!!?
"
LG92':"M*:<8'-"
"
A"
"
!!@
"
!!A
"
,8.G:*"!4"5)8R8-."&F98M89Q"8-"92*"='-R*Q"M*-9:&("<9:*&="8-":*<)'-<*"9'"92'G<&-7<"'6"8=&.*<"
!!B
"
L["5F2*=&aF"'6"92*"&::&Q"('F&a'-<4"/*":*F':7*7"<)8R8-."&FaM89Q"6:'="$!"`9&2"&::&Q<"ZB"S"B"
!!C
"
*(*F9:'7*<"*&F2[" 8=)(&-9*7" 8-" 93'" =&F&PG*" ='-R*Q<4" ,8.G:*" 5!"<2'3<" )8F9G:*<" 92&9"3*:*"
!#D
"
9&R*-"7G:8-."<G:.*:Q"&-7"F&-"H*"G<*7"6':"92*"):*F8<*"7*9*:=8-&a'-"'6"&::&Q"('F&a'-<4"
!#!
"
b["/*"):*<*-9*7"#?+#>B"8=&.*<"6:'="92*"UTVWX5"8=&.*"7&9&H&<*12+"328F2"3*:*"78M87*7"8-9'"
!##
"
&"9:&8-8-."&-7"&"9*<9"<*94"U2*":*F9&-.(*"*-F('<*<"!+B?>"8=&.*<+"328F2"F'::*<)'-7<"9'"!c!#th"'6"
!#$
"
92*"6G(("9:&8-8-."<*94"
!#>
"
f["U2*"='-R*Q<"=&8-9&8-*7"gS&a'-"7G:8-."&"<*PG*-F*"'6"6'G:"8=&.*<"Z#DD=<"'-+"#DD=<"'i[4"
!#?
"
5GFF*<<6G("gS&a'-"7G:8-."92*"*-a:*"<*PG*-F*"3&<"6'(('3*7"HQ"92*"7*(8M*:Q"'6"&"_G8F*":*3&:74"
!#@
"
LG92':"M*:<8'-"
"
B"
Y["b:'&7"H&-7"<8.-&("&9"&":*):*<*-9&aM*"*(*F9:'7*"ZWDDDC["8-"N!"7G:8-."&-"*S&=)(*"9:8&("Z9')["
!#A
"
&-7"92*" ('3)&<<" K,%" &-7"H&-7)&<<" <)8R8-." F'=)'-*-9<"ZH'1'=e" )G:)(*I" !O!DDT]e" ':&-.*I"
!#B
"
D4?O?jT][4"U2*":&3"<8.-&("F'-9&8-<"28.2"6:*PG*-FQ"<)8R8-."&FaM89Q"&-7"<('3*:"K,%"F'=)'-*-9<4"
!#C
"
Y&<2*7"(8-*I"<a=G(G<"'-<*9e"<2&7*7".:&Q":*.8'-I"<a=G(G<"^W")*:8'74"
!$D
"
dO,["0`L"&9":*):*<*-9&aM*"*(*F9:'7*<"8-"='-R*Q"W"Zd["&-7"='-R*Q","Z,[+"&M*:&.*7"&F:'<<"&(("
!$!
"
92*"9:8&(<"8-"H('FR<"'6"h#D=8-4",:'="92*"9')I"&"N!"*(*F9:'7*"Z:*7[+"N>"*(*F9:'7*"ZH(G*["&-7"VU"
!$#
"
*(*F9:'7*"Z.:**-[4"U2*"*(*F9:'7*")'<8a'-"8-"92*"&::&Q"8<"<2'3-"'-"92*"(*k4"U2*"<=&(("M&:8&H8(89Q"
!$$
"
&F:'<<" :*<)'-<*" &M*:&.*<" 8((G<9:&9*<" 92*" <9&H8(89Q" '6" 92*" <8.-&(" 38928-" &-7" &F:'<<" <*<<8'-<4"
!$>
"
,8.G:*"5#"<2'3<"*S&=)(*":*<G(9<"6:'="&"<)8R*O<':a-."&(.':892=4"
!$?
"
"
!$@
"
Quality, reliability and selectivity of spiking activity in V1, V4 and IT"
!$A
"
"/*"&<<*<<*7"92*"PG&(89Q"'6"92*":*F':78-.<"HQ"=*&<G:8-."92*"5W;"&-7"92*":*(8&H8(89Q"'6"
!$B
"
92*" 0`L" &9" 8-78M87G&(" :*F':78-." <89*<4" /*" =*&<G:*7" 92*" 5W;"HQ" 78M878-." 92*" )*&R" '6" 92*"
!$C
"
&M*:&.*":*<)'-<*" *(8F89*7"HQ"&(("8=&.*<"HQ"92*"<9&-7&:7"7*M8&98'-"'6"92*"&F98M89Q"H*6':*"92*"
!>D
"
<98=G(G<"'-<*9"Z<**"0*92'7<[4"W&9G:&("8=&.*<"2&M*"&"('3*:"F'-9:&<9"&-7"28.2*:"M&:8&-F*"92&-"
!>!
"
92*" &:9868F8&(" <98=G(8" '6" =':*" <9&-7&:7" *S)*:8=*-9<" Z*4.4" F2*FR*:H'&:7<[" &-7" <'=*" '6" 92*"
!>#
"
8=&.*<"*(8F89*7"8-28H898'-4"U2*"5W;"8<"92*:*6':*"&"F'-<*:M&98M*"=*&<G:*+"HG9"89"*SF**7*7"#+"&"
!>$
"
F'-M*-98'-&("92:*<2'(7"6':"*(*F9:'7*"8-F(G<8'-1517+"6':"='<9":*F':78-."<89*<"Z,8.G:*"#Le"A>4$l"
!>>
"
8-"='-R*Q"We"?>4Dl"8-"='-R*Q",[4""
!>?
"
" U2*" 5W;" '6" 92*" UN5Y" 8<" F'-<87*:&H(Q" 28.2*:" 92&-" <9&9*O'6O92*O&:9" -*G:'8=&.8-."
!>@
"
7&9&<*9<+"<GF2"&<"92*"W&9G:&("5F*-*<"Y&9&<*9"ZW5Y8[4"U2*"&G92':<"'6"92*"W5Y"F'=)G9*7"5W;"
!>A
"
H&<*7" '-" 92*" M&:8&-F*" '6" :*<)'-<*<" *(8F89*7" HQ" 92*" <&=*" 8=&.*<" Zmncsnr’8[+" &-7" 928<" 5W;"
!>B
"
=*&<G:*"8<"@"98=*<"(&:.*:"6':"92*"UN5Y4"U2*Q"):')'<*7"92*"F'-F*)9"'6"m*PG8M&(*-9"9:8&(<n+"328F2"
!>C
"
8<"):')':98'-&("9'"92*"-G=H*:"'6"9:8&(<"&-7"92*"<PG&:*"'6"92*"5W;"'6"92*"7&9&"&-7"F&-"H*"G<*7"
!?D
"
9'"F'=)&:*"7&9&<*9<4"U2*"&G92':<"-'9*7"92&9"92*"W5Y"*S)&-7*7"92*"*PG8M&(*-9"9:8&("-G=H*:"
!?!
"
HQ"&"6&F9':"'6"h##"'M*:"):*M8'G<"3':R4"LFF':78-."9'"928<"F&(FG(&98'-+"92*"UN5Y"3'G(7"8-F:*&<*"
!?#
"
92*"-G=H*:"'6"*PG8M&(*-9"9:8&(<"HQ"&-"&77898'-&("6&F9':"'6"!D4"
!?$
"
LG92':"M*:<8'-"
"
C"
U'"=*&<G:*"92*":*(8&H8(89Q"'6"92*"-*G:'-&(":*<)'-<*<+"3*"G<*7"92*"':&F(*"F'::*(&98'-5+"
!?>
"
328F2"8<"92*" F'::*(&98'-" H*93**-"92*"&M*:&.*":*<)'-<*"*(8F89*7"HQ" #C" '6"92*"$D"9*<9"<98=G(8"
!??
"
3892" 92*" :*<)'-<*" 9'" 92*" '-*" <98=G(G<" :*)*9898'-" 92&9" 3&<" 2*(7" 'G9" Z<**" 0*92'7<[4" 0'<9"
!?@
"
F2&--*(<"2&7"&"<8.-868F&-9"':&F(*"F'::*(&98'-"Z)"o"D4D?+"9O9*<9+"b'-6*::'-8"F'::*F9*7[I"BA4$l"8-"
!?A
"
='-R*Q"W" &-7"A?4Dl"8-"='-R*Q","Z,8.G:*"#b[4"U2*"':&F(*"F'::*(&98'-" F&-"H*"8-9*:):*9*7"&<"
!?B
"
92*"-'8<*OF*8(8-.+"84*4"92*"=&S8=G="F'::*(&98'-"'6"&"='7*("&FFG:&9*(Q"):*78F98-."92*"-*G:'-&("
!?C
"
:*<)'-<*<"*(8F89*7" HQ" 8-78M87G&(" 8=&.*<4" V-9*:*<98-.(Q+" =&-Q" F2&--*(<"3892" ('3" 5W;<" 2&7" &"
!@D
"
28.2" ':&F(*" F'::*(&98'-" Z*4.4" 92*" *S&=)(*" :*F':78-." <89*" '-" 92*" :8.29" 8-" ,8.G:*" $L[4" U28<"
!@!
"
&))&:*-9"78<F:*)&-FQ" 8<" F&G<*7"HQ" 92*" 7*68-898'-"'6"5W;+" 328F2" 3&<"H&<*7" '-" 92*"&M*:&.*"
!@#
"
:*<)'-<*"&-7"787"-'9"9&R*"92*"-*G:'-&("9G-8-."':"92*"):*<*-F*"'6"8-28H89':Q":*<)'-<*<"Z,8.G:*"
!@$
"
$L+f["8-9'"&FF'G-94"/*"92*:*6':*"&(<'"F'=)G9*7"&-'92*:"=*&<G:*"'6"92*"<8.-&(O9'O-'8<*":&98'+"
!@>
"
5W;Max+" 328F2" 8<" 92*" :*<)'-<*" 9'" 92*" ):*6*::*7" 9*<9" 8=&.*+" -':=&(8]*7" 9'" 92*" <9&-7&:7"
!@?
"
7*M8&98'-"'6"&F98M89Q"H*6':*"<98=G(G<"'-<*9"Z,8.G:*"5$L[4""
!@@
"
"/*"-*S9"*S&=8-*7"92*"(&9*-F8*<"'6"92*":*<)'-<*<"Z,8.G:*"#f[4"L<"*S)*F9*7+"N!"-*G:'-<"
!@A
"
3*:*"&F98M&9*7"68:<9+"&9"&"=*&-"(&9*-FQ"'6"$#=<"8-"H'92"='-R*Q<4"U28<"3&<"6'(('3*7"HQ"&:*&"
!@B
"
N>"3892"(&9*-F8*<"'6"$B"&-7"?!=<+"&-7"92*-"HQ"VU"3892"(&9*-F8*<"'6"?B"&-7"@@=<"8-"='-R*Q<"W"
!@C
"
&-7",+":*<)*F98M*(Q4"
!AD
"
LG92':"M*:<8'-"
"
!D"
"
!A!
"
,8.G:*"#4"58.-&(O9'O-'8<*":&98'+":*(8&H8(89Q"&-7"(&9*-FQ"'6"-*G:'-&(":*<)'-<*<"8-"N!+"N>"&-7"VU
!A#
"
L["U2*"5W;"&9"*&F2":*F':78-."<89*4"d(*F9:'7*<"3892"5W;"o"!"&:*"<2'3-"8-".:&Q4",8.G:*"5$L"<2'3<"
!A$
"
5W;Max+"F'=)G9*7"6':"92*"9*<9"<98=G(G<"92&9"*(8F89*7"92*"<9:'-.*<9"&F98M89Q"&9"&":*F':78-."<89*4"
!A>
"
b["U2*"':&F(*"F'::*(&a'-"&<"=*&<G:*"6':"92*"-'8<*"F*8(8-."&-7"92*":*(8&H8(89Q"'6"92*":*<)'-<*<4"
!A?
"
0'<9" :*F':78-." <89*<" 2&7" &" <8.-868F&-9" ':&F(*" F'::*(&98'-" Z.:&Q+" -'-O<8.-868F&-9" ':&F(*"
!A@
"
F'::*(&98'-<[4""
!AA
"
f["U2*"(&9*-FQ"'6"92*"-*G:'-&(":*<)'-<*<"3&<"<2':9*<9"8-"N!+"('-.*:"8-"N>"&-7"*M*-"('-.*:"8-"
!AB
"
VU"Z.:&Q+"G-:*(8&H(*"*<98=&9*<[4"V-<*9<"'-"92*"(*69"<2'3"92*"FG=G(&98M*"78<9:8HG98'-<"'6"(&9*-FQ"
!AC
"
&F:'<<" :*F':78-." <89*<" 8-" N!" Z:*7[+" N>" ZH(G*[" &-7" VU" Z.:**-[4" ,8.G:*" 5$b" <2'3<" 92*" <&=*"
!BD
"
78<9:8HG98'-<"&<"28<9'.:&=<4"
!B!
"
"
!B#
"
LG92':"M*:<8'-"
"
!!"
/*" *S&=8-*7" 92*" -*G:'-&(" <*(*F98M89Q" 9'" <*=&-98F" F(&<<*<" Z,8.G:*" $[4" ,':" *&F2"
!B$
"
*(*F9:'7*"3*"7*9*:=8-*7"328F2"'6"92*"!+B?>"8=&.*"F&9*.':8*<"*M'R*7"92*"28.2*<9":*<)'-<*"
!B>
"
&-7" &<<8.-*7" 92*" ):*6*::*7" F&9*.':Q" 9'" &" <G):&O':78-&9*" <*=&-98F" F(&<<+" 328F2" 2&7" H**-"
!B?
"
*<98=&9*7" 8-" &" ):*M8'G<" H*2&M8':&(" <9G7Q" 3892" 92*" UTVWX5" 8=&.*<214" ,8.G:*" $L" <2'3<" 92*"
!B@
"
<98=G(G<"<*(*F98M89Q"'6"93'":*):*<*-9&98M*":*F':78-."<89*<"8-"VU+"'-*"6:'="*&F2"='-R*Q4"d&F2"
!BA
"
7'9":*):*<*-9<"92*" -':=&(8]*7":*<)'-<*"9'"&"<8-.(*"8=&.*" :*)*9898'-"6:'="92*"9*<9"<*9"Z!DD"
!BB
"
8=&.*<+"$D" :*)*9898'-<" *&F2[+" 8-" 92*" 98=*" 38-7'3" A?O!A?=<+" F*-9*:*7"&:'G-7" 92*" )*&R" '6"
!BC
"
:*<)'-<*4"/*"-':=&(8]*7"92*":*<)'-<*<"HQ"<GH9:&F98-."92*"=*&-"&-7"78M878-."HQ"92*"<9&-7&:7"
!CD
"
7*M8&98'-"'6"92*":*<)'-<*<"9'"92*"8=&.*<"8-"92*"9*<9"<*9"<2'3-"8-"92*"<&=*"<*<<8'-"Z0*92'7<["
!C!
"
&-7"<':9*7"92*="&FF':78-."9'"92*"=*&-":*<)'-<*4",':"M8<G&(8]&98'-")G:)'<*<"3*"6'FG<*7"'-"
!C#
"
92*"9*<9"8=&.*<+"H*F&G<*"G<8-."&(("8=&.*<"'6"92*"9:&8-8-."<*9".&M*":8<*"9'"&"F:'37*7")('94"b'92"
!C$
"
<89*<"3*:*" 28.2(Q"<*(*F98M*"6':"<98=G(8" F'-9&8-8-."&-8=&(<+"3892")8F9G:*<"'6"='-R*Q<"*(8F898-."
!C>
"
92*"28.2*<9":*<)'-<*<"&-7")8F9G:*<"'6"8-&-8=&9*"'H_*F9<"*(8F898-."92*"('3*<9":*<)'-<*<"Z,8.G:*"
!C?
"
$L[4"
!C@
"
,8.G:*" $b" <2'3<" 92*" :*F':78-." <89*<" 32*:*" '-*" '6" 92*" 9')O?" F(&<<*<" *(8F89*7" 92*"
!CA
"
<9:'-.*<9":*<)'-<*"8-"92*"9:&8-8-."<*9"Z9')I"='-R*Q"We"H'99'=I"='-R*Q",[4"V-"H'92"='-R*Q<+"
!CB
"
=&-Q"VU":*F':78-."<89*<"3*:*"9G-*7"9'"&-8=&(<"Z>Al"8-"='-R*Q"We"#!l"8-"='-R*Q",[+"6''7"ZAl"
!CC
"
8-"='-R*Q"We"!$l"8-"='-R*Q",["':"H'7Q")&:9<"Z@l"8-"='-R*Q"We">l"8-"='-R*Q",[22264",8.G:*"
#DD
"
$f+" <2'3<" 92*" 28.2*<9" &-7" ('3*<9" :*<)'-<*<" *(8F89*7" HQ" 92*" 9*<9" 8=&.*<+" &M*:&.*7" &F:'<<"
#D!
"
:*F':78-." <89*<" 8-" *&F2" :*.8'-4" U2*" :*<)'-<*<" 2&7" &" 387*" :&-.*+" &-7" <'=*" '6" 92*" <98=G(8"
#D#
"
*(8F89*7"8-28H89':Q":*<)'-<*<4"/*"&(<'"'H9&8-*7"<*(*F98M*":*<)'-<*<"8-"N!"&-7"N>+"HG9"89"8<"(8R*(Q"
#D$
"
92&9"928<"<*(*F98M89Q"7'*<"-'9":*6(*F9"9G-8-."9'"<*=&-98F"F(&<<*<+"HG9"9'"92*"('F&("6*&9G:*<"8-"92*"
#D>
"
;,<"92&9"&:*"=':*"6:*PG*-9"8-"8=&.*<"'6"<)*F868F"F(&<<*<4"`<*:<"'6"92*"UN5Y"F&-"*S&=8-*"92*"
#D?
"
LG92':"M*:<8'-"
"
!#"
9G-8-."8-"N!"&-7" N>" HQ" *S9:&F98-." 92*"8=&.*"6*&9G:*<" 92&9" 6*((" 8-" 92*";,<"'6" 92*" N!" &-7" N>"
#D@
"
-*G:'-<"Z<**"H*('3"6':"&-"&-&(Q<8<"'6";,<[4""
#DA
"
"
#DB
"
#DC
"
,8.G:*"$4"5*(*F98M89Q"6':"<*=&-98F"F(&<<*<"'6"UTVWX5"8=&.*<"
#!D
"
L["U2*"<*(*F98M89Q"'6"*S&=)(*"VU":*F':78-."<89*<"8-"='-R*Q"W"Z(*69["&-7"='-R*Q","Z:8.29[4"Y&9&"
#!!
"
)'8-9<":*):*<*-9"92*"-':=&(8]*7" 0`L" :*<)'-<*" 8-"&"98=*"38-7'3"6:'="A?O!A?=<"9'"<8-.(*"
#!#
"
LG92':"M*:<8'-"
"
!$"
8=&.*" ):*<*-9&98'-<" ZWp$D" :*)*9898'-<" )*:" 8=&.*[4" V-<*9<+" 92*" 92:**" 8=&.*<" *(8F898-." 92*"
#!$
"
28.2*<9"&-7"('3*<9":*<)'-<*<+"328F2"<G)):*<<*7"&F98M89Q4"Y'9"F'(':<"F'::*<)'-7"9'"92*"<G):&O
#!>
"
':78-&9*"<*=&-98F"F(&<<*<"'6"92*"8=&.*<"Z92*"8=&.*<"<2'3-"&:*"(8F*-<*O6:**"&-7"7866*:"6:'="
#!?
"
92'<*"8-"92*" 9*<9"<*9[4"U2*"F'(':<":*):*<*-9"9')" ?"<*=&-98F"F(&<<*<+"328F2"7866*:*7"H*93**-"
#!@
"
92*"93'"='-R*Q<4",':"*S&=)(*+"H'7Q")&:9<" H*('-.*7"9'"92*"9')"?" F(&<<*<"6':"='-R*Q","HG9"
#!A
"
-'9"6':"='-R*Q"W4""
#!B
"
b[" 5*(*FaM89Q" 6':" 92*" <*=&-aF" F(&<<*<" '6" UTVWX5" 8=&.*<4" U2*" F'(':<" 8((G<9:&9*" 92*" <G):&O
#!C
"
':78-&9*" <*=&-aF" F(&<<21" '6" 92*" 'H_*F9<" *(8F8a-." 92*" <9:'-.*<9" :*<)'-<*<" Z9')I" ='-R*Q" We"
##D
"
H'1'=I"='-R*Q",[4"
##!
"
f["T8.2*<9"&-7"('3*<9":*<)'-<*<"8-"*&F2":*.8'-"6':"*&F2"='-R*Q"9'"92*"9*<9"8=&.*<+"&M*:&.*7"
###
"
&F:'<<":*)*9898'-<"&-7":*F':78-."<89*<"ZG<8-."&"#?=<"<=''928-."R*:-*(e"N!I":*7e"N>I"H(G*e"VUI"
##$
"
.:**-e"7&<2*7"(8-*"8-78F&9*<"92*"'-<*9"'6"92*"M8<G&("<98=G(G<[4"
##>
"
"
##?
"
Showcase #1: AI modeling of the response and the most exciting input at individual recording
##@
"
sites "
##A
"
U'" 8((G<9:&9*" 92*" )'9*-98&(" '6" 92*" UN5Y" 6':" LV" &))(8F&98'-<+" 3*" ='7*(*7" 92*" 0`L"
##B
"
:*<)'-<*<"&9"*&F2":*F':78-."<89*"3892"92*"&)):'&F2"'6"b&<28M&-"*9"&(47"Z8-F(G78-.":*F':78-."
##C
"
<89*<" 3892" &" =*&-" :*(8&H8(89Q" 28.2*:" 92&-" D4$+" <**" 0*92'7<e" ='-R*Q" WI" WpA?Ce" ='-R*Q" ,I"
#$D
"
Wp@D@[4" ,':" 928<" &-&(Q<8<+" 3*" G<*7" 92*" -':=&(8]*7" :*<)'-<*<+" &M*:&.*7" 8-" &" 98=*" 38-7'3"
#$!
"
F*-9*:*7"'-"92*")*&R":*<)'-<*+"328F2"7866*:*7"&F:'<<"92*"&:*&<"ZN!I"#?O!#?=<e"N>I"?DO!?D=<e"
#$#
"
VUI"A?O!A?=<[4"/*"7*9*:=8-*7"92*"(8-*&:"):'_*F98'-"6:'="92*"(&Q*:"'6"&"F'-M'(G98'-&("-*G:&("
#$$
"
-*93':R" ZfWW+" V-F*)98'-" M!27+" *4.4" (&Q*:" mF'-M#7#n" ':" m=8S*7>*n[" 92&9" H*<9" ):*78F9*7" 92*"
#$>
"
:*<)'-<*"'6"-*G:'-<"&9"*&F2"'6"92*":*F':78-."<89*<4"U28<"):*78F98'-"&(<'"Q8*(7*7"&-"*<98=&9*"'6"
#$?
"
92*";,"'6"-*G:'-<"&9"*&F2"'6"92*":*F':78-."<89*<"Z,8.G:*">["Z7*9&8(<"8-"0*92'7<[4""
#$@
"
"
#$A
"
LG92':"M*:<8'-"
"
!>"
"
#$B
"
,8.G:*">4";,"F*-9*:<"'6":*F':78-."<89*<4"
#$C
"
;,"F*-9*:<"'6":*F':78-."<89*<"8-"='-R*Q"W"Z(*k["&-7"='-R*Q","Z:8.29[+"*<a=&9*7"HQ"92*"='7*("
#>D
"
92&9"):*78F9<":*<)'-<*<"9'"92*"UN5Y4";*7+"N!"<89*<e"b(G*+"N>"<89*<e"X:**-+"VU"<89*<4"
#>!
"
"
#>#
"
U'" *M&(G&9*" 92*" PG&(89Q" '6" 92*" ='7*(" *<98=&9*" '6" 9G-8-.+" 3*" =*&<G:*7" 92*" F:'<<O
#>$
"
M&(87&9*7"F'::*(&98'-" '6"92*" :*<)'-<*" ):*78F9*7" HQ" *&F2" fWW" (&Q*:" Z9:&8-*7" G<8-." '-(Q" 92*"
#>>
"
8=&.*<"6:'="92*"9:&8-8-."<*9I"<**"0*92'7<["6':"92*"!DD"9*<9"8=&.*<"8-"92*"7&9&<*9"Z,8.G:*"?L[4"
#>?
"
U2*"='7*(")*:6':=&-F*"3&<"28.2"8-"*&F2"'6"92*"&:*&<4"U2*"=*&-"F'::*(&98'-<"3*:*"D4A?"&-7"
#>@
"
D4A@"8-"N!+"D4A@"&-7"D4A>"8-"N>"&-7"D4AC"&-7"D4A#"8-"VU"8-"='-R*Q<"W"&-7",+":*<)*F98M*(Q4"U2*"
#>A
"
8-F*)98'-"(&Q*:"-G=H*:"92&9"3&<"&H(*"9'"H*<9"='7*("92*"9G-8-."8-F:*&<*7"6:'="N!"9'"N>"&-7"
#>B
"
3&<"28.2*<9"8-"VU"Z,8.G:*"?b[4"
#>C
"
/*"G<*7"92*"='7*(" 9'" *<98=&9*" 92*" ='<9"*SF898-."8-)G9<" Z0dV<[5+" 328F2" &:*"8=&.*<"
#?D
"
92&9"&:*"):*78F9*7"9'"*(8F89"&"<9:'-.":*<)'-<*"&-7"92&9"8((G<9:&9*"92*"-*G:'-<n"<*(*F98M89Q4",8.G:*"
#?!
"
?f"<2'3<"&"6*3":*):*<*-9&98M*"0dV<"6:'="-*G:'-<"&9":*F':78-."<89*<"8-"92*"7866*:*-9"&:*&<"8-"
#?#
"
92*"93'"='-R*Q<4"U2*"0dV<" 8-F(G7*7" *7.*<" &-7" F'(':6G(")&9F2*<"8-"N!+" =':*" F'=)(*S" &-7"
#?$
"
9*S9G:8]*7cFG:M8(8-*&:")&99*:-<"8-"N>"92&9"<'=*98=*<":*<*=H(*7"*Q*<"&-7"6G::Q"&-8=&("6&F*<"
#?>
"
':"H'7Q")&:9<"&9"92*"*S&=)(*"VU":*F':78-."<89*<4"/*"<9:*<<"92&9"92*<*"0dV<"&:*"H&<*7"'-"0`L"
#??
"
&-7":*):*<*-9" 8=&.*<"92&9"*(8F89"<9:'-.":*<)'-<*<"8-".:'G)<"'6"-*G:'-<" &-7"7'"-'9"&FF'G-9"
#?@
"
6':"6':=<" '6" 9G-8-." 92&9"7866*:" H*93**-" -*8.2H':8-." -*G:'-<4"V-" &:*&" N!+"6':"*S&=)(*+" 92*"
#?A
"
LG92':"M*:<8'-"
"
!?"
):*F8<*"('F&98'-"'6"'-O"&-7"'66";,":*.8'-<"'6"-*8.2H':8-."<8=)(*"F*((<"=8.29"7866*:"&-7"3'G(7"
#?B
"
-'9"H*")8FR*7"G)"HQ"92*"='7*(4""
#?C
"
"
#@D
"
#@!
"
,8.G:*"?4"0'7*(8-."92*"9G-8-."&-7"92*"='<9"*SF898-."8-)G9"'6":*F':78-."<89*<"
#@#
"
L["LM*:&.*"F:'<<OM&(87&9*7"ZfN["F'::*(&a'-"H&<*7"'-"!!"(&Q*:<"'6"92*"8-F*)a'-"-*93':R"8-"N!"
#@$
"
Z:*7[+"N>"ZH(G*["&-7"VU"Z.:**-[4"U')+"='-R*Q"W4"b'1'=+"='-R*Q",4"/289*"(8-*+"=*78&-4"b'SI"
#@>
"
#?OA?th" )*:F*-a(*<4" /28<R*:<+" :&-.*" '6" &((" M&(G*<4" Y&<2*7" 2':8]'-9&(" (8-*+" =*78&-" ':&F(*"
#@?
"
F'::*(&a'-"&F:'<<"*(*F9:'7*<+"328F2":*):*<*-9<"92*"-'8<*"F*8(8-.4"52&7*7".:&Q":*.8'-+":&-.*"
#@@
"
'6"92*"':&F(*"F'::*(&a'-"&F:'<<"*(*F9:'7*<"Z84*4"=8-"&-7"=&S[4"
#@A
"
b["V-F*)98'-"(&Q*:"3892"92*"28.2*<9"fNOF'::*(&98'-"6':"*&F2":*F':78-."<89*4"
#@B
"
f[";*):*<*-9&98M*"0dV<"6:'="H'92"='-R*Q<"8-"N!"Z9')[+"N>"Z=877(*["&-7"VU"ZH'99'=[4"
#@C
"
#AD
"
Showcase #2: Noise correlations within and between cortical areas"
#A!
"
LG92':"M*:<8'-"
"
!@"
/*"&-&(Q]*7"92*")&99*:-"'6"F'::*(&9*7"6(GF9G&98'-<"8-"&F98M89Q"&F:'<<"9:8&(<+"R-'3-"&<"
#A#
"
-'8<*" F'::*(&98'-<+" &<" &" <*F'-7" 8((G<9:&98'-" '6" 92*" G98(89Q" '6" 92*" UN5Y4" W'8<*" F'::*(&98'-<"
#A$
"
:*):*<*-9" 92*" _'8-9" 6(GF9G&98'-<" '6" &F98M89Q" &F:'<<" :*F':78-." <89*<4" U2*Q" 'FFG:" 7G:8-." 92*"
#A>
"
-*G:'-&(":*<)'-<*<"9'"<)*F868F")8F9G:*<"&-7"7G:8-."<)'-9&-*'G<"&F98M89Q28304"%:*M8'G<"<9G78*<"
#A?
"
7*='-<9:&9*7"92&9"-'8<*"F'::*(&98'-<"&:*"<9:'-.*:"H*93**-"-*G:'-<"92&9"&:*"9G-*7"9'"<8=8(&:"
#A@
"
6*&9G:*<314"V-"'G:"&-&(Q<8<+"3*"<)(89"92*"$D"9:8&(<"3892"*&F2"9*<9"<98=G(G<"8-9'"93'"2&(M*<4"/*"
#AA
"
G<*7"92*"'77O-G=H*:*7"9:8&(<"9'"7*9*:=8-*"92*"-'8<*"F'::*(&98'-<+"&-7"92*"*M*-O-G=H*:*7"
#AB
"
9:8&(<"G<*7"9'"&<<*<<"<8=8(&:89Q"'6"9G-8-."'6"92*"-*G:'-<4""
#AC
"
/*"8((G<9:&9*"92*"-'8<*"F'::*(&98'-"6':"*S&=)(*")&8:"'6"N!":*F':78-."<89*<"8-",8.G:*"@L+b4"
#BD
"
U2*"&F98M89Q"8-"&"98=*O38-7'3"6:'="#?O!#?=<"3&<"F'::*(&9*7"&F:'<<"92*"'77"-G=H*:*7"9:8&(<+"
#B!
"
3892"&"F'::*(&98'-"F'*668F8*-9"'6"D4AB"ZWp!?"9:8&(<+")"o"!D-6+"9O9*<9[4"U2*":*<)'-<*<"9'"92*"'92*:"
#B#
"
9*<9"<98=G(8"3*:*"&(<'"F'::*(&9*7"&F:'<<"9:8&(<"&-7"3*"F'=)G9*7"92*")''(*7"-'8<*"F'::*(&98'-+"
#B$
"
HQ"68:<9"<GH9:&F98-."92*"&M*:&.*":*<)'-<*")*:"9*<9"<98=G(G<"&-7"92*-"F'::*(&98-."92*"<8-.(*"9:8&("
#B>
"
6(GF9G&98'-<" &:'G-7" 92*<*" &M*:&.*<" ZWp!+?DD" '77O-G=H*:*7" 9:8&(<+" ,8.G:*"@f+" (*69[324" U2*"
#B?
"
)''(*7"-'8<*"F'::*(&98'-"6':"928<"*S&=)(*")&8:"3&<"D4$>"Z)"o"!D-6+"9O9*<9[4""
#B@
"
U'"=*&<G:*"'6"92*"<8=8(&:89Q"'6"92*"9G-8-."'6"-*G:'-<"&9"92*"93'":*F':78-."<89*<"&F:'<<"
#BA
"
9*<9"8=&.*<+" 3*" F'=)G9*7"92*" <8.-&(" F'::*(&98'-+"84*4" 92*" F'::*(&98'-"H*93**-" 92*" &M*:&.*"
#BB
"
:*<)'-<*<"*(8F89*7"HQ"92*"!DD"9*<9"8=&.*<324"L"28.2"<8.-&("F'::*(&98'-"8=)(8*<"92&9"-*G:'-<"&9"
#BC
"
92*"93'"<89*<"):*6*:"92*"<&=*"8=&.*<4"U2*"*S&=)(*"N!ON!")&8:"2&7"&"<8.-&("F'::*(&98'-"D4B$"
#CD
"
Z)"o"!D-6+"9O9*<9e",8.G:*"@f+":8.29[4",8.G:*"@Y"8((G<9:&9*<"&"3*&R*:"-'8<*"F'::*(&98'-"6':"&-'92*:"
#C!
"
)&8:" '6" N!" <89*<" Z)" o" !D-6+" 9O9*<9[" 3892" ;,<" &9" &" (&:.*:" 78<9&-F*" &-7" 3892" &" 3*&R*:" <8.-&("
#C#
"
F'::*(&98'-4"
#C$
"
U2*"<8.-&("F'::*(&98'-"8<"&"<9:&8.296':3&:7"=*&<G:*"'6"9G-8-."<8=8(&:89Q+"HG9"89"7*)*-7<"
#C>
"
'-"92*"<98=G(G<"<*9+"328F2"8=)'<*<"&" )&99*:-" '6" <)&98&(" &-7"6*&9G:*"F'::*(&98'-<"&F:'<<"92*"
#C?
"
LG92':"M*:<8'-"
"
!A"
8=&.*<4"V-"-&9G:&("8=&.*<+"-*&:HQ" :*.8'-<" &:*" .*-*:&((Q" =':*"<8=8(&:"92&-":*.8'-<" 92&9" &:*"
#C@
"
6&:92*:"&)&:9+" F&G<8-."<9:'-.*:"<8.-&("F'::*(&98'-<"'6"-*G:'-<"3892"-*&:HQ";,<33364" `<*:<"'6"
#CA
"
92*"UN5Y"F&-"*S&=8-*"'92*:"=*&<G:*<"<GF2"&<"<8=8(&:89Q"'6"9G-8-."9'"F'(':"':"':8*-9&98'-+"&<"
#CB
"
3&<"=*-98'-*7"8-":*(&98'-"9'"<2'3F&<*"J!4"U2*"<8.-&("&-7"-'8<*"F'::*(&98'-":*(&9*"92*"&F98M89Q"
#CC
"
&9" 93'" :*F':78-." <89*<" HG9" &:*" =&92*=&98F&((Q" 8-7*)*-7*-94" W*G:'-<" =&Q" ):*6*:" <8=8(&:"
$DD
"
8=&.*<"HG9"2&M*"8-7*)*-7*-9"&F98M89Q"6(GF9G&98'-<"&F:'<<" <98=G(G<":*)*9898'-<"Z*4.4"-*G:'-<"
$D!
"
8-"7866*:*-9"8-78M87G&(<[4""
$D#
"
/*"-*S9"8-M*<98.&9*7"92*")&99*:-"'6"-'8<*"F'::*(&98'-<"&F:'<<"&(("N!ON!")&8:<"8-"='-R*Q"
$D$
"
W"ZWpA#+A!![4"U2*"=*78&-"-'8<*"F'::*(&98'-"3&<"D4!!"&-7"92*"=*78&-"<8.-&("F'::*(&98'-"D4$?"
$D>
"
ZU&H(*"!e" :*7"H&:<"8-",8.4"AL[4"U2*"-'8<*"F'::*(&98'-"3&<"3*&R*:"6':":*F':78-."<89*<"3892";,<"
$D?
"
92&9"3*:*" 6&:92*:"&)&:9"Z,8.4"@d[" Z)"o"!D-6+"9O9*<9["&-7"92*"<8.-&("F'::*(&98'-"3&<" 3*&R*:"9''"
$D@
"
Z,8.4"@,["Z:pOD4?Ce")"o"!D-6+"9O9*<9[4"U2*"<9:*-.92"'6"92*"-'8<*"F'::*(&98'-"3&<"):*78F9*7"HQ"92*"
$DA
"
<8.-&(" F'::*(&98'-+" 3892" &" F'::*(&98'-" F'*668F8*-9" '6" D4#B" Z)" o" !D-6+" 9O9*<9[" Z,8.G:*" @X[+"
$DB
"
&FF'G-98-."6':"hAl"'6"92*"M&:8&-F*"Z;2+"<PG&:*"'6"92*"F'::*(&98'-"F'*668F8*-9[4"T*-F*+"92*";,O
$DC
"
78<9&-F*" &-7" <8.-&(" F'::*(&98'-" H'92" ):*78F9*7" 92*" -'8<*" F'::*(&98'-+" &-7" 92*<*" ):*78F9':<"
$!D
"
3*:*"&(<'"F'::*(&9*7"&='-." 92*=<*(M*<4" /*" G<*7"&"M&:8&-F*" )&:9898'-8-." &-&(Q<8<25,3740" 9'"
$!!
"
7*9*:=8-*"2'3" =GF2" M&:8&-F*" '6" 92*" -'8<*" F'::*(&98'-" 3&<" G-8PG*(Q" *S)(&8-*7" HQ" 92*" ;,O
$!#
"
78<9&-F*+"92*"<8.-&("F'::*(&98'-"&-7"2'3" =GF2" M&:8&-F*" F'G(7" H*" *S)(&8-*7"HQ"H'92"6&F9':<"
$!$
"
Z,8.4"Af[4"^6"92*"M&:8&-F*+"$4>l"3&<"&<<'F8&9*7"3892"H'92"):*78F9':<+"&-7"92*"<8.-&("F'::*(&98'-"
$!>
"
*S)(&8-*7">4#l"G-8PG*"M&:8&-F*4"V-"F'-9:&<9+"92*";,O78<9&-F*"&FF'G-9*7"6':"'-(Q"D4!l"G-8PG*"
$!?
"
M&:8&-F*+"8-78F&98-."='<9"'6"92*"M&:8&-F*"&<<'F8&9*7"3892"92*";,O78<9&-F*"3&<"&(<'"&FF'G-9*7"
$!@
"
6':"HQ"92*"<8.-&("F'::*(&98'-4"/*"&(<'"*S&=8-*7"92*"-'8<*"F'::*(&98'-<"7G:8-."92*"<)'-9&-*'G<"
$!A
"
&F98M89Q"Z!DD=<"H*6':*"<98=G(G<"'-<*9e",8.G:*"Af+"U&H(*"!"&-7",8.G:*"5>["&-7"6'G-7"92&9"92*Q"
$!B
"
3*:*"<9:'-.*:"92&-"7G:8-."92*"M8<G&("<98=G(G<"Z)"o"!D-6+"9O9*<9[4"U2*"<8.-&("F'::*(&98'-"*S)(&8-*7"
$!C
"
LG92':"M*:<8'-"
"
!B"
!@4!l"'6"92*"M&:8&-F*"8-"92*"<)'-9&-*'G<"-'8<*"F'::*(&98'-"Z)"o"!D-6+"9O9*<9[4"V9"8<":*=&:R&H(*"
$#D
"
92&9"<8.-&("F'::*(&98'-<+" 8-7GF*7" HQ"M8*38-."-&9G:&(" 8=&.*<+" ):*78F9*7" 92*"<)'-9&-*'G<" N!"
$#!
"
&F98M89Q"6(GF9G&98'-<"H*F&G<*"92*Q"'FFG::*7"8-"92*"&H<*-F*"'6"&"M8<G&("<98=G(G<4"/*":*)(8F&9*7"
$##
"
92*<*":*<G(9<"8-"='-R*Q",+"8-"328F2"92*"=*78&-"-'8<*"F'::*(&98'-"7G:8-.")8F9G:*"):*<*-9&98'-"
$#$
"
3&<" D4DA+" 92*" <)'-9&-*'G<" -'8<*" F'::*(&98'-" 3&<" D4!!" &-7" 92*" <8.-&(" F'::*(&98'-" D4!C"
$#>
"
ZWp>!+CD?["ZU&H(*"!+",8.G:*"Ab+f[4"
$#?
"
"
$#@
"
"
$#A
"
,8.G:*"@4"W'8<*"F'::*(&98'-<"&-7"<8.-&("F'::*(&98'-<"8-"N!4"
$#B
"
L["dS&=)(*"9*<9"8=&.*"3892"92*";,<"'6"92:**"N!":*F':78-."<89*<"ZH(&FR+":*7"&-7"H:'3-"F8:F(*<[4"""
$#C
"
b["f'::*(&a'-"H*93**-"92*":*<)'-<*"'6"93'"N!":*F':78-."<89*<"ZH(&FR"&-7":*7";,<"8-"L["&F:'<<"
$$D
"
!?":*)*aa'-<"'6"92*"9*<9"8=&.*"Z#?O!#?=<"&k*:"<a=G(G<"'-<*9[4""
$$!
"
f["K*kI" -'8<*" F'::*(&98'-"'6" 92*" <&=*")&8:" '6" N!"<89*<" &F:'<<" !+?DD"9:8&(<" 3892" 92*"!DD" 9*<9"
$$#
"
8=&.*<+" &69*:" <GH9:&F98-." 92*" &M*:&.*" :*<)'-<*" )*:" 8=&.*4" ;8.29I" 58.-&(" F'::*(&98'-" '6" 92*"
$$$
"
<&=*")&8:"'6"N!"<89*<+"H*93**-"92*"&M*:&.*":*<)'-<*<"9'"92*"!DD"9*<9"8=&.*<4"
$$>
"
Y["5&=*"&<"f"6':"&-'92*:"N!ON!")&8:"Z:*7"&-7"H:'3-";,<"8-"L[":*F':7*7"<8=G(9&-*'G<(Q4"
$$?
"
d[" f'::*(&98'-" H*93**-" ;,O78<9&-F*" &-7" -'8<*" F'::*(&98'-" &F:'<<" N!ON!" )&8:<" 8-" ='-R*Q" W"
$$@
"
ZWpA#+A!![4""
$$A
"
,["f'::*(&98'-"H*93**-";,O78<9&-F*"&-7"<8.-&("F'::*(&98'-4"
$$B
"
LG92':"M*:<8'-"
"
!C"
X["f'::*(&98'-"H*93**-"92*"<8.-&("&-7"-'8<*"F'::*(&98'-"&F:'<<"N!ON!")&8:<4"
$$C
"
"
$>D
"
/*"&(<'"*S&=8-*7"92*"F'::*(&98'-<"H*93**-"-*G:'-<"38928-"N>"&-7"38928-"VU"Z,8.4"ALe"
$>!
"
U&H(*"![4" U2*<*"F'::*(&98'-")&99*:-<"H':*"<8=8(&:898*<"9'"92'<*"38928-"N!4"U2*"=*78&-"N>ON>"
$>#
"
-'8<*" F'::*(&98'-<" 7G:8-." <98=G(G<" ):*<*-9&98'-" Z98=*O38-7'3" F*-9*:*7" '-" 92*" )*&R"
$>$
"
:*<)'-<*+" ?DO!?D=<[" 3*:*" D4!D" ZW" p" !C+?D$[" &-7" D4DA" ZW" p" @+AB@[+" 6':" ='-R*Q<" W" &-7" ,+"
$>>
"
:*<)*F98M*(Q+"&-7"92*"=*78&-"VUOVU"-'8<*"F'::*(&98'-<"Z98=*O38-7'3"A?O!A?=<["3*:*"D4DC"ZW"
$>?
"
p"C+BAD["&-7"D4DB"ZW"p!#+#>@[4"U2*"<8.-&("F'::*(&98'-<"&FF'G-9*7"6':"#?4>l"&-7"#D4Dl"'6"92*"
$>@
"
M&:8&-F*"8-"92*"-'8<*"F'::*(&98'-<"'6"N>ON>")&8:<"8-"='-R*Q<"W"&-7",+":*<)*F98M*(Q+"&-7">#4Al"
$>A
"
&-7"!D4?l"'6"92*"-'8<*"F'::*(&98'-<"H*93**-"VUOVU")&8:<4"U2*"<)'-9&-*'G<"-'8<*"F'::*(&98'-<"
$>B
"
38928-"N>"&-7"38928-"VU"3*:*"'6"F'=)&:&H(*"<9:*-.92"&<"7G:8-."92*"M8<G&("<98=G(G<"ZU&H(*"![+"
$>C
"
&-7" ='<9" '6" 92*" M&:8&-F*" 3&<" *S)(&8-*7" HQ" 92*" <8.-&(" F'::*(&98'-<" Z,8.G:*" Af[4" T*-F*+" 92*"
$?D
"
<8.-&("F'::*(&98'-<"8-7GF*7"HQ"-&9G:&("8=&.*<"&:*"=8::':*7"HQ"&F98M89Q"6(GF9G&98'-<"38928-"N>"
$?!
"
&-7"VU"8-"92*"&H<*-F*"'6"&"<98=G(G<+"_G<9"&<"3&<"92*"F&<*"6':"&:*&"N!4""
$?#
"
/*"-*S9"8-M*<a.&9*7"92*"-'8<*"F'::*(&a'-<"H*93**-"-*G:'-<"8-"78i*:*-9"&:*&<4";*F*-9"
$?$
"
<9G78*<":*)':9*7"92&9"-'8<*"F'::*(&a'-<"H*93**-"N!"&-7"N>"&:*".*-*:&((Q")'<8aM*41,42+"HG9"92*"
$?>
"
-'8<*" F'::*(&a'-<" H*93**-" VU" &-7" ('3*:" M8<G&(" &:*&<" 2&M*+" 9'" 'G:" R-'3(*7.*+" -'9" H**-"
$??
"
7*<F:8H*74",8.G:*"AYOd"8((G<9:&9*<"92*"-'8<*"F'::*(&98'-<"H*93**-"*S&=)(*":*F':78-."<89*<"8-"
$?@
"
N!+"N>"&-7"VU"8-"='-R*Q"W4"U2*"-'8<*"F'::*(&98'-"3&<"D4$$"6':"92*"N!ON>")&8:"Z)"o"!D-6+"9O9*<9[+"
$?A
"
D4#C"6':"92*" N!OVU" )&8:"Z)"o"!D-6+"9O9*<9["&-7"D4>C"6':" 92*" N>OVU")&8:"Z)"o"!D-6+"9O9*<9["&-7" 92*"
$?B
"
:*<)*F98M*"<8.-&("F'::*(&98'-<"3*:*"D4>#+"D4DB"&-7"D4D@4"U28<"*S&=)(*"8<":*=&:R&H(*+"H*F&G<*"
$?C
"
89"8((G<9:&9*<"92*"F'G)(8-."'6"&F98M89Q"6(GF9G&98'-<"'6"-*G:'-<"8-"N!"&-7"VU+"6&:"&)&:9"8-"92*"M8<G&("
$@D
"
F':98F&("28*:&:F2Q4"
$@!
"
"
$@#
"
LG92':"M*:<8'-"
"
#D"
Table 1. Medians of the noise and signal correlations.
$@$
"
Monkey N
V1/V1
IT/IT
V1/IT
V4/IT
Spontanous noise correlation
0.19
0.09
0.17
0.14
Peak noise correlation
0.11
0.09
0.11
0.09
Signal correlation
0.35
0.14
0.27
0.12
Monkey F
Spontanous noise correlation
0.11
0.13
0.11
0.12
Peak noise correlation
0.07
0.08
0.07
0.07
Signal correlation
0.19
0.21
0.16
0.20
"
$@>
"
"
$@?
"
U2*" =*78&-" -'8<*" F'::*(&98'-<" &F:'<<" &((" N!ON>" )&8:<" 3*:*" D4!!" 8-" ='-R*Q" W" ZW" p"
$@@
"
A?+@$@["&-7"D4DA"8-"='-R*Q","ZWp$$+C$D["7G:8-."92*"):*<*-9&98'-"'6"92*"<98=G(G<+"&-7"D4!@"
$@A
"
&-7" D4!!" 7G:8-." <)'-9&-*'G<" &F98M89Q" ZU&H(*" ![4" U2*" -'8<*" F'::*(&98'-<" H*93**-" N!" &-7" VU"
$@B
"
3*:*"&(<'")'<898M*"7G:8-."92*"):*<*-9&98'-"'6"92*"M8<G&("<98=G(G<+"3892"&"=*78&-"'6"D4!!"&-7"
$@C
"
D4DA"8-"='-R*Q<"W"ZW"p"?$+B@#["&-7","ZW"p">?+?$D[+":*<)*F98M*(Q+"&-7"92*"=*78&-"<)'-9&-*'G<"
$AD
"
-'8<*" F'::*(&98'-<" 3*:*" D4!A" &-7" D4!!" Z,8.G:*" AL+b+" ,8.G:*" 5>[4" ,8-&((Q+" 92*" =*78&-" -'8<*"
$A!
"
F'::*(&98'-<"H*93**-"N>"&-7"VU"3*:*"D4DC"&-7"D4DA"7G:8-."M8<G&("<98=G(&98'-" &-7" D4!>" &-7"
$A#
"
D4!#"7G:8-."<)'-9&-*'G<"&F98M89Q"8-"='-R*Q<"W"ZWp"#A+C!B["&-7","ZW"p"!B+$@C[+":*<)*F98M*(Q4""
$A$
"
/*"3*:*"<G:):8<*7"92&9"92*"-'8<*"F'::*(&98'-<"'6"N!OVU")&8:<+"8-"&:*&<"92&9"&:*"78<9&-9"
$A>
"
&-7"6G-F98'-&((Q"78<<8=8(&:+"2&7"&" <9:*-.92" 92&9" 3&<" F'=)&:&H(*"9'"92&9"'6"N!ON>"&-7"N>OVU"
$A?
"
)&8:<4" ^-*" )'<<8H8(89Q" 8<" 92&9" 92*<*" ('-.O78<9&-F*" -'8<*" F'::*(&98'-<" &:*" F&G<*7" HQ" .('H&("
$A@
"
6(GF9G&98'-<"8-"H:&8-"<9&9*+"*4.4"F&G<*7"HQ"M&:8&98'-<"8-"92*"&-8=&(<n"&(*:9-*<<"':"='98M&98'-4"
$AA
"
L-" &(9*:-&98M*" <F*-&:8'" 8<" 92&9" 92*Q" :*6(*F9" =':*" ('F&(" M&:8&98'-<" 8-" 92*" &F98M89Q" '6" <)*F868F"
$AB
"
-*G:'-&("F8:FG89<+"328F2"F'G(7"H*":*(&9*7+"6':"*S&=)(*+"9'"M&:8&98'-<"8-"32*:*" 8-"92*"78<)(&Q"
$AC
"
92*" &-8=&(" 78:*F9*7" &99*-98'-4" X('H&(" H:&8-" 6(GF9G&98'-<" <2'G(7" 'FFG:" 6':" &((" -*G:'-<"
$BD
"
LG92':"M*:<8'-"
"
#!"
8::*<)*F98M*"'6"92*8:"9G-8-.+"32*:*&<"M&:8&98'-<"8-"92*"&F98M89Q"8-"<)*F868F"F8:FG89<"&:*"*S)*F9*7"
$B!
"
9'"H*"<2&:*7"HQ"-*G:'-<"3892"<8=8(&:"9G-8-.+"328F2"&:*"78:*F9(Q"':"8-78:*F9(Q"F'--*F9*74""
$B#
"
/*" 92*:*6':*" *S&=8-*7" 2'3" 92*" 8-9*:&:*&(" -'8<*" F'::*(&98'-<" 7*)*-7*7" '-" 9G-8-."
$B$
"
<8=8(&:89Q+"PG&-9868*7"&<"92*"<8.-&("F'::*(&98'-4"%&8:<"3892"&"<9:'-."<8.-&("F'::*(&98'-"9*-7*7"9'"
$B>
"
2&M*"&"<9:'-.*:"-'8<*"F'::*(&98'-"7G:8-."92*"M8<G&("<98=G(G<+"3892"F'::*(&98'-"F'*668F8*-9<"'6"
$B?
"
D4#@"6':"N!ON>")&8:<+"D4#@"6':"N!OVU")&8:<"&-7"D4?D"6':"N>OVU")&8:<"Z)"o"!D-6+"9O9*<9["Z,8.G:*"A,[4"
$B@
"
V-9*:*<98-.(Q+"92*"<8.-&("F'::*(&98'-"&(<'"):*78F9*7"92*"-'8<*"F'::*(&98'-"7G:8-."<)'-9&-*'G<"
$BA
"
&F98M89Q"Z,8.G:*"5@b[+"3892"&"F'::*(&98'-"F'*668F8*-9"'6"D4>$"6':"N!ON>")&8:<+"D4$C"6':"N!OVU")&8:<"
$BB
"
&-7"D4?$"6':"N>OVU")&8:<4"U2*<*":*<G(9<"3*:*"F'-68:=*7"HQ"92*"M&:8&-F*")&:9898'-8-."&-&(Q<8<+"
$BC
"
<2'38-."92&9"&"<GH<9&-98&("6:&F98'-"'6"92*"M&:8&-F*"8-"92*<*"8-9*:&:*&("-'8<*"F'::*(&98'-<"3&<"
$CD
"
F&)9G:*7"HQ"92*"<8.-&("F'::*(&98'-"Z,8.G:*"AX[4"U2*"68-78-."92&9"92*"-'8<*"F'::*(&98'-<"H*93**-"
$C!
"
-*G:'-<"8-"N!+"N>"&-7"VU"7G:8-."<)'-9&-*'G<"&F98M89Q"&:*"):*78F9*7"HQ"92*8:"9G-8-."<8=8(&:89Q"
$C#
"
<G..*<9" 92&9" 92*Q" :*6(*F9" 92*" <*(*F98M89Q" '6" F':98F'OF':98F&(" F8:FG89<+" 328F2" 8<" &" 68-78-." 3892"
$C$
"
8=)':9&-9"92*':*98F&("8=)(8F&98'-<+"&<"38(("H*"78<FG<<*7"H*('34""
$C>
"
^G:"8-898&("*S)(':&98'-"'6"-'8<*"F'::*(&98'-<"38928-"&-7"H*93**-"N!+"N>"&-7"VU"6G:92*:"
$C?
"
8((G<9:&9*<"92*"G98(89Q"'6"92*"NU5Y"&-7"=&Q"8-<)8:*"89<"G<*:<"9'"<9G7Q"6G:92*:"&<)*F9<"'6"92*<*"
$C@
"
F'::*(&98'-<+"8-F(G78-."92*8:"7*)*-7*-F*"'-"92*"):*M8'G<"<98=G(8"&-7"6(GF9G&98'-<"8-"92*")'3*:"
$CA
"
'6"92*"K,%"6:*PG*-FQ"H&-7<4""
$CB
"
"
$CC
"
LG92':"M*:<8'-"
"
##"
"
>DD
"
,8.G:*"A4"W'8<*"F'::*(&98'-<"&-7"92*8:"7*)*-7*-F*"'-"92*"<8.-&("F'::*(&98'-4"
>D!
"
L+b["W'8<*" F'::*(&98'-<"7G:8-."92*"M8<G&("<98=G(G<" Z(*69["&-7"<8.-&("F'::*(&98'-<"Z:8.29[" 8-"92*"
>D#
"
93'"='-R*Q<4",8.G:*"5>"<2'3<"92*"-'8<*"F'::*(&98'-<"7G:8-."<)'-9&-*'G<"&F98M89Q4",8.G:*"5?"
>D$
"
8((G<9:&9*<"92*"F'=)(*9*")&99*:-<"'6"-'8<*"F'::*(&98'-<+";,O78<9&-F*<"&-7"<8.-&("F'::*(&98'-<4"
>D>
"
f["N&:8&-F*")&:9898'-8-."38928-"F':98F&("&:*&<4"U2*"78<9&-F*"H*93**-";,<"Z)G:)(*"H&:<["*S)(&8-<"
>D?
"
(899(*"G-8PG*" M&:8&-F*" 8-" H'92"<)'-9&-*'G<" Z(*69[" &-7" <98=G(G<"7:8M*-" &F98M89Q" Z:8.29[4"X:**-"
>D@
"
H&:<I"92*"G-8PG*"M&:8&-F*"*S)(&8-*7"HQ"<8.-&("F'::*(&98'-"8<"28.2"6':"H'92"92*"<)'-9&-*'G<"&-7"
>DA
"
<98=G(G<" 7:8M*-" -'8<*" F'::*(&98'-4" q*(('3" H&:<I" 92*" F'=='-" M&:8&-F*" H*93**-" <8.-&("
>DB
"
F'::*(&98'-"&-7";,O78<9&-F*"):*78F9<"&"<=&((*:"&='G-9"'6"M&:8&-F*"8-"H'92"='-R*Q<4"U2*"*::':"
>DC
"
H&:<+"F'=)G9*7"G<8-."H''9<9:&))8-.+"3*:*"9''"<=&(("9'"H*"7*)8F9*74""
>!D
"
LG92':"M*:<8'-"
"
#$"
Y["dS&=)(*"9*<9"8=&.*"3892"92*";,<"'6"92:**":*F':78-."<89*<"8-"N!"Z:*7[+"N>"ZH(G*["&-7"VU"Z.:**-["
>!!
"
8-"='-R*Q"W4"L::'3<")'8-9"9'"92*"F'::*<)'-78-."*(*F9:'7*"&::&Q<"ZH'1'=[4"
>!#
"
d["W'8<*"F'::*(&98'-"'6"92*"N!ON>")&8:"Z(*69[+"N!OVU")&8:"Z=877(*[" &-7"N>OVU")&8:"Z:8.29["&F:'<<"
>!$
"
!+?DD"9:8&(<"3892"92*"!DD"9*<9"8=&.*<+"&69*:"<GH9:&F98-."92*"&M*:&.*":*<)'-<*")*:"8=&.*4"
>!>
"
,[" f'::*(&98'-" H*93**-" 92*" <8.-&(" F'::*(&98'-" &-7" -'8<*" F'::*(&98'-" &F:'<<" &((" N!ON>" )&8:<"
>!?
"
Z(*69[+" N!OVU" )&8:<" Z=877(*[" &-7" N>OVU" )&8:<" 8-" ='-R*Q" W4" Y&9&" 6:'=" ='-R*Q" ," &-7" 6':"
>!@
"
<)'-9&-*'G<"&F98M89Q"8<"<2'3-"8-",8.G:*"5@+"&('-."3892"&-"&-&(Q<8<"'6"92*"F'-<8<9*-FQ"&F:'<<"
>!A
"
98=*O38-7'3<4"
>!B
"
X["N&:8&-F*" )&:9898'-8-."H*93**-"F':98F&("&:*&<4"U2*":*<G(9<"3*:*"<8=8(&:"9'"92'<*"'H<*:M*7"
>!C
"
38928-"F':98F&("&:*&<4"
>#D
"
>#!
"
Discussion"
>##
"
/*"):*<*-9*7"92*"UN5Y+"&"28.2OF2&--*("F'G-9"7&9&<*9"'6"<)8R8-."&F98M89Q"8-":*<)'-<*"
>#$
"
9'"9*-<"'6"92'G<&-7<"'6"-&9G:&("8=&.*<"92&9"*-&H(*<"92*"9*<98-."'6"2Q)'92*<*<"&-7"='7*(<"'-"
>#>
"
92*"6G-F98'-<"'6"92*"M*-9:&("<9:*&=4"U2*"UN5Y"F'=)(*=*-9<"*S8<98-."-*G:'8=&.8-."7&9&<*9<"
>#?
"
H&<*7"'-"-&9G:&("8=&.*"):'F*<<8-."3892"<)8R8-."7&9&"3892"<GHO=8((8<*F'-7"):*F8<8'-+"H*F&G<*"
>#@
"
92*"-*G:')2Q<8'('.8F&("<8.-&(<"3*:*"<&=)(*7"&9"$DRT]4"V9"&(<'"F'=)(*=*-9<"&":*F*-9"F&(F8G="
>#A
"
8=&.8-." <9G7Q" '6" 92*" 6G-F98'-&(" 9')'.:&)2Q" '6" &:*&" N>" 8-" 92*" ='-R*Q43" &-7" &" :*F*-9"
>#B
"
*(*F9:')2Q<8'('.F&(" <9G7Q" 8-" 92*" ='-R*Q" M*-9:&(" <9:*&=" 7G:8-." 6:**"M8*38-." '6" -&9G:&("
>#C
"
8=&.*<444" /*" 8=):'M*7" 92*" 7*<8.-" '6" &" 28.2OF2&--*(OF'G-9" F':98F&(" 8=)(&-913,14+" 328F2"
>$D
"
*-&H(*<" 28.2" 7*-<89Q" :*F':78-.<" &F:'<<" <*M*:&(" &:*&<" '6" 92*" M*-9:&(" <9:*&=4" U2*"
>$!
"
=8F:'*(*F9:'7*<" 3*:*" 8=)(&-9*7" F2:'-8F&((Q+" &(('38-." G<" 9'" ):*<*-9" &" (&:.*" <*9" '6" 8=&.*<"
>$#
"
&F:'<<"<*M*:&(":*F':78-."7&Q<4"
>$$
"
L-"8=)':9&-9"&7M&-9&.*"'6"92*"UTVWX5"8=&.*"7&9&H&<*"8<"92&9"89"&(<'"2&<"H**-"G<*7"8-"
>$>
"
=&-Q" ):*M8'G<" -*G:'8=&.8-." &-7" H*2&M8':&(" <9G78*<9–12,21,45484" V9" 92*:*6':*" *-&H(*<" &"
>$?
"
F'=)&:8<'-" '6" 92*" -*G:'-&(" :*):*<*-9&98'-<" &F:'<<" 7866*:*-9" :*F':78-." 9*F2-8PG*<" &-7"
>$@
"
<)*F8*<4"/*"H*(8*M*"92&9"92*"UN5Y"38(("&(<'"H*"G<*6G("6':"92*"F:*&98'-"'6"-*3"LV"='7*(<"'6"92*"
>$A
"
M*-9:&(" <9:*&=4" /*" 8((G<9:&9*7" 89<" )'9*-98&(" HQ" ='7*(8-." -*G:'-&(" <*(*F98M89Q" 3892" 92*"
>$B
"
V-F*)98'-" ='7*(+" &FF'G-98-." 6':" &" 28.2" )*:F*-9&.*" '6" 92*" *S)(&8-&H(*" M&:8&-F*4" `<8-." 928<"
>$C
"
LG92':"M*:<8'-"
"
#>"
&)):'&F2+" 3*" 'H9&8-*7" =*&-8-.6G(" &-7" 8-<8.296G(" 0dV<" 6':" 92*" 92:**" M8<G&(" :*.8'-<4"
>>D
"
V-9*:*<98-.(Q+"92*"0dV<"6':"=&-Q"VU"-*G:'-<":*):*<*-9*7"&-8=&("H'7Q")&:9<"&-7"6&F*<+"&<"3&<"
>>!
"
*S)*F9*7"6:'="92*"<*(*F98M89Q"&-&(Q<8<"Z,8.G:*"$[4"T'3*M*:+"92*"0dV<".'"H*Q'-7"<*(*F98M89Q"6':"
>>#
"
'H_*F9"F(&<<"HQ":*M*&(8-."92*"8=&.*"F'-68.G:&98'-"92&9"8<"):*78F9*7"9'"7:8M*"&"<9:'-.":*<)'-<*4"
>>$
"
U2*" F'=)G9&98'-" '6" 0dV<" :*PG8:*<" LV" ='7*(<+" 328F2" F&-" -'3" H*" 78:*F9(Q" 9:&8-*7" '-" 92*"
>>>
"
-*G:'-&(":*<)'-<*<"9'"92*"UN5Yn<"(&:.*"-G=H*:<"'6"8=&.*<8,12,46,474""
>>?
"
/*"*S&=8-*7"92*")&99*:-"'6"-'8<*"F'::*(&98'-<"38928-"&-7"H*93**-"&:*&<"N!+"N>"&-7"
>>@
"
VU4"/*"6'G-7"92&9"92*"&M*:&.*"=&.-89G7*"'6"-'8<*"F'::*(&98'-<"H*93**-"-*G:'-<"8-"92*"<&=*"
>>A
"
&:*&"8<"F'=)&:&H(*"9'"92*"&M*:&.*"=&.-89G7*"'6"-'8<*"F'::*(&98'-"H*93**-"-*G:'-<"8-"&:*&<"
>>B
"
92&9"&:*" &<"78<9&-9"&<"N!"&-7" VU4"U2*<*"&:*&<"&:*" &-&9'=8F&((Q"'-(Q"F'--*F9*7"8-78:*F9(Q49,50"
>>C
"
&-7"&:*"<)*F8&(8]*7"6':"92*"):'F*<<8-."'6"7866*:*-9"8=&.*"):')*:98*<+"(8R*"<2':9+"':8*-9*7"*7.*<"
>?D
"
8-"N!"&-7"'H_*F9"&-7"6&F*"87*-989Q"8-"VU4""T*-F*+"92*"N!OVU"-'8<*"F'::*(&98'-<"):*<G=&H(Q":*6(*F9"
>?!
"
8-78:*F9"F'G)(8-.+"92'G.2"8-9*:=*78&9*"F':98F&("&:*&<"8-F(G78-."N>"&-7"<GHF':98F&("<9:GF9G:*<"
>?#
"
<GF2"&<"92*")G(M8-&:4"
>?$
"
W*G:'-<"9G-*7"9'"<8=8(&:")8F9G:*<"2&7"<9:'-.*:"-'8<*"F'::*(&98'-<"92&-"-*G:'-<"9G-*7"
>?>
"
9'"7866*:*-9")8F9G:*<4" U28<":*<G(9"F'-9:&<9<"3892"&"):*M8'G<" <9G7Q"8-"N!"'6"&-*<92*98]*7"=8F*"
>??
"
:*)':98-." 92&9" 92*" <8.-&(" F'::*(&98'-" 8<" &" )'':" ):*78F9':" '6" 92*" -'8<*" F'::*(&98'-354" U28<"
>?@
"
&))&:*-9"78<F:*)&-FQ"8<"):*<G=&H(Q":*(&9*7"9'"'G:"8-M*<98.&98'-"'6"-'8<*"F'::*(&98'-<"&F:'<<"
>?A
"
(&:.*:" F':98F&(" 78<9&-F*<+" &-7" 8-" &3&R*" ='-R*Q<" 8-<9*&7" '6" &-*<92*98]*7" =8F*4" U2*"
>?B
"
7*)*-7*-FQ" '6" 92*" -'8<*" F'::*(&98'-" '-" 9G-8-." 8-78F&9*<" 92&9" 89"8<" -'9" F&G<*7" HQ" .('H&("
>?C
"
6(GF9G&98'-<" 8-" F':98F&(" *SF89&H8(89Q+" HG9" :&92*:" :*6(*F9<" 92*" 8-9*:&F98'-<" H*93**-" F8:FG89<" '6"
>@D
"
-*G:'-<"92&9"&:*"9G-*7"9'"92*"<&=*"8=&.*<4";*=&:R&H(Q+"9G-8-."<8=8(&:89Q"&(<'"):*78F9*7"92*"
>@!
"
<9:*-.92"'6"92*"-'8<*"F'::*(&98'-"7G:8-."<)'-9&-*'G<"&F98M89Q+"328(*"92*"='-R*Q"3&<"M8*38-."
>@#
"
&".:*Q" <F:**-" &-7"3&898-." 6':" 92*"68:<9" 8=&.*" 9'"&))*&:4" U'" 92*"*S9*-9" 92&9" 'G:"<*9" '6" 9*<9"
>@$
"
LG92':"M*:<8'-"
"
#?"
8=&.*<"&)):'S8=&9*7"92*"8=&.*"<9&98<98F<"92&9"='-R*Q<"*S)*:8*-F*"7G:8-."92*8:"(86*+"92*"<8.-&("
>@>
"
F'::*(&98'-"&)):'S8=&9*7"92*")&99*:-<"'6"-*G:'-<"92&9"&:*"F'O&F98M&9*7"7G:8-."-&9G:&("M8<8'-4"
>@?
"
^G:":*<G(9<"<2'3"92&9"92*<*"&F98M89Q")&99*:-<"38928-"&-7"&F:'<<"N!+"N>"&-7"VU":**=*:.*"7G:8-."
>@@
"
<)'-9&-*'G<"&F98M89Q4"U28<":*<G(9"<G))':9<"):*M8'G<"<9G78*<"7*='-<9:&98-."92&9"<)'-9&-*'G<"
>@A
"
&F98M89Q" )&99*:-<" 8-" 92*" M8<G&(" F':9*S" :*)(&Q" &<)*F9<" '6" -&9G:&(" M8<8'-36,51,52+" &<" 2&<" H**-"
>@B
"
):*78F9*7" HQ" ='7*(<" '6" <Q-&)aF" )(&<aF89Q" 8-" 28*:&:F28F&(" -*G:&(" -*93':R<534" U2*"-'8<*"
>@C
"
F'::*(&98'-" &-&(Q<8<" 92*:*HQ" 8((G<9:&9*<" 2'3" 92*" UN5Y" F&-" .8M*" 8-<8.29" 8-9'" 8-9*:&F98'-<"
>AD
"
H*93**-" -*G:'-<" 8-" N!+" N>" &-7" VU4" V9" =&Q" 8-<)8:*" 89<" G<*:<" 9'" 6G:92*:" *S)(':*" 2'3" -'8<*"
>A!
"
F'::*(&98'-<"&-7")&99*:-<"'6"-*G:'-&("<Q-F2:'-8]&98'-"&9"&"28.2*:"9*=)':&("):*F8<8'-"7*)*-7"
>A#
"
'-"92*"9G-8-."'6"92*"-*G:'-<+"92*"6*&9G:*<"8-"92*8:";,<"&-7"92*"F'-9*S9"):'M87*7"HQ"92*":*<9"
>A$
"
'6"92*"8=&.*4""
>A>
"
0':*".*-*:&((Q+"3*"*S)*F9"92&9"92*"):*<*-9"7&9&<*9"F'G(7"F'-9:8HG9*"9'"-*3"M*-G*<"
>A?
"
'6":*<*&:F2"'-"92*"8-9*:&F98'-"'6"M8<G&("F':98F&(":*.8'-<"&-7"2'3"92*Q"7*9*:=8-*"92*"-*G:'-&("
>A@
"
9:&-<6':=&98'-<"92&9"&:*":*PG8:*7"6':"92*"9:&-<(&98'-"'6"8=&.*<"8-9'"F&9*.':8F&("8-6':=&98'-"
>AA
"
&H'G9"M8<G&("'H_*F9<4"/*"(''R"6':3&:7"9'"6G9G:*":*<*&:F2"H&<*7"'-"92*"UN5Y"92&9"=&Q"):'M87*"
>AB
"
-*3"8-<8.29<"8-9'"92*" ):'F*<<8-." '6" 92*" M*-9:&(" <9:*&="&-7"92*"-*G:'-&("<Q-9&S" G-7*:(Q8-."
>AC
"
'H_*F9":*F'.-898'-4"
>BD
"
" "
>B!
"
LG92':"M*:<8'-"
"
#@"
Acknowledgments
>B#
"
" /*" 92&-R" j':" b:&-7<=&+" L--*R*" Y89*38.+" U&8_<2&" M&-" ;**<" &-7" K*S" b**R=&-" 6':"
>B$
"
H8'9*F2-8F&(" <G))':9e" 0&G:8F*" T**=<R*:R" &-7" 08R*" N8-R" 6':" 9*F2-8F&(" &<<8<9&-F*e" r&=*<"
>B>
"
Y8f&:('+"j'2898_"j&:"&-7"0&:R"bGFR(*Q"6':"8-<8.296G("78<FG<<8'-<"'-"92*"<G:.8F&("&)):'&F24""
>B?
"
U2*" 3':R" 3&<" <G))':9*7" HQ" W/^" .:&-9<" Zf:'<<'M*:" .:&-9" !A@!C" sVWUdW5dt" &-7"
>B@
"
sYbV#t+"&"X:&M89&98'-"):'.:&="'6"92*"YG9F2"08-8<9:Q"'6"5F8*-F*[+"92*"dG:')*&-"`-8'-"T':8]'-"
>BA
"
#D#D",:&=*3':R" %:'.:&=" G-7*:" <)*F868F" .:&-9"&.:**=*-9" C>??$C" sTG=&-" b:&8-" %:'_*F9t+"
>BB
"
.:&-9"&.:**=*-9"BCC#BA"sW*G:&N8)*:+t"&-"d;f".:&-9"Z!D!D?#C@$"sW`0d;^`5t["9'"%;;"&-7"
>BC
"
HQ"W/^".:&-9<"Z^fdW/4u5##4#4DCA"&-7"NV4N*-84###4#!A["9'"%%4"
>CD
"
>C!
"
Author contributions
>C#
"
%%" F'-F*8M*7" '6" 92*" <9G7Q4" %;;+" %%" &-7" 0/5" )*:6':=*7" 92*" <G:.*:Q4" %%" &-7" ,/"
>C$
"
)*:6':=*7"92*"*S)*:8=*-9<4"%%"&-&(Q]*7"92*"7&9&4"%%"&-7"%;;"'H9&8-*7"6G-78-.4"%%"&-7"%;;"
>C>
"
3:'9*"92*")&)*:"3892"F'-9:8HG98'-<"6:'="&(("&G92':<4"
>C?
"
>C@
"
Declaration of interests
>CA
"
%;;"8<"6'G-7*:"&-7" <2&:*2'(7*:" '6" %2'<)2'*-8S+" &" F'=)&-Q"92&9"&8=<"9'" 7*M*(')" &"
>CB
"
M8<G&("H:&8-"):'<92*<8<"6':"H(8-7")*')(*4"
>CC
"
?DD
"
Methods
?D!
"
"
?D#
"
;*<'G:F*"&M&8(&H8(89Q"
?D$
"
Lead contact
?D>
"
LG92':"M*:<8'-"
"
#A"
,G:92*:"8-6':=&a'-"&-7":*PG*<9<"6':":*<'G:F*<"&-7":*&.*-9<"<2'G(7"H*"78:*F9*7"9'"&-7"38(("
?D?
"
H*"6G(g((*7"HQ"92*"K*&7"f'-9&F9<+"%&'('"%&)&(*"Z)4)&)&(*v-8-4R-&34-(["':"%8*9*:";'*(6<*=&"
?D@
"
Z)4:'*(6<*=&v-8-4R-&34-([4"
?DA
"
"
?DB
"
Materials availability
?DC
"
U28<"<9G7Q"787"-'9".*-*:&9*"-*3"G-8PG*":*&.*-9<4"
?!D
"
"
?!!
"
Data and code availability
?!#
"
;&3"&-7"0`L"7&9&"2&M*"H**-"7*)'<89*7"&9"XOW'7*"XVW"&-7"&:*")GH(8F(Q"&M&8(&H(*"&<"'6"92*"
?!$
"
7&9*"'6")GH(8F&a'-"&9"299)<Icc7'84':.c!D4!#A?!c.O-'7*42FA](M4"
?!>
"
L((" ':8.8-&(" F'7*" 2&<" H**-" 7*)'<89*7" &9" XOW'7*" XVW" &9"299)<Icc7'84':.c!D4!#A?!c.O
?!?
"
-'7*42FA](M"&-7"8<")GH(8F(Q"&M&8(&H(*"&<"'6"92*"7&9*"'6")GH(8F&a'-4"
?!@
"
L-Q"&778a'-&("8-6':=&a'-":*PG8:*7"9'":*&-&(Q]*"92*"7&9&":*)':9*7"8-"928<")&)*:"8<"&M&8(&H(*"
?!A
"
6:'="92*"(*&7"F'-9&F9"G)'-":*PG*<94"
?!B
"
"
?!C
"
"
?#D
"
dS)*:8=*-9&("='7*("&-7"<GH_*F9"7*9&8(<"
?#!
"
L(("):'F*7G:*<"F'=)(8*7"3892"92*"WVT"XG87*"6':"f&:*"&-7"`<*"'6"K&H':&9':Q"L-8=&(<"&-7"3*:*"
?##
"
&)):'M*7" HQ" 92*" sf*-9:&(*" f'==8<<8*" Y8*:):'*M*-t" '6" 92*" W*92*:(&-7<4" U3'" =&F&PG*"
?#$
"
='-R*Q<" Z=&(*<+" A" &-7" @" Q*&:<" '(7[" )&:98F8)&9*7" 8-" 92*" *S)*:8=*-9<4" U2*Q" 3*:*" <'F8&((Q"
?#>
"
2'G<*7" 8-" <9&H(*" )&8:<" 8-" &" <)*F8&(8]*7" ):8=&9*" 6&F8(89Q" 3892" -&9G:&(" 7&Q(8.29+" F'-9:'((*7"
?#?
"
2G=8789Q"&-7"9*=)*:&9G:*4"U2*"2'=*OF&.*"3&<"&"(&:.*"6('':O9'OF*8(8-."F&.*"328F2"&(('3*7"
?#@
"
-&9G:&("F(8=H8-."&-7"<38-.8-."H*2&M8':4"U2*"F&.*"2&7"&"<'(87"6('':+"F'M*:*7"3892"<&37G<9"&-7"
?#A
"
3&<" *-:8F2*7" 3892" 9'Q<" &-7" 6':&.8-." 89*=<4" U2*8:" 78*9" F'-<8<9*7" '6" ='-R*Q" F2'3+"
?#B
"
LG92':"M*:<8'-"
"
#B"
<G))(*=*-9*7"3892" 6:*<2"6:G894"U2*8:"&FF*<<"9'"6(G87"3&<"F'-9:'((*7+"&FF':78-."9'"&"F&:*6G((Q"
?#C
"
7*<8.-*7":*.8=*" 6':" 6(G87" G)9&R*4" YG:8-." 3**R7&Q<+" 92*" &-8=&(<" :*F*8M*7" 3&9*:" ':" 78(G9*7"
?$D
"
6:G89"_G8F*" 8-" 92*"*S)*:8=*-9&(" <*9OG)" G)'-" F'=)(*9*7" 9:8&(<4" /*" *-<G:*7" 92&9"92*" &-8=&(<"
?$!
"
7:&-R"<G668F8*-9"6(G87" 8-"92*"<*9OG)"&-7"<G))(*=*-9*7"92*" &-8=&(<"3892"*S9:&"6(G87"&69*:"92*"
?$#
"
:*F':78-."<*<<8'-"86"92*Q"787"-'9"7:8-R"*-'G.24"YG:8-."92*"3**R*-7+"92*Q":*F*8M*7"&"6G(("H'99(*"
?$$
"
'6" 3&9*:" ZADDOC>D" =(" )*:" 7&Q[" 8-" 92*" 2'=*" F&.*4" U2*" &-8=&(<" 3*:*" :*.G(&:(Q" F2*FR*7" HQ"
?$>
"
M*9*:8-&:Q" <9&66" &-7" &-8=&(" F&:*9&R*:<" &-7" 92*8:" 3*8.29" &-7" .*-*:&(" &))*&:&-F*" 3*:*"
?$?
"
:*F':7*7"7&8(Q"8-"&-"*(*F9:'-8F"('.H''R"7G:8-."6(G87OF'-9:'(")*:8'7<4"
?$@
"
?$A
"
0*92'7"7*9&8(<"
?$B
"
"
?$C
"
Surgeries "
?>D
"
d&F2"&-8=&(":*F*8M*7"92:**"F:&-8&("8=)(&-9<"7G:8-."<*)&:&9*"<G:.8F&("):'F*7G:*<4"U2*"
?>!
"
8=)(&-9<"3*:*"FG<9'=8]*7+"8-O2'G<*O7*<8.-*7+"$YO):8-9*7"989&-8G="F'=)'-*-9<544"V-"92*"68:<9"
?>#
"
<G:.*:Q" 3*" 8=)(&-9*7" &" 2*&7" )'<9" 6':" 2*&7" 68S&98'-+" *-<G:8-." 92&9" 92*" *Q*" 9:&FR*:" F'G(7"
?>$
"
F&)9G:*"92*"*Q*")'<898'-4"L69*:"<*M*:&("='-92<+"9'"&(('3"H'-*".:'392"&-7"&"68:="&99&F2=*-9"
?>>
"
9'"92*" <RG((+" 3*"8=)(&-9*7" &" H&<*)(&9*"8-" &" <*F'-7"<G:.*:Q4" U2*" H&<*")(&9*" 3&<" 8=)(&-9*7"
?>?
"
G-7*:"92*"<R8-"&-7"<*:M*7"&<"H&<*"6':"92*"!D#>"F2&--*("F'--*F9':"92&9"3'G(7"H*"8=)(&-9*7"
?>@
"
(&9*:+"8-"92*"928:7"<G:.*:Q544"U2*"928:7"<G:.*:Q"6'(('3*7"&69*:"<*M*:&("='-92<4"/*"8=)(&-9*7"
?>A
"
!@" `9&2" *(*F9:'7*" &::&Q<" Zb(&FR:'FR" 08F:'<Q<9*=<[+" &99&F2*7" 3892" 38:*" HG-7(*<" 9'" &"
?>B
"
FG<9'=8]*7"$YO):8-9*7"989&-8G=" )*7*<9&(" 3892" &"!D#>"F2&--*(" F'--*F9':" 92&9" 3&<"<*FG:*7"
?>C
"
3892" <F:*3<" 9'" 92*" H&<*)(&9*4" /*" HG8(9" &" <9*:*'9&F98F" 6:&=*" 92&9" &(('3*7" :'9&98'-" '6" 92*"
??D
"
&-*<92*98]*7"='-R*Q"&:'G-7"28<"&S8<4"/*"G<*7"928<"6:&=*"9'")'<898'-"VU"8-"&"=':*"2':8]'-9&("
??!
"
':8*-9&98'-+"6&F8(89&98-."92*"8-<*:98'-"'6"92*"*(*F9:'7*"&::&Q<4"U'"6G:92*:"8=):'M*"92*"&FF*<<"9'"
??#
"
LG92':"M*:<8'-"
"
#C"
VU+"3*":*='M*7" &")&:9"'6"92*"]Q.'=&98F"&:F2+" &(('38-."92*"9*=)':&(8<"=G<F(*"9'" 6(&)"9'"92*"
??$
"
<87*"&-7" 92*:*HQ" :*M*&(8-." &"(&:.*:" )':98'-" '6" <RG((554"U2*" F:&-8'9'=Q" <)&--*7"6:'=" N!" 9'"
??>
"
&-9*:8':"VU" 3892" &" 38792" '6" h#F=" &-7" &" (*-.92" '6" hA" F=" &H'M*" 92*" (*69" 2*=8<)2*:*4" `)'-"
???
"
')*-8-."'6"92*"7G:&"=&9*:+"3*"8=)(&-9*7"!@"&::&Q<+"HG9"8-"='-R*Q"W+"'-*"'6"92*"&::&Q<"F&=*"
??@
"
'G9"7G:8-."92*"8=)(&-9&98'-"&-7"3&<"FG94"U2*" ('F&98'-" '6" 92*" &::&Q<" 3&<" 7'FG=*-9*7" 3892"
??A
"
)2'9'.:&)2<4"L9"92*"*-7"'6"92*"<G:.*:Q+"3*"<G9G:*7"92*"7G:&"'M*:"92*"&::&Q<+")(&F*7"H&FR"92*"
??B
"
<RG(("6(&)"3892"989&-8G="<9:8)<+":*&99&F2*7"92*"=G<F(*<"&-7"92*-"F('<*7"92*"<R8-4""
??C
"
"
?@D
"
Cortical implant"
?@!
"
d&F2"&::&Q"F'-9&8-*7"&-"BOHQOB".:87"3892"@>" 8:878G=" 'S87*" *(*F9:'7*<4" U2*" (*-.92" '6"
?@#
"
*&F2"*(*F9:'7*"<2&-R"3&<"!4?w=="&-7"92*"<)&F8-."H*93**-"&7_&F*-9"<2&-R<"3&<">DD"x=4"U2*"
?@$
"
):*O8=)(&-9&98'-" 8=)*7&-F*" '6" 92*" *(*F9:'7*<" :&-.*7" 6:'=" ?D" 9'" A?" Ry" Z=*&<G:*7" HQ"
?@>
"
b(&FR:'FR"08F:'<Q<9*=<[4"d&F2"*(*F9:'7*"3&<"F'--*F9*7"9'"&"F'-9&F9")&7"'-"92*"K&-7OX:87O
?@?
"
L::&Q"ZKXL["8-9*:6&F*"'6"92*")*7*<9&(4";*6*:*-F*" 38:*<" 3*:*" &99&F2*7" 9'" *M*:Q" '92*:" &::&Q+"
?@@
"
&-7" *&F2" :*6*:*-F*" 38:*" <*:M*7" &<" 92*" :*6*:*-F*" 6':" 93'" &::&Q<+" <'" 92&9" 92*:*" 3*:*" *8.29"
?@A
"
:*6*:*-F*"38:*<"8-"9'9&(4"d&F2":*6*:*-F*"38:*"*S89*7"92*"38:*"HG-7(*"<*M*:&("=="H*6':*"92*"
?@B
"
)'8-9" 32*:*" 92*" 38:*" HG-7(*" =*9" 92*" &::&Q4" U2*" '92*:" *-7" '6" 92*" :*6*:*-F*" 38:*" 3&<"
?@C
"
F'--*F9*7"9'"'-*"'6"92*"F'-9&F9")&7<"'-"92*"KXL"'6"92*")*7*<9&(+"328F2"3&<"F'--*F9*7"9'""&"
?AD
"
b(&FR:'FR"f*:*)(*S"0"2*&7<9&.*4"U2*"F'--*F98'-<"3*:*"2&:738:*7"<'"92&9"92*"*(*F9:'7*<"'-"
?A!
"
*&F2")&8:"'6"&::&Q<"G<*7"92*"<8.-&("6:'="92*":*6*:*-F*"38:*"&<":*6*:*-F*4"U3'":*6*:*-F*"38:*<"
?A#
"
8-" ='-R*Q" W" 787" -'9" 3':R" &<" 8-9*-7*7+" &-7" 3*" (&9*:" :*<'(M*7" 92*" 8<<G*" HQ" <2':98-." &(("
?A$
"
:*6*:*-F*<"'-" 92*" *S)&-<8'-" H'&:74" L<"&" :*<G(9+" &((" &::&Q<" <2&:*7" 92*" <&=*" :*6*:*-F*4"U2*"
?A>
"
<&=*"3&<"9:G*"6':"='-R*Q","H*F&G<*"3*"G<*7"92*"<&=*"*S)&-<8'-"H'&:74""
?A?
"
"
?A@
"
LG92':"M*:<8'-"
"
$D"
Data acquisition"
?AA
"
d(*F9:')2Q<8'('.8F&("<8.-&(<"6:'="N!+"N>"&-7"VU" 3*:*" :*F':7*7" &9" &" <&=)(8-." :&9*" '6"
?AB
"
$DwRT]"&-7"):'F*<<*7"HQ"*PG8)=*-9"6:'="b(&FR:'FR"08F:'<Q<9*=<4"U2*"-*G:'-&("<8.-&(<"3*:*"
?AC
"
)&<<8M*(Q" F'-7GF9*7" M8&" 92*" KXL" 8-9*:6&F*" '-" 92*" !D#>OF2&--*(" )*7*<9&(" 9'" &-" *(*F9:'-8F"
?BD
"
8-9*:6&F*"H'&:7"ZdVb[+"328F2"3&<"&-"&7&)9*:"3892"$#"$@OF2&--*("^=-*98F<"F'--*F9':<+"328F2+"
?B!
"
8-"9G:-+"8-9*:6&F*7"3892"*8.29"!#BOF2&--*("f*:*%(*S"0"2*&7<9&.*<"Z<**":*64"14"6':"7*9&8(<[4"d&F2"
?B#
"
f*:*%(*S" 0" ):'F*<<*7" <8.-&(<" 6:'=" 93'" @>OF2&--*(" `9&2" &::&Q<+" &))(Q8-." &" D4$zA?DDwT]"
?B$
"
&-&('."68(9*:"&9"G-89Q".&8-"Z-'"<8.-&("&=)(868F&98'-[4"U2*"f*:*%(*S"0")*:6':=*7"&"!@OH89"&-&('.O
?B>
"
9'O78.89&(" F'-M*:<8'-" ZLYf[" 3892" &" <*-<898M89Q" '6" #?D" -NcH894" U2*" 7&9&" 6:'=" 0'-R*Q" W" &-7"
?B?
"
='-R*Q","3*:*":*F':7*7"8-"93'"7866*:*-9"<*9G)<"3892"<(8.29(Q"7866*:*-9"b(&FR:'FR"*PG8)=*-94""
?B@
"
V-"92*"<*9G)"'6"='-R*Q"W+"92*"78.898]*7"<8.-&("'-"*&F2"f*:*%(*S"0"3&<"<*-9"9'"&"!#BO
?BA
"
F2&--*("Y8.89&("TGH4"d&F2"Y8.89&("TGH":*F*8M*7"7&9&"6:'="'-*"f*:*%(*S"0+"328F2"':8.8-&9*7"
?BB
"
6:'="93'"*(*F9:'7*"&::&Q<4" U2*" Y8.89&(" TGH"F'-M*:9*7"92*"78.89&("<8.-&("8-9'"&-"')98FO78.89&("
?BC
"
6':=&9+"328F2"3&<"92*-"<*-9"M8&"&-"')98FO68H:*"F&H(*"9'"&"!#BOF2&--*("W*G:&("58.-&("%:'F*<<':"
?CD
"
ZW5%["6':"6G:92*:"):'F*<<8-."&-7"<9':&.*4"d&F2"Y8.89&("TGH"7*(8M*:*7"92*"<8.-&("9'"&"<8-.(*"W5%4"
?C!
"
U2*:*"3*:*"*8.29"W5%<"&-7"*&F2"W5%"):'F*<<*7"92*"7&9&"7*:8M*7"6:'="93'"*(*F9:'7*"&::&Q<4"
?C#
"
f'-9:'("'6"92*"W5%<"3&<"F&::8*7"'G9"'-"93'"%f<"G<8-."92*"b(&FR:'FR"f*-9:&("5'693&:*"5G89*4"
?C$
"
V-"92*"<*9G)" '6" ='-R*Q",+"92*"78.898]*7"<8.-&("'6"93'" f*:*%(*S" 0"3&<"<*-9"9'"&"#?@O
?C>
"
F2&--*(" X*=8-8" TGH+" 84*4" *&F2" X*=8-8" TGH" ):'F*<<*7" 7&9&" F'=8-." 6:'=" 93'" f*:*%(*S" 0+"
?C?
"
328F2"8-"9G:-"':8.8-&9*7"6:'="6'G:"*(*F9:'7*"&::&Q<4"d&F2"X*=8-8"TGH"<*-9"92*"78.89&("<8.-&("
?C@
"
M8&"&-"d92*:-*9"F&H(*"9'"&"?!#OF2&--*("X*=8-8"W5%"6':"6G:92*:"):'F*<<8-."&-7"<9':&.*4"U2*:*"
?CA
"
3*:*"#"X*=8-8"W5%<"Z&-7"6'G:"X*=8-8" TGH<[" 8-" 9'9&(4" d&F2" X*=8-8" W5%" ):'F*<<*7"92*"7&9&"
?CB
"
6:'="#"X*=8-8"TGH<+"&-7"92G<"B"*(*F9:'7*"&::&Q<4"f'-9:'("'6"92*"X*=8-8"W5%<"3&<"&(<'"F&::8*7"
?CC
"
'G9"'-"93'"%f<"G<8-."92*"b(&FR:'FR"f*-9:&("5'693&:*"5G89*4"
@DD
"
LG92':"M*:<8'-"
"
$!"
L-"8-6:&:*7"*Q*"9:&FR*:"ZU;df"dUO>Cb+"U2'=&<";*F':78-."X=HT["3&<"G<*7"9'"<&=)(*"
@D!
"
92*"*Q*")'<898'-"&-7")G)8("78&=*9*:"Z='-R*Q"W"'-(Q["6':"H'92"92*"uO"&-7"qO&S*<"3892"&"6:&=*"
@D#
"
:&9*"'6"#$DwT]+"&-7"92*"7&9&"3*:*"<9':*7"&9"&"<&=)(8-.":&9*"'6" $DwRT]4"U2*"*Q*O9:&FR*:"3&<"
@D$
"
F'-9:'((*7"HQ"&"7*78F&9*7"%f+"328F2"6':3&:7*7"92*"&-&('."<8.-&(<":*.&:78-."92*"*Q*")'<898'-"
@D>
"
&-7")G)8("78&=*9*:"Z8-"='-R*Q"W["78:*F9(Q"9'"'-*"'6"92*"W5%cX*=8-8"W5%<4""
@D?
"
L"<98=G(G<"F'-9:'("F'=)G9*:" 3&<" G<*7" 6':" 92*"*S*FG98'-"'6"9&<RO:*(&9*7"*M*-9<" &-7"
@D@
"
<98=G(8" 3892" 28.2" 9*=)':&(" ):*F8<8'-" 7G:8-." *&F2" 92*" 9&<R4" L-" *M*-9" :*(&9*7" F'7*+"
@DA
"
F'::*<)'-78-." 9'" 92*" <98=G(G<" '-<*9" 3&<" <*-9" 'G9" 6:'=" 92*" %f" M8&" &" Y&9&" LFPG8<898'-" &-7"
@DB
"
f'-9:'("5Q<9*="ZYL5["0G(986G-F98'-"L-&('."&-7"Y8.89&("Vc^"H'&:7"ZYL5!@D#c!@+"0*&<G:*=*-9"
@DC
"
f'=)G98-.[4"V-" ='-R*Q"W+"92*"<98=G(G<"'-<*9"3&<"7*(8M*:*7" 92:'G.2"&"<)(899*:"F&H(*+"9'"92*"
@!D
"
78.89&(" 8-)G9" )':9<" Z!@OH89" Yb$A[" '-" *&F2" '6" 92*" W5%<4" U2*" F'::*<)'-78-." F2&--*(<" '-" 92*"
@!!
"
78.89&("8-)G9")':9"'6"*&F2"W5%"<&=)(*7"92*"8-F'=8-."<8.-&("&9"$DwRT]"&-7"3*:*"F'-68.G:*7"9'"
@!#
"
7*9*F9"32*-"&-" 8-F'=8-."H89"3&<"<*9"9'" &"m28.2n"M&(G*"'-"'-*" '6"92*")8-<4"V-"='-R*Q",+" 92*"
@!$
"
<98=G(G<"'-<*9"3&<"7*(8M*:*7"92:'G.2"&-"&-&('."F'--*F98'-"9'"H'92"X*=8-8"W5%<4"
@!>
"
"/*"G<*7"92*" <8.-&(" 6:'=" &")2'9'78'7*"9'" :*F':7" 92*" ):*F8<*" 98=8-."'6"92*" <98=G(G<"
@!?
"
'-<*94"/*")(&F*7"92*"78'7*"'-"92*"('3*:"(*69"<F:**-+"&-7"3*"):*<*-9*7"&"<=&(("3289*"<PG&:*"
@!@
"
Z8-M8<8H(*"9'"92*"='-R*Q["8-"<Q-F"3892"92*"8=&.*"):*<*-9&98'-4"U2*"78'7*"<*-9"'G9"&-"&-&('."
@!A
"
M'(9&.*"<8.-&("92&9"3&<"<&=)(*7"&9"$DRT]"HQ"W5%J!"6':"='-R*Q"W+"&-7"HQ"H'92"X*=8-8"W5%<"
@!B
"
6':"='-R*Q",4""
@!C
"
V-"='-R*Q"W+"&-&('."<8.-&(<"3*:*":*F':7*7"8-"92*"68:<9"'6"92*" B"W5%<"Z84*4"W5%"J!["&<"
@#D
"
&77898'-&("F2&--*(<4"u"&-7"q"*Q*")'<898'-<"3*:*":*F':7*7"'-"F2&--*(<"!#C"&-7"!$D"'6"W5%"J!+"
@#!
"
:*<)*F98M*(Q+" 328(*" u" &-7" q" )G)8(" 78&=*9*:" 3*:*" :*F':7*7" '-" F2&--*(<" !$!" &-7" !$#+"
@##
"
:*<)*F98M*(Q4" f2&--*(" !>$" :*F':7*7" 92*" 78'7*" <8.-&(4" Y8.89&(" 'G9)G9<" 3*:*" :*F':7*7" 8-" 92*"
@#$
"
LG92':"M*:<8'-"
"
$#"
*M*-9<"68(*"Z4-*M["HQ"92*"W5%<+"&-7"92*"9:8&(O:*(&9*7"*M*-9"92&9"'FFG::*7"&9"92*"='=*-9"92&9"
@#>
"
92*"H89"3&<"<*9"9'"m28.2n4"
@#?
"
V-"='-R*Q",+"&-&('."<8.-&(<"3*:*":*F':7*7"8-"&"<*)&:&9*"4-<@"68(*4"f2&--*("!":*F':7*7"
@#@
"
92*"<98=G(G<"'-<*9"'6"H'92"X*=8-8"W5%<4"u"&-7"q"*Q*")'<898'-<"3*:*":*F':7*7"'-"F2&--*(<"#"
@#A
"
&-7"$"'6"X*=8-8"W5%J!+" :*<)*F98M*(Q4" U2*" 78'7*" <8.-&(" 3&<":*F':7*7"8-"F2&--*(" !?" '6" H'92"
@#B
"
X*=8-8"W5%<4"
@#C
"
U2*":*F':78-.<"6:'="92*"B"W5%<"8-"='-R*Q"W+"&-7"92*"#"X*=8-8"W5%<"8-"='-R*Q","3*:*"
@$D
"
&<Q-F2:'-'G<"&-7"3*:*":*O&(8.-*7"'66(8-*"G<8-."92*"<98=G(G<"'-<*9"8-6':=&98'-4",':"='-R*Q"
@$!
"
W+":*F':78-.<"6:'="92*"B"W5%<"3*:*"&(8.-*7"9'"92*"78.89&("'G9)G9"F'::*<)'-78-."9'"92*"<98=G(G<"
@$#
"
'-<*94",':"='-R*Q",+"92*"&-&('."<8.-&("F'::*<)'-78-."9'"92*"<98=G(G<"'-<*9"3&<"G<*74"V-"H'92"
@$$
"
='-R*Q<+" 92*" 78'7*" <8.-&(" 3&<" G<*7" 9'" 87*-986Q" 92*" F'::*F9" <98=G(G<" '-<*9" 98=*" 6':" )'<9O
@$>
"
):'F*<<8-.+"&<"92*"<8.-&("F'=8-."6:'="92*"<98=G(G<"%f"3'G(7"G<G&((Q"H*"7*(&Q*7"HQ"'-*"':"93'"
@$?
"
<F:**-"6(8)<4""
@$@
"
U2*")GH(8<2*7"7&9&"8-F(G7*"-*G:'-&("&F98M89Q"6:'="&((":*F':78-."<89*<+"92*"*Q*O)'<898'-"
@$A
"
<8.-&(+"92*"<8.-&("6:'="&")2'9'78'7*"92&9"):'M87*7"):*F8<*"8-6':=&98'-"&H'G9"92*"'-<*9"'6"92*"
@$B
"
M8<G&("<98=G(8+"&-7"<8.-&(<":*(&9*7"9'"98=8-." '6" 92*" '-<*9" '6" 92*" 9:8&(" &-7"92*"7*(8M*:Q"'6"92*"
@$C
"
6(G87":*3&:74""
@>D
"
"
@>!
"
MUA pre-processing
@>#
"
V-"&77898'-"9'"92*"$Dj2]":&3" <8.-&(+" 3*" &(<'" ):'M87*" 92*"):*O):'F*<<*7"0`L"7G:8-."
@>$
"
92*"):*<*-9&98'-"'6"92*"8=&.*<"92&9"3*:*" F'=)(*9*7"HQ"92*"='-R*Q+"328F2"8=)(8*<"92&9"2*"
@>>
"
=&8-9&8-*7" 68S&98'-" 6':" 92*" 7G:&98'-" '6" 92*" <98=G(G<" )(G<"#DD=<4" /*" &(<'" 8-F(G7*" 92*"
@>?
"
-':=&(8]*7" -*G:'-&(" :*<)'-<*" 6':" *&F2" 9:8&(" &-7" *(*F9:'7*+" &M*:&.*7" 8-" &" 98=*" 38-7'3"
@>@
"
F*-9*:*7"&:'G-7"92*")*&R"'6":*<)'-<*+"&('-."3892"92*"5W;+"(&9*-FQ+"&-7":*(8&H8(89Q"=*&<G:*<4"
@>A
"
LG92':"M*:<8'-"
"
$$"
U'" .*-*:&9*" 92*" 0`L+" 92*" :&3" 7&9&" 3*:*" 68(9*:*7" H*93**-" D4?" &-7" ?wRT]4" L" 6G((O3&M*"
@>B
"
:*F9868F&98'-"3&<")*:6':=*7"'-"92*"68(9*:*7"<8.-&(+"6'(('3*7"HQ"&"('3O)&<<"68(9*:"'6"#DDwT]"Z*4.4"
@>C
"
39,56[4" L77898'-&(" H&-7O<9')" 68(9*:<" 3*:*" G<*7" 9'" :*='M*" ?D2]" &-7" @D2]" -'8<*" Z&-7" =&8-"
@?D
"
2&:='-8F<[4" ,8(9*:8-." 3&<" F&::8*7" 'G9" G<8-." &" bG99*:3':92" 68(9*:" '6" ':7*:" #4" U2*" 7&9&" 3*:*"
@?!
"
7'3-O<&=)(*7"HQ"&"6&F9':"'6"$D+"Q8*(78-."&"0`L"<8.-&("3892"&"<&=)(8-.":&9*"'6"!wRT]4"U2*"0`L"
@?#
"
:*{*F9<" 92*" )')G(&a'-" <)8R8-." '6" -*G:'-<" 38928-" !DDO!?D" µ=" '6" 92*" *(*F9:'7*" &-7" 92*"
@?$
"
)')G(&a'-":*<)'-<*<"&:*"M*:Q"<8=8(&:"9'"92'<*"'H9&8-*7"HQ")''(8-."&F:'<<"<8-.(*"G-89<16,19"4"
@?>
"
U2*"-':=&(8]*7":*<)'-<*<"&:*" &(<'" ):'M87*7+" &<"&"<8=)(*"HG9"G<*6G("8-)G9"6':"='7*("
@??
"
9:&8-8-."&-7"9*<98-.4",8:<9+"3*"&M*:&.*7"92*":*<)'-<*"&F:'<<"8=&.*<"6':"*M*:Q":*F':78-."<89*"8-"
@?@
"
&"98=*O38-7'3"F*-9*:*7" '-"92*")*&R":*<)'-<*"'6"*&F2" :*.8'-I" #?O!#?=<"6':"N!+"?DO!?D=<"
@?A
"
6':" N>" &-7" A?O!A?=<" 6':" VU4" U2*-+" 3*" -':=&(8]*7" 92*" 7&9&" Z&<" 8-" :*64" 7[4" ,':" *&F2" 7&Q" '6"
@?B
"
:*F':78-.+"3*"F'=)G9*7"92*"=*&-"&-7"<9&-7&:7"7*M8&98'-"'6"92*"&F98M89Q"*(8F89*7"HQ"92*"9*<9"
@?C
"
8=&.*<4"/*"F&((*7"928<"<GHO.:'G)"'6"9:8&(<"mtest pool’4"U2*"-':=&(8]*7"0`L"6':"*&F2"9:8&("
𝑡
"&-7"
@@D
"
*(*F9:'7*"
𝑛
"3&<"92*-"F&(FG(&9*7"&<"6'(('3<I"
@@!
"
𝑁𝑜𝑟𝑚𝑀𝑈𝐴*!
"#$ =*%&'!
"
#
$
(%)*#
"
#
$
!%&!'())*
+,
"
#
$
!%&!'())*
"""
@@#
"
"/*"-'9*"92&9"&"6*3"VU":*F':78-."<89*<"8-"='-R*Q","ZA"<89*<["&-7"8-"='-R*Q"W"Z?#"<89*<["
@@$
"
*S28H89*7"&" H:8*6+"9:&-<8*-9":*<)'-<*"H*93**-"<98=G(G<"'-<*9"&-7"92*":*.G(&:":*<)'-<*4"/*"
@@>
"
787"-'9"87*-986Q"89<"<'G:F*"HG9"<G<)*F9"&"-*G:'-&("':8.8-"Z*4.4"&S'-<"6:'="92*"KXW")&<<8-."F('<*"
@@?
"
9'"92*" *(*F9:'7*<[4"U2*" 98=*O38-7'3<" 92&9" 3*" F2'<*" 6':" 7&9&" &-&(Q<8<" 787" -'9" 8-F(G7*" 928<"
@@@
"
9:&-<8*-9"<8.-&(4"`-(*<<"928<"9:&-<8*-9":*<)'-<*"8<"'6"8-9*:*<9"6':"&"<)*F868F":*<*&:F2"PG*<98'-+"
@@A
"
3*" <G..*<9" 9'" *892*:" G<*" 92*" 98=*O38-7'3<" 3*" ):')'<*7+" 328F2" *SF(G7*" 928<" 9:&-<8*-9"
@@B
"
:*<)'-<*+"':"9'"87*-986Q"&-7"*SF(G7*"92'<*":*F':78-."<89*<"6:'="92*"&-&(Q<*<4"
@@C
"
@AD
"
LG92':"M*:<8'-"
"
$>"
Data formats"
@A!
"
L((" 7&9&" 8<" &M&8(&H(*" &9" &" XOW'7*" XVW" :*)'<89':Q" Z<**" H*('3" sY&9&" &-7" F'7*"
@A#
"
&M&8(&H8(89Qt[4"U2*" :&3"7&9&"&:*"):'M87*7"8-"92*"):'):8*9&:Q"b(&FR:'FR"6':=&9+"4-<@4"U2*"):*O
@A$
"
):'F*<<*7"0`L"<8.-&(<"&-7"92*"('."68(*<"&:*"<9':*7"&<"4=&9"68(*<"92&9"F&-"H*"('&7*7"G<8-."*4.4"
@A>
"
%Q92'-"':"0&9(&H4" b&<8F"=*9&7&9&"&H'G9"92*":*F':78-."<Q<9*=" &:*"<&M*7"8-"92*"):'):8*9&:Q"
@A?
"
b(&FR:'FR"6':=&9<+"4-*M" &-7" 4FF64"^)*-"<'G:F*"F'7*"9'"('&7"92*":&3" <8.-&(" 8<"&M&8(&H(*"6:'="
@A@
"
b(&FR:'FR"08F:'<Q<9*=<"6':" H'92" %Q92'-" &-7" 0&9(&H"G<*:<4"L" F')Q" '6" 92*"0&9(&H"(8H:&:Q" 8<"
@AA
"
<9':*7"&('-."3892"92*"7&9&"&9"299)<Icc7'84':.c!D4!#A?!c.O-'7*42FA](M4"
@AB
"
"
@AC
"
Stimulus presentation
@BD
"
/*"):*<*-9*7"##+#>B"-&9G:&("8=&.*<"6:'="92*"UTVWX5"8=&.*"7&9&H&<*12"F'-<8<98-."'6"
@B!
"
!#"8=&.*<"'6"*&F2"'6"!+B?>"<98=G(G<"F&9*.':8*<+"?DDS?DD")8S*(<"*&F24"/*"G<*7"7866*:*-9"f;U"
@B#
"
='-89':<"6':"92*" 93'" ='-R*Q<4",':"='-R*Q"W+"3*"G<*7"&" f;U" ='-89':"3892":*6:*<2":&9*"'6"
@B$
"
B?T]"&-7" :*<'(G98'-" '6" !D#>SA@B" )8S*(<+" 328F2" 3&<" M8*3*7"6:'=" &" 78<9&-F*" '6" >AF="Z#B4C"
@B>
"
)8S*(<" )*:" 7*.:**[4" ,':" ='-R*Q" ,+" 3*" G<*7" &" f;U" ='-89':" 3892" :*6:*<2" :&9*" '6" @DT]" &-7"
@B?
"
:*<'(G98'-"'6" !D#>SA@B" )8S*(<+" 328F2"3&<" M8*3*7" 6:'=" &"78<9&-F*" '6" ?BF=" Z#@4C")8S*(<")*:"
@B@
"
7*.:**[4"V-"H'92"<*9G)<+"<98=G(8"3*:*".*-*:&9*7"8-"0&9(&H"G<8-."92*"f^XdWU"X:&)28F<"9''(H'S"
@BA
"
Z7*M*(')*7" HQ" r'2-" ;'=&Q&" &9" 92*" K^W" &9" 92*" /*((F'=*" Y*)&:9=*-9" '6" V=&.8-."
@BB
"
W*G:'<F8*-F*[" &-7" FG<9'=" F'-9:'(" <'693&:*574" U'" *-<G:*" 92&9" &((" ;,<" 3'G(7" 6&((" 8-<87*" 92*"
@BC
"
8=&.*+"3*"<2869*7"92*"<98=G(8"!DD" )8S*(<" 7'3-3&:7" &-7" !DD" )8S*(<"92*":8.29":*(&98M*"9'"92*"
@CD
"
F*-9*:" '6" .&]*4" /*" )<*G7':&-7'=8]*7" 92*" ':7*:" '6" ):*<*-9&98'-" '6" 8=&.*<" 6':" 92*" *-98:*"
@C!
"
*S)*:8=*-9")*:"='-R*Q"H*6':*"92*"H*.8--8-."'6"92*"*S)*:8=*-94"U*<9"8=&.*<"2&7"$D"*-9:8*<"
@C#
"
8-"92*":&-7'=8]&98'-"9&H(*"&-7"92*"'92*:"8=&.*<"'-(Q"'-*4"
@C$
"
LG92':"M*:<8'-"
"
$?"
L9"92*"<9&:9" '6" 92*" 9:8&(+"92*" ='-R*Q" 78:*F9*7" 89<".&]*" 9'" &" :*7"68S&98'-" )'8-9" 3892" &"
@C>
"
78&=*9*:"'6"D4$"7*."'-"&".:&Q"H&FR.:'G-7"Z(G=8-&-F*">D4!"F7c=2"6':"='-R*Q"We"!?4A"F7c=2"
@C?
"
6':"='-R*Q",[4"/*" ):*<*-9*7" 92*" 68:<9" 8=&.*" '-F*"92*"='-R*Q"2&7"=&8-9&8-*7"68S&98'-"6':"
@C@
"
$DD=<"8-"&"68S&98'-"38-7'3"3892"&"78&=*9*:"'6"!7M&4"U'":*F*8M*"&":*3&:7+"92*"&-8=&("2&7"9'"
@CA
"
=&8-9&8-" 89<" .&]*" 8-" 928<" 38-7'3" 7G:8-." 92*" <*PG*-F*" '6" 6'G:" 8=&.*<+" 328F2" 3*:*" *&F2"
@CB
"
):*<*-9*7" 6':" #DD=<+" 6'(('3*7" HQ" #DD=<" '6" 3892" &" .:&Q" <F:**-4" /*" ('..*7" 8=&.*<" &<"
@CC
"
F'=)(*9*7"32*-"92*"='-R*Q"=&8-9&8-*7"68S&98'-"7G:8-."92*8:"):*<*-9&98'-"&-7"92*"6'(('38-."
ADD
"
*)'F2"'6"#DD=<"3892"&".:&Q"<F:**-4"V6"9:8&(<"3*:*"&H':9*7"H*F&G<*"92*"='-R*Q"H:'R*"68S&98'-+"
AD!
"
F'=)(*9*7"8=&.*<"3*:*"-'9"<2'3-"&.&8-+"&-7"92*"'92*:"8=&.*<"'6"92*"&H':9*7"<*PG*-F*"3*:*"
AD#
"
):*<*-9*7"'-"92*"-*S9"9:8&(4"
AD$
"
AD>
"
Signal-to-noise ratio (SNR)
AD?
"
U'"&<<*<<"92*"PG&(89Q"'6"92*" :*F':7*7" -*G:'-&(" &F98M89Q+" 3*" F&(FG(&9*7" 92*"<8.-&(O9'O
AD@
"
-'8<*":&98'"Z5W;["6':"*&F2":*F':78-."<89*4"U2*"5W;"8<"92*"M8<G&((Q" 7:8M*-":*<)'-<*":*(&98M*" 9'"
ADA
"
H&<*(8-*"&F98M89Q4"/*"F&(FG(&9*7"92*"=*&-"&-7"<9&-7&:7"7*M8&98'-"Z5Y["'6"92*"0`L"7G:8-."&"
ADB
"
!DDO=<" 98=*" H&<*(8-*" *)'F2" ):8':" 9'" <98=G(G<" '-<*9" 6':" *&F2" 9:8&(4" W*S9+" 3*" &M*:&.*7" 92*"
ADC
"
:*<)'-<*"&F:'<<"&(("8=&.*<+"<=''92*7"89"3892"&"='M8-."38-7'3"'6"#?=<"&-7"87*-9868*7"92*"
A!D
"
)*&R"(*M*("'6"&F98M89Q4"U2*"5W;"3&<"92*-"F&(FG(&9*7"&<"6'(('3<I"
A!!
"
"
A!#
"
𝑆𝑁𝑅=*-)*.&!+,-*-&'./%0.1%(%)*#2.&%*+#%
+,2.&%*+#%
"""
A!$
"
" "
A!>
"
b*F&G<*"<'=*"<98=G(8"*(8F89*7"8-28H89':Q":*<)'-<*<"&9"<'=*"'6"92*":*F':78-."<89*<+"3*"
A!?
"
):'M87*"&-"&77898'-&("=*&<G:*+"
𝑆𝑁𝑅/*0,*
328F2"8<"H&<*7"'-"92*":*<)'-<*"9'"92*"):*6*::*7"9*<9"
A!@
"
8=&.*I"
A!A
"
LG92':"M*:<8'-"
"
$@"
"
A!B
"
𝑆𝑁𝑅/*0 =*-)*.(0%3%00%4'!%&!'+,.1%(%)*#2.&%*+#%
+,2.&%*+#%
"""
A!C
"
" "
A#D
"
Latency estimation
A#!
"
U'"F'=)G9*"92*"(&9*-FQ"'6"-*G:&(":*<)'-<*<"&9"8-78M87G&(":*F':78-."<89*<+"3*"6899*7"&"
A##
"
6G-F98'-"9'"92*"98=*OF'G:<*"'6"92*":*<)'-<*"&M*:&.*7"&F:'<<"9:8&(<+"<*)&:&9*(Q"6':"*&F2"7&Q"'6"
A#$
"
:*F':78-.56,584"U2*"6G-F98'-"3&<"7*:8M*7"6:'="92*"&<<G=)98'-"92&9"92*"'-<*9"'6"92*":*<)'-<*"
A#>
"
2&7" &" X&G<<8&-" 78<9:8HG98'-" &F:'<<" 9:8&(<+" &-7" 92&9" &" 6:&F98'-" '6" 92*" :*<)'-<*" 78<<8)&9*7"
A#?
"
*S)'-*-98&((Q+"Q8*(78-."92*"6'(('38-."*PG&98'-I"
A#@
"
"
A#A
"
𝑓(𝑡)=*𝑑exp*(𝜇𝛼+0.5𝜎1𝛼1𝛼𝑡)(𝐺
(
t,µ+σ1α,σ
)
+𝑐𝐺(t,µ,σ)
"
A#B
"
"
A#C
"
32*:*"
𝐺(t,µ,σ)
"8<"&"FG=G(&aM*"X&G<<8&-"7*-<89Q"3892"=*&-"
𝜇
"&-7"<9&-7&:7"7*M8&a'-"
𝜎
+"
𝛼(2
"
A$D
"
8<"92*"a=*"F'-<9&-9"'6" 92*" 78<<8)&a'-+" &-7"
𝑐
"&-7"
𝑑
":*):*<*-9"92*" F'-9:8HGa'-" '6" 92*" -'-O
A$!
"
78<<8)&a-."&-7"78<<8)&a-."F'=)'-*-9<4"U2*"6G-Fa'-"3&<"g9"G<8-."&"-'-O(8-*&:"(*&<9"<PG&:*<"
A$#
"
=*92'7"Zfit.m"8-"0LUKLb[4"U2*"(&9*-FQ"3&<"7*g-*7"&<"92*")'8-9"&9"328F2"92*"g1*7"6G-Fa'-"
A$$
"
:*&F2*7"$$l"'6"89<"=&S8=G=4"U2*"(&9*-FQ"*<a=&9*<"6':":*F':78-."<89*<"8-",8.G:*"#"2&M*"H**-"
A$>
"
&M*:&.*7" &F:'<<" :*F':78-." 7&Q<4" U2*" a=8-." '6" 92*" <a=G(G<" '-<*9" 3&<" 7*9*:=8-*7" 3892" &"
A$?
"
)2'9'78'7*+"328F2"3&<")(&F*7"8-"92*"(*kOH'1'="F':-*:"'6"92*"f;U4"/*"*<a=&9*"92&9"92*"7*(&Q"
A$@
"
H*93**-"92*"&))*&:&-F*"'6"92*"<a=G(G<"8-"92*"-*G:'-<n";,"&-7"92*"<8.-&("'6"92*")2'9'78'7*"
A$A
"
3&<"D4C"9'"$4C=<+"F&G<8-."&-"G-7*:*<a=&a'-"'6"92*"-*G:'-&("(&9*-F8*<"HQ"&"6*3"=<4"U2*"VU"
A$B
"
:*F':78-."<89*<"3892"&"<2':9"(&9*-FQ"9:&-<8*-9":*<)'-<*"8-"='-R*Q"W"Z<**"&H'M*["3*:*"*SF(G7*7"
A$C
"
32*-"F'=)Ga-."92*"(&9*-FQ"78<9:8HGa'-"Z,8.G:*"#[4""
A>D
"
"
A>!
"
LG92':"M*:<8'-"
"
$A"
Reliability estimations
A>#
"
" U'"*<98=&9*" 92*" :*(8&H8(89Q" '6"92*" 0`L" <8.-&(" &9"&" :*F':78-." <89*+" 3*"F'=)G9*7" 93'"
A>$
"
=*&<G:*<e"92*"=*&-":*(8&H8(89Q"&-7"92*"':&F(*"F'::*(&98'-54"U2*"=*&-":*(8&H8(89Q"3&<"7*68-*7"
A>>
"
&<"92*"&M*:&.*"'6"92*")&8:38<*"F'::*(&98'-<"H*93**-"*&F2"F'=H8-&98'-"'6":*)*9898'-<"9'"92*"
A>?
"
!DD"9*<9"8=&.*<4",':"92*"':&F(*"F'::*(&98'-+"3*" F'=)G9*7"92*"&M*:&.*"F'::*(&98'-"H*93**-"
A>@
"
92*"&F98M89Q"*(8F89*7" HQ" &" <8-.(*"):*<*-9&98'-" '6" *&F2" 9*<9"8=&.*+"&-7" 92*" &M*:&.*":*<)'-<*"
A>A
"
*(8F89*7"HQ"92*":*=&8-8-."#C":*)*9898'-<"'6"92*"9*<9"8=&.*"Z84*4"(*&M*O!O'G9[4"/*":*)*&9*7"92*"
A>B
"
):'F*7G:*"6':" *&F2" '6" 92*"$D" 8=&.*" :*)*9898'-<" &-7"92*-" &M*:&.*7" &F:'<<"92*" :*<G(98-." $D"
A>C
"
F'::*(&98'-"M&(G*<4""
A?D
"
U2*"':&F(*"F'::*(&98'-"):'M87*<"&-"*<98=&9*"'6"92*"-'8<*"F*8(8-.+"328F2"8<"92*"=&S8=G="
A?!
"
&F28*M&H(*" F'::*(&98'-" '-" <8-.(*" 9:8&(<4" U2*" =*&-" :*(8&H8(89Q" 8<" H&<*7" '-" 92*" F'::*(&98'-"
A?#
"
H*93**-"&F98M89Q"(*M*(<"'-"8-78M87G&("9:8&(<+"&-7"89"8<"92*:*6':*"*S)*F9*7"9'"H*"('3*:"92&-"92*"
A?$
"
':&F(*"F'::*(&98'-4" U2*"):*78F98'-"'6"&".''7"='7*("'6"92*"&F98M89Q"'6"-*G:'-<"&9"&":*F':78-."
A?>
"
<89*"'-"&"<8-.(*"9:8&("8<"*S)*F9*7"9'"&)):'&F2"92*"':&F(*"F'::*(&98'-+"HG9"9'"H*"28.2*:"92&-"92*"
A??
"
=*&-":*(8&H8(89Q4"/*"92*:*6':*"<G..*<9"G<8-."92*"oracle"F'::*(&98'-+"92*"=*&-":*(8&H8(89Q"':"H'92+"
A?@
"
8-<9*&7"'6"92*"5W;+"9'"8-F(G7*":*F':78-."<89*<"8-"='7*("9:&8-8-."&-7"'92*:"&-&(Q<*<4"
A?A
"
"
A?B
"
Showcase #1 - Convolutional neural network (CNN) and most exciting inputs (MEIs)
A?C
"
" /*"G<*7"92*"-':=&(8]*7"0`L"&<"8-)G9"9'"92*"):*78F98M*"='7*(4"/*"G<*7"92*"&M*:&.*"
A@D
"
:*<)'-<*"8-"92*"98=*"38-7'3"<)&--8-."92*")*&R":*<)'-<*"Z#?O!#?=<"6':"N!+"?DO!?D=<"6':"N>+"
A@!
"
A?O!A?=<"6':"VU[" 9'" 92*"8=&.*<" 8-" 92*" 9:&8-8-."<*94" /*" 8-F(G7*7" *(*F9:'7*<"3892" &" =8-8=&("
A@#
"
:*(8&H8(89Q"8-7*S"'6"D4$"&-7"6899*7"<*)&:&9*"='7*(<"6':"*&F2"&::&Q4"/*"G<*7"&"):*O9:&8-*7"fWW"
A@$
"
ZsV-F*)98'-" M!t27[" &-7" 689" &" =&))8-." 6G-F98'-" 9'" 92*" &F98M&98'-<" '6" &" <)*F868F" (&Q*:7,594"
A@>
"
5)*F868F&((Q+"6':"*&F2"-*G:'-"3*"(*&:-*7"&"<*9"'6"#Y"<)&98&("3*8.29<"Z
𝑊3
["3892"<8]*"*PG&("9'"92*"
A@?
"
LG92':"M*:<8'-"
"
$B"
-G=H*:"'6")8S*(<"&-7"&"<*9"'6"6*&9G:*"3*8.29<"Z
𝑊4
["3892"<8]*"*PG&("9'"92*"-G=H*:"'6"F2&--*(<"
A@@
"
'6"92*"fWW"(&Q*:4"L69*:"9:&8-8-.+"92*"
𝑊3
":*):*<*-9<"92*"<)&98&(";,"'6"*&F2"*(*F9:'7*+"&-7"92*"
A@A
"
𝑊4
"92*"6*&9G:*" <*(*F98M89Q" &<"&"3*8.29*7"<G="'M*:"92*"6*&9G:*<" '6" 92*"fWW"(&Q*:4"U2G<+"92*"
A@B
"
):*78F9*7":*<)'-<*"'6"&-"*(*F9:'7*"
𝑛
"Z84*4+"
𝑦#
G["3&<"F'=)G9*7"&FF':78-."9'I"
A@C
"
"
AAD
"
𝑦#
G
=
HI
𝑊3
"
#
$
𝑋5
670)53
L
*𝑊4
"
#
$"
AA!
"
"
AA#
"
/2*:*"
𝑋5
"&:*"92*"H&9F2O-':=&(8]*7"&F98M&98'-<"6:'="92*"9&:.*9"fWW"(&Q*:"
𝑙
+"&-7"
":*):*<*-9<"
AA$
"
F'-M'(G98'-4"U2*"3*8.29<"3*:*"_'8-9(Q"')98=8]*7"&F:'<<"&(("*(*F9:'7*<"'6"92*"&::&Q"9'"=8-8=8]*"
AA>
"
92*" ):*78F98'-" *::':"
)8898
+" :*.G(&:8]*7" HQ" &" F'=H8-&98'-" '6" &" <=''928-." K&)(&F8&-" ('<<"
AA?
"
Z
:*65*;)
+"<**"H*('3["'M*:"
𝑊3
7+"9'"*-F'G:&.*"<=''92"<)&98&(";,<+"&"K!"('<<"Z
2
I"<**"H*('3["
AA@
"
'M*:"
𝑊4
4,5 9'"*-F'G:&.*"<)&:<89Qc<*(*F98M89Q"'6"92*"6*&9G:*<"&-7"&"K#"('<<"Z
1
I"<**"H*('3["'M*:"
AAA
"
H'92")&:&=*9*:<5,74"U2G<+"92*"F'<9"6G-F98'-"3&<"F'=)G9*7"&<"6'(('3<I"
AAB
"
"
AAC
"
)8898 =
OI
(𝑦#
G
𝑦#)1
#
"
ABD
"
:*65*;) =𝜆3
OI
(𝑊3
"
#
$
𝐿)1
#,𝐿=*
R
0 −1 0
−1 4 −1
0 −1 0
U"
AB!
"
"
AB#
"
2=𝜆4
IV
𝑊4
"#$
V
#
"
AB$
"
"
AB>
"
1=𝜆1
I
(𝑊3
"#$1+𝑊4
"#$1)
#
"
AB?
"
LG92':"M*:<8'-"
"
$C"
"
AB@
"
=)8898 +*:*65*;) +*2*+*1
"
ABA
"
"
ABB
"
/2*:*"
𝑦#
"8<"92*":*<)'-<*"'6"*(*F9:'7*"
𝑛
4"U2*"2Q)*:O)&:&=*9*:<"Z
𝑙
+"
𝜆3
+"
𝜆4
+"
𝜆1
"&-7"92*"(*&:-8-."
ABC
"
:&9*["3*:*"<*(*F9*7"G<8-."&".:87"<*&:F2"6':"92*"28.2*<9"F'::*(&98'-"8-"&"H&9F2"'6"8=&.*<"'6"92*"
ACD
"
9:&8-8-."<*94"U2*"='7*("3&<"8=)(*=*-9*7"8-"%QU':F260"&-7"')98=8]*7"3892"L7&=61"8-" !+DDD"
AC!
"
*)'F2<"3892"&"H&9F2"<8]*"'6"!DD"&-7"*&:(Q"<9'))8-."Z*M*:Q"$$$"*)'F2<+"7*F&Q"6&F9':"p"$[4"U2*"
AC#
"
F:'<<OM&(87&9*7")*:6':=&-F*"'6"92*"='7*("3&<"&<<*<<*7"HQ"F'=)G98-."92*"F'::*(&98'-"'6"92*"
AC$
"
='7*("):*78F98'-<"3892"92*":*<)'-<*<"8-"92*"9*<9"<*9+"&M*:&.*7"&F:'<<":*)*9898'-<+"328F2"3*:*"
AC>
"
-'9"G<*7"6':"='7*("9:&8-8-.4"
AC?
"
L69*:" 9:&8-8-.+" 3*" .*-*:&9*7" 92*" 0dV<" HQ" 7*9*:=8-8-." 92*" )8S*(<" 8-" 92*" ;," 92&9"
AC@
"
=&S8=8]*"92*"):*78F9*7"='7*(":*<)'-<*"'6"*&F2":*F':78-."<89*"8-7*)*-7*-9(Q4"U'"*-<G:*"92*"
ACA
"
<9&H8(89Q"&-7"8-9*:):*9&H8(89Q"'6"92*"0dV<+"3*"8-9:'7GF*7"):*F'-7898'-8-."&-7":*.G(&:8]&98'-<"
ACB
"
9'" &M'87" 28.2O6:*PG*-FQ" &:986&F9<62+" &-7" <)*F868F&((Q" 6:*PG*-FQ" )*-&(8]&98'-" &-7"
ACC
"
9:&-<6':=&98'-<"Z)&778-.+"_899*:8-.+":'9&98'-"&-7"<F&(8-.[4"^)98=8]&98'-"'6"92*"0dV<"3&<"7'-*"
BDD
"
G<8-."KGF*-9"Z84*4"%QU':F2"8=)(*=*-9&98'-"'6"KGF87"*(*F9:'7*62["3892" L7&=" Z(*&:-8-." :&9*" p"
BD!
"
D4D!+"3*8.29"7*F&Q"p"D4DD!["8-"?D"*)'F2<4"
BD#
"
"
BD$
"
Showcase #2 – Analysis of noise correlations across the temporal stream
BD>
"
" /*"8-F(G7*7"92*":*F':78-."<89*<"'6"52'3F&<*"J!"&-7"G<*7"92*";,"*<98=&9*<"'H9&8-*7"
BD?
"
3892"92*"fWW"9'"F'=)G9*"92*"dGF(87*&-"78<9&-F*"H*93**-"92*";,<"'6")&8:<"'6":*F':78-."<89*<4"
BD@
"
/*"*S&=8-*7"&9"92*"-*G:'-<n"&F98M89Q"8-"93'"98=*O38-7'3<I"92*"<)'-9&-*'G<")2&<*"Z!DD=<"
BDA
"
H*6':*"<98=G(G<"'-<*9["&-7"92*")*&R"'6":*<)'-<*"Z#?O!#?=<"&69*:"<98=G(G<"'-<*9"6':"N!+"?DO
BDB
"
LG92':"M*:<8'-"
"
>D"
!?D=<"6':"N>+"A?O!A?=<"6':"VU[4"U2*":*<G(9<"787"-'9"<9:'-.(Q"7*)*-7"'-"92*"98=*O38-7'3<"92&9"
BDC
"
3*"F2'<*"Z,8.G:*"5?f[4"
B!D
"
/*"):*78F9*7"92*"<9:*-.92"'6"92*" -'8<*" F'::*(&98'-" H*93**-" )&8:<" '6" :*F':78-."<89*<"
B!!
"
H&<*7"'-"92*";,"78<9&-F*"&-7"92*"<8.-&("F'::*(&98'-4"/*"G<*7"92*"$D"9:8&(<"3892"92*"!DD"9*<9"
B!#
"
8=&.*<" &-7" <)(89" 92*=" 8-" '77" &-7" *M*-" 9:8&(<" 9'" F'=)G9*" 92*" -'8<*" F'::*(&98'-" &-7" <8.-&("
B!$
"
F'::*(&98'-"8-"8-7*)*-7*-9" <*9<"'6"9:8&(<4",':"*&F2" )&8:" '6":*F':78-."<89*<+"3*"F'=)G9*7" 92*"
B!>
"
)''(*7"-'8<*"F'::*(&98'-"&F:'<<"8=&.*<+"68:<9"<GH9:&F98-."92*"&M*:&.*":*<)'-<*" 9'" *&F2" 9*<9"
B!?
"
8=&.*"Z'77"9:8&(<[+"&-7"92*-"F'=)G98-."92*"%*&:<'-n<"F'::*(&98'-"&F:'<<"92*":*<87G&(<"Z,8.G:*"
B!@
"
@f+Y"&-7",8.G:*"Ad[4" ,':" 92*" F'=)G9&98'-"'6"92*" -'8<*" F'::*(&98'-" 7G:8-." 92*"<)'-9&-*'G<"
B!A
"
&F98M89Q"3*"G<*7"&"98=*"38-7'3"'6"!DD"=<"H*6':*"92*"68:<9"8=&.*"'6"92*"<*PG*-F*"3892"6'G:"
B!B
"
8=&.*<4" T*-F*+" 3*" 787" -'9" 8-F(G7*" &F98M89Q" 8-" 92*" 8-9*:M&(" H*93**-" 8=&.*<" '6" 92*" <&=*"
B!C
"
<*PG*-F*4" U2*" <8.-&(" F'::*(&98'-" 3&<" 92*" %*&:<'-n<" F'::*(&98'-" H*93**-" 92*" &M*:&.*"
B#D
"
:*<)'-<*<" Z*M*-" :*)*9898'-<[" *(8F89*7" HQ" 92*" !DD" 9*<9" 8=&.*<4" W'8<*OF'::*(&98'-<" 3*:*" M*:Q"
B#!
"
<9&H(*"&F:'<<"7&Q<"Z=*&-":"p"D4A@"8-"='-R*Q"We"=*&-":"p"D4B#"8-"='-R*Q",["<'"3*")''(*7"7&9&"
B##
"
&F:'<<"7&Q<4"
B#$
"
/*")*:6':=*7"=G(98)(*"(8-*&:":*.:*<<8'-<"9'"9*<9"92*"8-6(G*-F*"'6";,"78<9&-F*"&-7"92*"
B#>
"
<8.-&("F'::*(&98'-"'-" 92*"-'8<*"F'::*(&98'-"&-7"G<*7" M&:8&-F*")&:9898'-8-."9'"8<'(&9*"G-8PG*+"
B#?
"
':92'.'-&("F'=)'-*-9<"'6"92*"*S)(&8-*7"M&:8&-F*+"&FF'G-98-."6':"=G(98F'((8-*&:89Q"8-"=G(98)(*"
B#@
"
:*.:*<<8'-25,37,404"U2*"=*92'7"8<"H&<*7"'-"92*"7866*:*-F*"8-"*S)(&8-*7"M&:8&-F*"32*-"8-F(G78-."
B#A
"
':"*SF(G78-."*&F2"):*78F9':"6:'="92*":*.:*<<8'-4",':"8-<9&-F*+"92*"G-8PG*"*S)(&8-*7"M&:8&-F*"
B#B
"
HQ"92*";,"78<9&-F*+"
𝑅<=>?73!*#;)
1
+"8<"F'=)G9*7"&<"6'(('3<I"
B#C
"
"
B$D
"
𝑅<=>?73!*#;)
1*=*𝑅4@55
1*𝑅+7A#*5B;988)5*!79#
1
"
B$!
"
"
B$#
"
LG92':"M*:<8'-"
"
>!"
/2*:*"
𝑅4@55
1
"8<"92*"M&:8&-F*"*S)(&8-*7"HQ"&"(8-*&:":*.:*<<8'-"8-F(G78-."&(("92*"):*78F9':<+"&-7"
B$$
"
𝑅+7A#*5B;988)5*!79#
1*
8<" 92*" M&:8&-F*" *S)(&8-*7" HQ" &" (8-*&:" :*.:*<<8'-" 92&9" 8-F(G7*<" 92*" <8.-&("
B$>
"
F'::*(&a'-"HG9"-'9";,O78<9&-F*+"&<"):*78F9':<4"/*"&(<'"9:8*7"('.Z;,O78<9&-F*[+"9'"&FF'G-9"6':"&"
B$?
"
-'-O(8-*&:":*(&a'-<28)"H*93**-"-'8<*"F'::*(&a'-"&-7";,O78<9&-F*+"HG9"928<"):*78F9':"787"-'9"
B$@
"
:*<G(9"8-"&-"8=):'M*=*-9"F'=)&:*7"9'";,O78<9&-F*4"
B$A
"
"
B$B
"
Spike sorWng
B$C
"
L(92'G.2"3*"7'"-'9"):'M87*"<':9*7"<)8R*<+"3*"8((G<9:&9*"<)8R*O<':98-."8-"92*"68:<9"H('FR"
B>D
"
'6"7&9&"8-"='-R*Q"W+"3892"&"(*-.92"'6"h#D"=8-<"7G:8-."92*"):*<*-9&98'-"'6"!$AB"<98=G(84"/*"
B>!
"
G<*7"/&M*F(G<$20"6':"<)8R*O<':98-."&-7" F(G<9*:" 87*-9868F&98'-" G<8-."92*"7*6&G(9" )&:&=*9*:<4"
B>#
"
/*"F*-<':*7"98=*O)'8-9<"8-"92*"M8F8-89Q"'6"92*":*3&:7"7*(8M*:Q+"H*F&G<*"'6"&:986&F9<"F&G<*7"HQ"
B>$
"
92*":*3&:7"7*(8M*:Q"<Q<9*=+"&-7"3*"&(<'"*SF(G7*7"F(G<9*:<"3892"6*3*:"92&-"!?D"<)8R*<4"
B>>
"
"
B>?
"
|G&-agF&a'-"&-7"<9&a<aF&("&-&(Q<8<""
B>@
"
B>A
"
Variance partitioning"
B>B
"
,':" *&F2" a=*O38-7'3+" 3*" 9*<9*7" 92*" <8.-8gF&-F*" '6" 92*" 78i*:*-F*" 8-" G-8PG*" M&:8&-F*"
B>C
"
*S)(&8-*7" H*93**-" 92*" H*<9" &-7" <*F'-7OH*<9"):*78F9':" G<8-." &" H''9<9:&)" 9*<9" Z!+DDD"
B?D
"
89*:&a'-<[4"
B?!
"
"
B?#
"
" "
B?$
"
LG92':"M*:<8'-"
"
>#"
References
B?>
"
!4" /G+"04f4+"Y&M87+"54"N+"&-7"X&((&-9+"r4K4"Z#DD@[4"f'=)(*9*"6G-Fa'-&("F2&:&F9*:8]&a'-"'6"
B??
"
<*-<':Q"-*G:'-<"HQ"<Q<9*="87*-agF&a'-4"L--G";*M"W*G:'<F8"29+">AAz?D?4"
B?@
"
21)<Icc7'84':.c!D4!!>@c&--G:*M4-*G:'4#C4D?!@D?4!!$D#>4"
B?A
"
#4"j&Q+"j4W4"Z#D!![4"`-7*:<9&-78-."M8<G&(":*):*<*-9&a'-"HQ"7*M*(')8-.":*F*)aM*Og*(7"
B?B
"
='7*(<4"N8<G&(")')G(&a'-"F'7*<I"U'3&:7<"&"F'=='-"=G(aM&:8&9*"6:&=*3':R"6':"F*(("
B?C
"
:*F':78-."&-7"6G-Fa'-&("8=&.8-.+"!$$z!@#4"
B@D
"
$4" ,*(<*-+"X4+"&-7"Y&-+"q4"Z#DD?[4"L"-&9G:&("&)):'&F2"9'"<9G7Q8-."M8<8'-4"W&9"W*G:'<F8"8+"
B@!
"
!@>$z!@>@4"21)<Icc7'84':.c!D4!D$Bc--!@DB4"
B@#
"
>4" f&7*-&+"54L4+"Y*-g*(7+"X4T4+"/&(R*:+"d4q4+"X&9Q<+"K4L4+"U'(8&<+"L454+"b*92.*+"04+"&-7"
B@$
"
dFR*:+"L454"Z#D!C[4"Y**)"F'-M'(Ga'-&("='7*(<"8=):'M*"):*78Fa'-<"'6"=&F&PG*"N!"
B@>
"
:*<)'-<*<"9'"-&9G:&("8=&.*<4"%K^5"f'=)G9"b8'("15+"*!DD@BCA4"
B@?
"
21)<Icc7'84':.c!D4!$A!c_'G:-&(4)FH84!DD@BCA4"
B@@
"
?4" /&(R*:+"d4q4+"58-]+",4T4+"f'H'<+"d4+"0G2&==&7+"U4+",:'G7&:&R8<+"d4+",&2*Q+"%4X4+"dFR*:+"
B@A
"
L454+";*8=*:+"r4+"%89R'3+"u4+"&-7"U'(8&<+"L454"Z#D!C[4"V-F*)a'-"('')<"78<F'M*:"32&9"
B@B
"
*SF89*<"-*G:'-<"='<9"G<8-."7**)"):*78FaM*"='7*(<4"W&9"W*G:'<F84"
B@C
"
21)<Icc7'84':.c!D4!D$Bc<>!?C$OD!COD?!AOS4"
BAD
"
@4" XGF(G+"`4+"&-7"M&-"X*:M*-+"04L4"Z#D!?[4"Y**)"W*G:&("W*93':R<";*M*&("&"X:&78*-9"8-"
BA!
"
92*"f'=)(*S89Q"'6"W*G:&(";*):*<*-9&a'-<"&F:'<<"92*"N*-9:&("59:*&=4"r"W*G:'<F8"35+"
BA#
"
!DDD?z!DD!>4"21)<Icc7'84':.c!D4!?#$crWd`;^5fV4?D#$O!>4#D!?4"
BA$
"
A4" b&<28M&-+"%4+"j&:+"j4+"&-7"Y8f&:('+"r4r4"Z#D!C[4"W*G:&(")')G(&a'-"F'-9:'("M8&"7**)"8=&.*"
BA>
"
<Q-92*<8<4"5F8*-F*"Z!CAC[4"21)<Icc7'84':.c!D4!!#@c<F8*-F*4&&MC>$@4"
BA?
"
B4"L((*-+"d4r4+"59OqM*<+"X4+"/G+"q4+"b:**7('M*+"r4K4+"%:8-F*+"r454+"Y'37(*+"K4U4+"W&G+"04+"f&:'-+"
BA@
"
b4+"%*<a((8+",4+"f2&:*<9+"V4+"*9"&(4"Z#D##[4"L"=&<<8M*"AU"60;V"7&9&<*9"9'"H:87.*"F'.-8aM*"
BAA
"
-*G:'<F8*-F*"&-7"&:agF8&("8-9*((8.*-F*4"W&9"W*G:'<F8"25+"!!@z!#@4"
BAB
"
21)<Icc7'84':.c!D4!D$Bc<>!?C$OD#!ODDC@#OS4"
BAC
"
C4" X8i':7+"L4U4+"Y38M*78+"j4+";'8.+"X4+"&-7"f8F2Q+";404"Z#D##[4"L"(&:.*"&-7":8F2"ddX"7&9&<*9"
BBD
"
6':"='7*(8-."2G=&-"M8<G&("'H_*F9":*F'.-8a'-4"W*G:'8=&.*"264+"!!CA?>4"
BB!
"
21)<Icc7'84':.c!D4!D!@c_4-*G:'8=&.*4#D##4!!CA?>4"
BB#
"
!D4" X:''9<3&.*:<+"U4+"}2'G+"V4+";'H8-<'-+"L4j4+"T*H&:9+"04W4+"&-7"f&:(<'-+"U4L4"Z#D##[4"
BB$
"
TG=&-"ddX":*F':78-.<"6':"!+B?>"F'-F*)9<"):*<*-9*7"8-":&)87"<*:8&("M8<G&("
BB>
"
):*<*-9&a'-"<9:*&=<4"5F8"Y&9&"9+"!zA4"21)<Icc7'84':.c!D4!D$Bc<>!?CAOD#!OD!!D#OA4"
BB?
"
!!4" T*H&:9+"04W4+"f'-a*:+"^4+"U*8F2=&--+"K4+";'FR9*:+"L4T4+"}2*-.+"f4q4+"j877*:+"L4+"
BB@
"
f'::8M*&G+"L4+"N&]8:8O%&<2R&=+"04+"&-7"b&R*:+"f4V4"Z#D#$[4"UTVWX5O7&9&+"&"=G(a='7&("
BBA
"
F'((*Fa'-"'6"(&:.*O<F&(*"7&9&<*9<"6':"8-M*<a.&a-."'H_*F9":*):*<*-9&a'-<"8-"2G=&-"
BBB
"
H:&8-"&-7"H*2&M8':4"d(86*"124"21)<Icc7'84':.c!D4A??>c*K86*4B#?BD4"
BBC
"
!#4" T*H&:9+"04W4+"Y8FR9*:+"L4T4+"j877*:+"L4+"j3'R+"/4q4+"f'::8M*&G+"L4+"N&-"/8FR(8-+"f4+"&-7"
BCD
"
b&R*:+"f4V4"Z#D!C[4"UTVWX5I"L"7&9&H&<*"'6"!+B?>"'H_*F9"F'-F*)9<"&-7"=':*"92&-"#@+DDD"
BC!
"
-&9G:&(8<aF"'H_*F9"8=&.*<4"%K'5"^-*"14+"*D##$AC#4"
BC#
"
21)<Icc7'84':.c!D4!$A!c_'G:-&(4)'-*4D##$AC#4"
BC$
"
!$4" f2*-+"u4+"/&-.+",4+",*:-&-7*]+"d4+"&-7";'*(6<*=&+"%4;4"Z#D#D[4"52&)*")*:F*)a'-"M8&"&"
BC>
"
28.2OF2&--*(OF'G-9"-*G:'):'<92*<8<"8-"='-R*Q"M8<G&("F':9*S4"5F8*-F*"Z!CAC["3704"
BC?
"
21)<Icc7'84':.c!D4!!#@c<F8*-F*4&H7A>$?4"
BC@
"
!>4" f2*-+"u4+"0':&(*<OX:*.':8'+"L4+"5):*-.*:+"r4+"j(*8-_'2&--+"L4+"5:872&:+"54+"M&-"L(H&7&+"
BCA
"
54r4+"X:~-+"54+"&-7";'*(6<*=&+"%4;4"Z#D##[4"!D#>OF2&--*("*(*F9:')2Q<8'('.8F&("
BCB
"
LG92':"M*:<8'-"
"
>$"
:*F':78-.<"8-"=&F&PG*"N!"&-7"N>"7G:8-.":*<a-."<9&9*4"5F8"Y&9&"9+"!z!@4"
BCC
"
21)<Icc7'84':.c!D4!D$Bc<>!?CAOD##OD!!BDO!4"
CDD
"
!?4" 5*(6+"04/4+"j''8_=&-<+";4W4+"5G)*:+"T4+"K&==*+"N4L4+"&-7";'*(6<*=&+"%4;4"Z#D!#[4"
CD!
"
Y8i*:*-9".(G9&=&9*":*F*)9':<"F'-M*Q"6**76':3&:7"&-7":*FG::*-9"):'F*<<8-."8-"
CD#
"
=&F&PG*"N!4"%:'F"W&9("LF&7"5F8"`"5"L"109+"!!D$!z!!D$@4"
CD$
"
21)<Icc7'84':.c!D4!DA$c)-&<4!!!C?#A!DC4"
CD>
"
!@4" 5G)*:+"T4+"&-7";'*(6<*=&+"%4;4"Z#DD?[4"f2:'-8F"=G(aG-89":*F':78-.<"8-"H*2&M8-."
CD?
"
&-8=&(<I"&7M&-9&.*<"&-7"(8=89&a'-<4"%:'."b:&8-";*<"147+"#@$z#B#4"
CD@
"
21)<Icc7'84':.c!D4!D!@c5DDACO@!#$ZD>[>AD#DO>4"
CDA
"
!A4" %&(=*:+"f4+"f2*-.+"54q4+"&-7"5*87*=&--+"d4"Z#DDA[4"K8-R8-."-*G:'-&("&-7"H*2&M8':&("
CDB
"
)*:6':=&-F*"8-"&":*&Fa'-Oa=*"M8<G&("7*9*Fa'-"9&<R4"r"W*G:'<F8"27+"B!##zB!$A4"
CDC
"
21)<Icc7'84':.c!D4!?#$crWd`;^5fV4!C>DODA4#DDA4"
C!D
"
!B4" f'2*-+"04;4+"&-7"0&G-<*((+"r4T4"Z#DDC[4"L1*-a'-"8=):'M*<")*:6':=&-F*"):8=&:8(Q"HQ"
C!!
"
:*7GF8-."8-9*:-*G:'-&("F'::*(&a'-<4"W&9"W*G:'<F8"12+"!?C>z!@DD4"
C!#
"
21)<Icc7'84':.c!D4!D$Bc--4#>$C4"
C!$
"
!C4" U:&G9=&--+"d404+"59&M8<RQ+"54Y4+"K&28:8+"54+"L=*<+"j4f4+"j&G6=&-+"04U4+"^n52*&+"Y4r4+"
C!>
"
NQ&<+"54+"5G-+"u4+";QG+"54V4+"X&-.G(8+"54+"*9"&(4"Z#D!C[4"LFFG:&9*"d<a=&a'-"'6"W*G:&("
C!?
"
%')G(&a'-"YQ-&=8F<"3892'G9"5)8R*"5':a-.4"W*G:'-"103+"#C#O$DB"*>4"
C!@
"
21)<Icc7'84':.c!D4!D!@c_4-*G:'-4#D!C4D?4DD$4"
C!A
"
#D4" f2&G:*+",4r4+";*Q+"T4X4+"&-7"|G8&-"|G8:'.&+";4"Z#D!B[4"L"-'M*("&-7"6G((Q"&G9'=&aF"
C!B
"
<)8R*O<':a-."8=)(*=*-9&a'-"3892"M&:8&H(*"-G=H*:"'6"6*&9G:*<4"r"W*G:')2Q<8'("120+"
C!C
"
!B?Cz!BA!4"21)<Icc7'84':.c!D4!!?#c_-4DD$$C4#D!B4"
C#D
"
#!4" T*H&:9+"04W4+"}2*-.+"f4q4+"%*:*8:&+",4+"&-7"b&R*:+"f4V4"Z#D#D[4";*M*&(8-."92*"
C#!
"
=G(a78=*-<8'-&("=*-9&(":*):*<*-9&a'-<"'6"-&9G:&("'H_*F9<"G-7*:(Q8-."2G=&-"
C##
"
<8=8(&:89Q"_G7.*=*-9<4"W&9"TG="b*2&M"4+"!!A$z!!B?4"
C#$
"
21)<Icc7'84':.c!D4!D$Bc<>!?@#OD#DODDC?!O$4"
C#>
"
##4" b&'+"%4+"52*+"K4+"0FX8((+"04+"&-7"U<&'+"Y4q4"Z#D#D[4"L"=&)"'6"'H_*F9"<)&F*"8-"):8=&9*"
C#?
"
8-6*:'9*=)':&("F':9*S4"W&9G:*"583+"!D$z!DB4"21)<Icc7'84':.c!D4!D$Bc<>!?B@OD#DO
C#@
"
#$?DO?4"
C#A
"
#$4" ^)"7*"b**FR+"T4%4+"Y8F&:('+"r4r4+"X'*-<*+"r4b4+"X:8((O5)*F9':+"j4+"%&)&-&<9&<<8'G+"L4+"
C#B
"
U&-86G_8+"04+"&-7"U<&'+"Y4q4"Z#DDB[4",8-*O<F&(*"<)&a&("':.&-8]&a'-"'6"6&F*"&-7"'H_*F9"
C#C
"
<*(*FaM89Q"8-"92*"9*=)':&("('H*I"7'"6G-Fa'-&("=&.-*aF":*<'-&-F*"8=&.8-.+"')aF&("
C$D
"
8=&.8-.+"&-7"*(*F9:')2Q<8'('.Q"&.:**•"r"W*G:'<F8"28+"!!AC@z!!BD!4"
C$!
"
21)<Icc7'84':.c!D4!?#$crWd`;^5fV4$ACCODB4#DDB4"
C$#
"
#>4" j&Q&*:9+"X4+"b8*7*:=&-+"V4+"&-7"N'.*(<+";4"Z#DD?[4";*):*<*-9&a'-"'6":*.G(&:"&-7"
C$$
"
8::*.G(&:"<2&)*<"8-"=&F&PG*"8-6*:'9*=)':&("F':9*S4"f*:*H"f':9*S"15+"!$DBz!$#!4"
C$>
"
21)<Icc7'84':.c!D4!DC$cF*:F':cH28D!>4"
C$?
"
#?4" %&)&(*+"%4+"K*'+"L4+"T&-7_&:&<+"X4+"f*FF2*€+"K4+"%8*9:8-8+"%4+"&-7";8FF8&:78+"d4"Z#D#D[4"
C$@
"
52&)*"F'78-."8-"'FF8)89'O9*=)':&("F':9*S":*(8*<"'-"'H_*F9"<8(2'G*1*+"FG:M&9G:*"&-7"
C$A
"
=*78&(O&S8<4"r"W*G:')2Q<8'(4"21)<Icc7'84':.c!D4!!?#c_-4DD#!#4#D#D4"
C$B
"
#@4" j2'<(&+"04+";&9&-"0G:9Q+"W4L4+"&-7"j&-38<2*:+"W4"Z#D##[4"L"28.2(Q"<*(*FaM*":*<)'-<*"
C$C
"
9'"6''7"8-"2G=&-"M8<G&("F':9*S":*M*&(*7"HQ"2Q)'92*<8<O6:**"M'S*("7*F'=)'<8a'-4"
C>D
"
fG::*-9"b8'('.Q"32+">!?CO>!A!4*C4"21)<Icc7'84':.c!D4!D!@c_4FGH4#D##4DB4DDC4"
C>!
"
#A4" 5]*.*7Q+"f4+"K8G+"/4+"r8&+"q4+"5*:=&-*9+"%4+";**7+"54+"L-.G*('M+"Y4+"d:2&-+"Y4+"N&-2'GFR*+"
C>#
"
N4+"&-7";&H8-'M8F2+"L4"Z#D!?[4"X'8-."Y**)*:"3892"f'-M'(Ga'-<4"V-"fN%;4"
C>$
"
LG92':"M*:<8'-"
"
>>"
#B4" 5=892+"04L4+"&-7"j'2-+"L4"Z#DDB[4"5)&a&("&-7"9*=)':&("<F&(*<"'6"-*G:'-&("F'::*(&a'-"
C>>
"
8-"):8=&:Q"M8<G&("F':9*S4"r'G:-&("'6"W*G:'<F8*-F*"28+"!#?C!z!#@D$4"
C>?
"
21)<Icc7'84':.c!D4!?#$crWd`;^5fV4#C#CODB4#DDB4"
C>@
"
#C4" }'2&:Q+"d4+"52&7(*-+"04W4+"&-7"W*3<'=*+"/4U4"Z!CC>[4"f'::*(&9*7"-*G:'-&("78<F2&:.*"
C>A
"
:&9*"&-7"89<"8=)(8F&a'-<"6':")<QF2')2Q<8F&(")*:6':=&-F*4"W&9G:*"370+"!>Dz!>$4"
C>B
"
21)<Icc7'84':.c!D4!D$Bc$AD!>D&D4"
C>C
"
$D4" f'2*-+"04;4+"&-7"j'2-+"L4"Z#D!![4"0*&<G:8-."&-7"8-9*:):*a-."-*G:'-&("F'::*(&a'-<4"
C?D
"
%:*):8-9"&9"W&9G:*"%GH(8<28-."X:'G)+"21)<Icc7'84':.c!D4!D$Bc--4#B>#"
C?!
"
21)<Icc7'84':.c!D4!D$Bc--4#B>#4"
C?#
"
$!4" ;'*(6<*=&+"%4;4+"d-.*(+"L4j4+"j•-8.+"%4+"&-7"58-.*:+"/4"Z!CCA[4"N8<G'='9':"8-9*.:&a'-"8<"
C?$
"
&<<'F8&9*7"3892"]*:'"a=*O(&."<Q-F2:'-8]&a'-"&='-."F':aF&("&:*&<4"W&9G:*"385+"!?Az
C?>
"
!@!4"21)<Icc7'84':.c!D4!D$Bc$B?!?A&D4"
C??
"
$#4" b&8:+"/4+"}'2&:Q+"d4+"&-7"W*3<'=*+"/4U4"Z#DD![4"f'::*(&9*7"g:8-."8-"=&F&PG*"M8<G&("
C?@
"
&:*&"0UI"U8=*"<F&(*<"&-7":*(&a'-<28)"9'"H*2&M8':4"r'G:-&("'6"W*G:'<F8*-F*"21+"!@A@z
C?A
"
!@CA4"21)<Icc7'84':.c!D4!?#$c_-*G:'<F84#!OD?OD!@A@4#DD!4"
C?B
"
$$4" 5F23&:9]+"^4+"&-7"58='-F*((8+"d4%4"Z#DD![4"W&9G:&("<8.-&("<9&a<aF<"&-7"<*-<':Q".&8-"
C?C
"
F'-9:'(4"W&9"W*G:'<F8"4+"B!CzB#?4"21)<Icc7'84':.c!D4!D$BcCD?#@4"
C@D
"
$>4" 58='-F*((8+"d4%4+"&-7"^(<2&G<*-+"b4L4"Z#DD![4"W&9G:&("8=&.*"<9&a<aF<"&-7"-*G:&("
C@!
"
:*):*<*-9&a'-4"L--G";*M"W*G:'<F8"24+"!!C$z!#!@4"
C@#
"
21)<Icc7'84':.c!D4!!>@c&--G:*M4-*G:'4#>4!4!!C$4"
C@$
"
$?4" ;8R2Q*+";4"N4+"&-7"5G:+"04"Z#D!?[4"5)&a&("F'::*(&a'-<"8-"-&9G:&("<F*-*<"='7G(&9*"
C@>
"
:*<)'-<*":*(8&H8(89Q"8-"='G<*"M8<G&("F':9*S4"r'G:-&("'6"W*G:'<F8*-F*"35+"!>@@!z!>@BD4"
C@?
"
21)<Icc7'84':.c!D4!?#$crWd`;^5fV4!@@DO!?4#D!?4"
C@@
"
$@4" b*:R*<+"%4+"^:H‚-+"X4+"K*-.Q*(+"04+"&-7",8<*:+"r4"Z#D!![4"5)'-9&-*'G<"F':aF&("&FaM89Q"
C@A
"
:*M*&(<"2&((=&:R<"'6"&-"')a=&("8-9*:-&("='7*("'6"92*"*-M8:'-=*-94"5F8*-F*"Z!CAC["
C@B
"
331+"B$zBA4"21)<Icc7'84':.c!D4!!#@c<F8*-F*4!!C?BAD4"
C@C
"
$A4" K*<F:'&:9+"04Y4+"59&-<HG:Q+"Y4d4+"&-7"X&((&-9+"r4K4"Z#D!?[4",'G:8*:")'3*:+"<GH_*FaM*"
CAD
"
78<9&-F*+"&-7"'H_*F9"F&9*.':8*<"&(("):'M87*")(&G<8H(*"='7*(<"'6"b^KY":*<)'-<*<"8-"
CA!
"
<F*-*O<*(*FaM*"M8<G&("&:*&<4",:'-9"f'=)G9"W*G:'<F8"9+"!$?4"
CA#
"
21)<Icc7'84':.c!D4$$BCc6-F'=4#D!?4DD!$?4"
CA$
"
$B4" ;&QO0GR2*:_**+"r4+"W8='-+"j4+"0GR2*:_**+"54+"0'::8<+"Y4/4+"5('9'3+";4+"&-7"T&=*:+"04"
CA>
"
Z#D!>[4"`<8-."F'=='-&(89Q"&-&(Q<8<"8-"=G(a)(*":*.:*<<8'-<I"&"9''("9'"7*F'=)'<*"
CA?
"
:*.:*<<8'-"*i*F9<"8-"92*"6&F*"'6"=G(aF'((8-*&:89Q4"0*92'7<"dF'("dM'("5+"$#Dz$#B4"
CA@
"
$C4" %&)&(*+"%4+"}G87*:H&&-+"/4+"U**G3*-+";4;404+"X8(2G8<+"L4+"5*(6+"04/4+";'*(6<*=&+"%4;4+"
CAA
"
&-7"YG='G(8-+"54^4"Z#D#![4"U2*"8-{G*-F*"'6"'H_*F92''7"'-"92*":*):*<*-9&a'-"'6"
CAB
"
-&9G:&("8=&.*<"8-"92*"M8<G&("F':9*S4"H8';S8M+"#D#!4DC4#!4>@!#DC4"
CAC
"
21)<Icc7'84':.c!D4!!D!c#D#!4DC4#!4>@!#DC4"
CBD
"
>D4" %&)&(*+"%4+"b*1&+"04+"T&-7_&:&<+"X4+"0&(6&€+"X4+"f*FF2*€+"K4+";&=)8-8-8+"L4+"%8*9:8-8+"%4+"
CB!
"
;8FF8&:78+"d4+"UG:*((&+"K4+"&-7"K*'+"L4"Z#D!C[4"f'=='-"<)&a'9*=)':&("):'F*<<8-."'6"
CB#
"
M8<G&("6*&9G:*<"<2&)*<"'H_*F9":*):*<*-9&a'-4"5F8";*)"9+"!zB4"
CB$
"
21)<Icc7'84':.c!D4!D$Bc<>!?CBOD!CO>$C?@O$4"
CB>
"
>!4" 5*=*7'+"r4Y4+"r&<)*:+"L4V4+"}&-7M&R8(8+"L4+"j:8<2-&+"L4+"L<F2-*:+"L4+"0&F2*-<+"f4j4+"j'2-+"
CB?
"
L4+"&-7"qG+"b404"Z#D##[4",**76':3&:7"&-7"6**7H&FR"8-9*:&Fa'-<"H*93**-"M8<G&("
CB@
"
F':aF&("&:*&<"G<*"78i*:*-9")')G(&a'-"&FaM89Q")&1*:-<4"W&9"f'==G-"13+"!z!>4"
CBA
"
21)<Icc7'84':.c!D4!D$Bc<>!>@AOD##O#B??#O34"
CBB
"
LG92':"M*:<8'-"
"
>?"
>#4" 5=892+"r4d4U4+"&-7"%&:R*:+"L4r4"Z#D#![4"f'::*(&9*7"<9:GF9G:*"'6"-*G:'-&("g:8-."8-"
CBC
"
=&F&PG*"M8<G&("F':9*S"(8=89<"8-6':=&a'-"6':"H8-'FG(&:"7*)92"78<F:8=8-&a'-4"r"
CCD
"
W*G:')2Q<8'("126+"#A?z$D$4"21)<Icc7'84':.c!D4!!?#c_-4DD@@A4#D#D4"
CC!
"
>$4" /&-.+"U4+"K**+"U454+"q&'+"T4+"T'-.+"r4+"K8+"q4+"r8&-.+"T4+"L-7'(8-&+"V404+"&-7"U&-.+"54"Z#D#>[4"
CC#
"
K&:.*O<F&(*"F&(F8G="8=&.8-.":*M*&(<"&"<Q<9*=&aF"N>"=&)"6':"*-F'78-."-&9G:&("<F*-*<4"
CC$
"
W&9"f'==G-"15+"@>D!4"21)<Icc7'84':.c!D4!D$Bc<>!>@AOD#>O?DB#!O]4"
CC>
"
>>4" u8&'+"/4+"52&:=&+"54+"j:*8=&-+"X4+"&-7"K8M8-.<9'-*+"0454"Z#D#>[4",*&9G:*O<*(*FaM*"
CC?
"
:*<)'-<*<"8-"=&F&PG*"M8<G&("F':9*S"6'(('3"*Q*"='M*=*-9<"7G:8-."-&9G:&("M8<8'-4"W&9"
CC@
"
W*G:'<F8"27+"!!?Az!!@@4"21)<Icc7'84':.c!D4!D$Bc<>!?C$OD#>OD!@$!O?4"
CCA
"
>?4" 0G1*-92&(*:+"K4+"&-7"T*H&:9+"04W4"Z#D#![4"UTVWX5M8<8'-I"L"%Q92'-"U''(H'S"6':"
CCB
"
59:*&=(8-8-."92*"dS9:&Fa'-"'6"LFaM&a'-<",:'="Y**)"W*G:&("W*93':R<4",:'-9"
CCC
"
W*G:'8-6':="15+"@ACB$B4"21)<Icc7'84':.c!D4$$BCc6-8-64#D#!4@ACB$B4"
!DDD
"
>@4" 59'8-<R8+"K404+"%*:RG2-+"r4+"&-7"T*H&:9+"04W4"Z#D#>[4"UTVWX5)(G<I"W*3"W':=<"&-7"
!DD!
"
0*9&7&9&"6':"92*"UTVWX5"Y&9&H&<*"'6"!+B?>"^H_*F9"f'-F*)9<"&-7"#@+!DA"W&9G:&("
!DD#
"
^H_*F9"V=&.*<4"21)<Icc7'84':.c!D4$!#$>c^5,4V^cdu`C,4"
!DD$
"
>A4" T&-<*-+"T4+"&-7"T*H&:9+"04W4"Z#D##[4"5*=&-aF"6*&9G:*<"'6"'H_*F9"F'-F*)9<".*-*:&9*7"
!DD>
"
3892"X%UO$4"%:'F**78-.<"'6"92*">>92"L--G&("0**a-."'6"92*"f'.-8aM*"5F8*-F*"5'F8*9QI"
!DD?
"
f'.-8aM*"Y8M*:<89Q+"f'.5F8"#D##+"AACzAB@4"
!DD@
"
>B4" j:&=*:+"04L4+"T*H&:9+"04W4+"b&R*:+"f4V4+"&-7"b&8-H:87.*+"/4L4"Z#D#$[4"U2*"6*&9G:*<"
!DDA
"
G-7*:(Q8-."92*"=*=':&H8(89Q"'6"'H_*F9<4"5F8"L7M"94"
!DDB
"
21)<Icc7'84':.c!D4!!#@c5fVLYN4LYY#CB!4"
!DDC
"
>C4" ,*((*=&-+"Y4r4+"&-7"N&-"d<<*-+"Y4f4"Z!CC![4"Y8<9:8HG9*7"28*:&:F28F&("):'F*<<8-."8-"92*"
!D!D
"
):8=&9*"F*:*H:&("F':9*S4"f*:*H:&("f':9*S"1+"!z>A4"
!D!!
"
?D4" 0&:R'M+"W4U4+"d:F<*QO;&M&<]+"04+"N&-"d<<*-+"Y4f4+"j-'H(&GF2+"j4+"U':'F]R&8+"}4+"&-7"
!D!#
"
j*--*7Q+"T4"Z#D!$[4"f':aF&("28.2O7*-<89Q"F'G-9*:<9:*&="&:F289*F9G:*<4"%:*):8-9"&9"
!D!$
"
L=*:8F&-"L<<'F8&a'-"6':"92*"L7M&-F*=*-9"'6"5F8*-F*+"
!D!>
"
21)<Icc7'84':.c!D4!!#@c<F8*-F*4!#$B>D@"21)<Icc7'84':.c!D4!!#@c<F8*-F*4!#$B>D@4"
!D!?
"
?!4" W.GQ*-+"W4Y4+"KG9&<+"L4+"L=<&(*=+"^4+",*:-&-7'+"r4+"L2-+"L4q4d4+"T&R8=+";4+"N*:.&:&+"r4+"
!D!@
"
0F0&2'-+"r4+"Y8=87<F2<9*8-+"r4+"5&H&a-8+"b4K4+"*9"&(4"Z#D#>[4"f':aF&(":*&FaM&a'-<"
!D!A
"
):*78F9"6G9G:*"<*-<':Q":*<)'-<*<4"W&9G:*"625+"!!Dz!!B4"
!D!B
"
21)<Icc7'84':.c!D4!D$Bc<>!?B@OD#$OD@B!DO!4"
!D!C
"
?#4" U<'7QR<+"04+"j*-*9+"U4+"X:8-M&(7+"L4+"&-7"L:8*(8+"L4"Z!CCC[4"K8-R8-."<)'-9&-*'G<"&FaM89Q"
!D#D
"
'6"<8-.(*"F':aF&("-*G:'-<"&-7"92*"G-7*:(Q8-."6G-Fa'-&("&:F289*F9G:*4"5F8*-F*"Z!CAC["
!D#!
"
286+"!C>$z!C>@4"21)<Icc7'84':.c!D4!!#@c<F8*-F*4#B@4?>>@4!C>$4"
!D##
"
?$4" T8-9'-+"X4d4+"Y&Q&-+"%4+",:*Q+"b4r4+"&-7"W*&(+";404"Z!CC?[4"U2*"s3&R*O<(**)t"&(.':892="
!D#$
"
6':"G-<G)*:M8<*7"-*G:&("-*93':R<4"5F8*-F*"Z!CAC["268+"!!?Bz!!@!4"
!D#>
"
21)<Icc7'84':.c!D4!!#@c<F8*-F*4AA@!B$!4"
!D#?
"
?>4" f2*-+"u4+"%'<<*(+"r4j4+"/&F'-.-*+"f4+"M&-"T&=+"L4,4+"j(8-R+"%4f4+"&-7";'*(6<*=&+"%4;4"
!D#@
"
Z#D!A[4"$Y"):8-a-."&-7"='7*((8-."'6"FG<9'=8]*7"8=)(&-9<"&-7"<G:.8F&(".G87*<"6':"-'-O
!D#A
"
2G=&-"):8=&9*<4"r"W*G:'<F8"0*92'7<"286+"$Bz??4"
!D#B
"
21)<Icc7'84':.c!D4!D!@c_4_-*G=*924#D!A4D?4D!$4"
!D#C
"
??4" L]&78+";4+"b'2-+"54+"d(7:87.*+"04L4X4+"&-7"L6:&]+"L4"Z#D#$[4"5G:.8F&("%:'F*7G:*"6':"
!D$D
"
V=)(&-9&a'-"'6"^)9'OL::&Q"8-"W'-2G=&-"%:8=&9*<4"fG::"%:'9'F"34"
!D$!
"
21)<Icc7'84':.c!D4!DD#cF)]!4AD>4"
!D$#
"
?@4" %&)&(*+"%4+"/&-.+",4+"0':.&-+"L4U4+"f2*-+"u4+"X8(2G8<+"L4+"%*9:'+"K454+"0GFR(8+"K4+"
!D$$
"
;'*(6<*=&+"%4;4+"&-7"5*(6+"04/4"Z#D#$[4"U2*":*):*<*-9&a'-"'6"'FF(G7*7"8=&.*":*.8'-<"
!D$>
"
LG92':"M*:<8'-"
"
>@"
8-"&:*&"N!"'6"='-R*Q<"&-7"2G=&-<4"fG::*-9"b8'('.Q"33+"$B@?O$BA!4*$4"
!D$?
"
21)<Icc7'84':.c!D4!D!@c_4FGH4#D#$4DB4D!D4"
!D$@
"
?A4" U'.9+"f4"M&-"7*:+"j(8-R+"f4+"%&)&(*+"%4+"&-7"U**G3*-+";4"Z#D##[4"
!D$A
"
N8<8'-&-7f'.-8a'-cU:&FR*:I"%GH(8F";*(*&<*4"
!D$B
"
21)<Icc7'84':.c!D4?#B!c}dW^Y^4@>BCD!>4"
!D$C
"
?B4" ;'*(6<*=&+"%4;4+"j2&Q&9+"%454+"&-7"5)*R:*8_<*+"T4"Z#DD$[4"5GH9&<R"<*PG*-F8-."8-"92*"
!D>D
"
):8=&:Q"M8<G&("F':9*S4"%:'F"W&9("LF&7"5F8"`"5"L"100+"?>@Az?>A#4"
!D>!
"
21)<Icc7'84':.c!D4!DA$c)-&<4D>$!D?!!DD4"
!D>#
"
?C4" 5*8.-*1*+"j4+"r&=&--+"W4+"%&)&(*+"%4+"U*::&+"T4+"%':-*<'+";4%4^4+"j:&R*:+"K4"7*+"U'.9+"f4"
!D>$
"
M&-"7*:+"L&+"04"M&-"7*:+"W**:8-.+"%4+";G8=<F2'9*(+"d4+"*9"&(4"Z#D#$[4"dS)*:8*-F*"52&)*<"
!D>>
"
f2&-7*(8*:"f*((",G-Fa'-"&-7"59:GF9G:*"8-"92*"N8<G&("f':9*S4"d(86*"124"
!D>?
"
21)<Icc7'84':.c!D4A??>cdKV,d4C!!?$4#4"
!D>@
"
@D4" %&<]R*+"L4+"X:'<<+"54+"0&<<&+",4+"K*:*:+"L4+"b:&7HG:Q+"r4+"f2&-&-+"X4+"j8((**-+"U4+"K8-+"}4+"
!D>A
"
X8=*(<2*8-+"W4+"L-a.&+"K4+"*9"&(4"Z#D!C[4"%QU':F2I"L-"V=)*:&aM*"59Q(*+"T8.2O
!D>B
"
%*:6':=&-F*"Y**)"K*&:-8-."K8H:&:Q4"L:u8M4"
!D>C
"
@!4" j8-.=&+"Y4%4+"&-7"b&+"r4K4"Z#D!?[4"L7&=I"L"=*92'7"6':"<9'F2&<aF"')a=8]&a'-4"V-"$:7"
!D?D
"
V-9*:-&a'-&("f'-6*:*-F*"'-"K*&:-8-.";*):*<*-9&a'-<+"VfK;"#D!?"O"f'-6*:*-F*"U:&FR"
!D?!
"
%:'F**78-.<"ZV-9*:-&a'-&("f'-6*:*-F*"'-"K*&:-8-.";*):*<*-9&a'-<+"VfK;[4"
!D?#
"
@#4" ^(&2+"f4+"0':7M8-9<*M+"L4+"&-7"5F2GH*:9+"K4"Z#D!A[4",*&9G:*"M8<G&(8]&a'-4"Y8<a(("2+"*A4"
!D?$
"
"
!D?>
"
"
!D??
"
... Recently, there has been an increase in both collection and use of large-scale visual neural datasets, suggesting that the field of vision science is entering a new era of big open data (1)(2)(3)(4)(5)(6)(7). One example of this recent trend in the context of human fMRI is the Natural Scenes Dataset (NSD) which provides high-quality 7T fMRI responses to over 70,000 images of naturalistic scenes across 8 subjects (1). ...
... White noise (large block): Image subclass 2 (images [5][6][7][8]. Pixels are grouped into block sizes of 42 pixels × 42 pixels (0.49° × 0.49°). Luminance values for each block were randomly drawn from a uniform distribution between 0-1. ...
Preprint
Full-text available
Large-scale visual neural datasets such as the Natural Scenes Dataset (NSD) are boosting NeuroAI research by enabling computational models of the brain with performances beyond what was possible just a decade ago. However, these datasets lack out-of-distribution (OOD) components, which are crucial for the development of more robust models. Here, we address this limitation by releasing NSD-synthetic, a dataset consisting of 7T fMRI responses from the eight NSD subjects for 284 carefully controlled synthetic images. We show that NSD-synthetic's fMRI responses reliably encode stimulus-related information and are OOD with respect to NSD. Furthermore, OOD generalization tests on NSD-synthetic reveal differences between models of the brain that are not detected with NSD - specifically, self-supervised deep neural networks better explain neural responses than their task-supervised counterparts. These results showcase how NSD-synthetic enables OOD generalization tests that facilitate the development of more robust models of visual processing, and the formulation of more accurate theories of human vision.
... Using ANNs to understand tuning has proven a powerful tool to study the highly non-linear tuning properties in highlevel regions such as area V4 and the inferotemporal cortex (IT) 5,7,8 . However, this data-driven method also has some limitations. ...
Preprint
Full-text available
Finding the tuning of visual neurons has kept neuroscientists busy for decades. One approach to this problem has been to test specific hypotheses on the relevance of a visual property (e.g. orientation or color), build a set of "artificial" stimuli that vary along that property and then record neural responses to those stimuli. Here, we present a complementary, data-driven method to retrieve the tuning properties of visual neurons. Exploiting deep generative networks and electrophysiology in monkeys, we first used a method to reconstruct any stimulus from its evoked neuronal activity in the inferotemporal cortex (IT). Then, by arbitrarily perturbing the response of individual cortical sites in the model, we generated naturalistic and interpretable sequences of images that strongly influence neural activity of that site. This method enables the discovery of previously unknown tuning properties of high-level visual neurons that are easily interpretable, which we tested with carefully controlled stimuli. When we knew which images drove the neurons, we activated the cells with electrical microstimulation and observed a predicable shift of the monkey perception in the direction of the preferred image. By allowing the brain to tell us what it cares about, we are no longer limited by our experimental imagination.
Article
Full-text available
Biological visual systems have evolved to process natural scenes. A full understanding of visual cortical functions requires a comprehensive characterization of how neuronal populations in each visual area encode natural scenes. Here, we utilized widefield calcium imaging to record V4 cortical response to tens of thousands of natural images in male macaques. Using this large dataset, we developed a deep-learning digital twin of V4 that allowed us to map the natural image preferences of the neural population at 100-µm scale. This detailed map revealed a diverse set of functional domains in V4, each encoding distinct natural image features. We validated these model predictions using additional widefield imaging and single-cell resolution two-photon imaging. Feature attribution analysis revealed that these domains lie along a continuum from preferring spatially localized shape features to preferring spatially dispersed surface features. These results provide insights into the organizing principles that govern natural scene encoding in V4.
Article
Full-text available
In natural vision, primates actively move their eyes several times per second via saccades. It remains unclear whether, during this active looking, visual neurons exhibit classical retinotopic properties, anticipate gaze shifts or mirror the stable quality of perception, especially in complex natural scenes. Here, we let 13 monkeys freely view thousands of natural images across 4.6 million fixations, recorded 883 h of neuronal responses in six areas spanning primary visual to anterior inferior temporal cortex and analyzed spatial, temporal and featural selectivity in these responses. Face neurons tracked their receptive field contents, indicated by category-selective responses. Self-consistency analysis showed that general feature-selective responses also followed eye movements and remained gaze-dependent over seconds of viewing the same image. Computational models of feature-selective responses located retinotopic receptive fields during free viewing. We found limited evidence for feature-selective predictive remapping and no viewing-history integration. Thus, ventral visual neurons represent the world in a predominantly eye-centered reference frame during natural vision.
Article
Full-text available
Detailed characterization of interneuron types in primary visual cortex (V1) has greatly contributed to understanding visual perception, yet the role of chandelier cells (ChCs) in visual processing remains poorly characterized. Using viral tracing we found that V1 ChCs predominantly receive monosynaptic input from local layer 5 pyramidal cells and higher-order cortical regions. Two-photon calcium imaging and convolutional neural network modeling revealed that ChCs are visually responsive but weakly selective for stimulus content. In mice running in a virtual tunnel, ChCs respond strongly to events known to elicit arousal, including locomotion and visuomotor mismatch. Repeated exposure of the mice to the virtual tunnel was accompanied by reduced visual responses of ChCs and structural plasticity of ChC boutons and axon initial segment length. Finally, ChCs only weakly inhibited pyramidal cells. These findings suggest that ChCs provide an arousal-related signal to layer 2/3 pyramidal cells that may modulate their activity and/or gate plasticity of their axon initial segments during behaviorally relevant events.
Article
Full-text available
Many theories of offline memory consolidation posit that the pattern of neurons activated during a salient sensory experience will be faithfully reactivated, thereby stabilizing the pattern1,2. However, sensory-evoked patterns are not stable but, instead, drift across repeated experiences3–6. Here, to investigate the relationship between reactivations and the drift of sensory representations, we imaged the calcium activity of thousands of excitatory neurons in the mouse lateral visual cortex. During the minute after a visual stimulus, we observed transient, stimulus-specific reactivations, often coupled with hippocampal sharp-wave ripples. Stimulus-specific reactivations were abolished by local cortical silencing during the preceding stimulus. Reactivations early in a session systematically differed from the pattern evoked by the previous stimulus—they were more similar to future stimulus response patterns, thereby predicting both within-day and across-day representational drift. In particular, neurons that participated proportionally more or less in early stimulus reactivations than in stimulus response patterns gradually increased or decreased their future stimulus responses, respectively. Indeed, we could accurately predict future changes in stimulus responses and the separation of responses to distinct stimuli using only the rate and content of reactivations. Thus, reactivations may contribute to a gradual drift and separation in sensory cortical response patterns, thereby enhancing sensory discrimination⁷.
Article
Full-text available
What makes certain images more memorable than others? While much of memory research has focused on participant effects, recent studies using a stimulus-centric perspective have sparked debate on the determinants of memory, including the roles of semantic and visual features and whether the most prototypical or atypical items are best remembered. Prior studies have typically relied on constrained stimulus sets, limiting a generalized view of the features underlying what we remember. Here, we collected more than 1 million memory ratings for a naturalistic dataset of 26,107 object images designed to comprehensively sample concrete objects. We establish a model of object features that is predictive of image memorability and examined whether memorability could be accounted for by the typicality of the objects. We find that semantic features exert a stronger influence than perceptual features on what we remember and that the relationship between memorability and typicality is more complex than a simple positive or negative association alone.
Article
Full-text available
Understanding object representations requires a broad, comprehensive sampling of the objects in our visual world with dense measurements of brain activity and behavior. Here we present THINGS-data, a multimodal collection of large-scale neuroimaging and behavioral datasets in humans, comprising densely-sampled functional MRI and magnetoencephalographic recordings, as well as 4.70 million similarity judgments in response to thousands of photographic images for up to 1,854 object concepts. THINGS-data is unique in its breadth of richly-annotated objects, allowing for testing countless hypotheses at scale while assessing the reproducibility of previous findings. Beyond the unique insights promised by each individual dataset, the multimodality of THINGS-data allows combining datasets for a much broader view into object processing than previously possible. Our analyses demonstrate the high quality of the datasets and provide five examples of hypothesis-driven and data-driven applications. THINGS-data constitutes the core public release of the THINGS initiative (https://things-initiative.org) for bridging the gap between disciplines and the advancement of cognitive neuroscience.
Article
Full-text available
The human brain achieves visual object recognition through multiple stages of linear and nonlinear transformations operating at a millisecond scale. To predict and explain these rapid transformations, computational neuroscientists employ machine learning modeling techniques. However, state-of-the-art models require massive amounts of data to properly train, and to the present day there is a lack of vast brain datasets which extensively sample the temporal dynamics of visual object recognition. Here we collected a large and rich dataset of high temporal resolution EEG responses to images of objects on a natural background. This dataset includes 10 participants, each with 82,160 trials spanning 16,740 image conditions. Through computational modeling we established the quality of this dataset in five ways. First, we trained linearizing encoding models that successfully synthesized the EEG responses to arbitrary images. Second, we correctly identified the recorded EEG data image conditions in a zero-shot fashion, using EEG synthesized responses to hundreds of thousands of candidate image conditions. Third, we show that both the high number of conditions as well as the trial repetitions of the EEG dataset contribute to the trained models’ prediction accuracy. Fourth, we built encoding models whose predictions well generalize to novel participants. Fifth, we demonstrate full end-to-end training of randomly initialized DNNs that output EEG responses for arbitrary input images. We release this dataset as a tool to foster research in visual neuroscience and computer vision.
Article
Neuronal activity in the primary visual cortex (V1) is driven by feedforward input from within the neurons’ receptive fields (RFs) and modulated by contextual information in regions surrounding the RF. The effect of contextual information on spiking activity occurs rapidly and is therefore challenging to dissociate from feedforward input. To address this challenge, we recorded the spiking activity of V1 neurons in monkeys viewing either natural scenes or scenes where the information in the RF was occluded, effectively removing the feedforward input. We found that V1 neurons responded rapidly and selectively to occluded scenes. V1 responses elicited by occluded stimuli could be used to decode individual scenes and could be predicted from those elicited by non-occluded images, indicating that there is an overlap between visually driven and contextual responses. We used representational similarity analysis to show that the structure of V1 representations of occluded scenes measured with electrophysiology in monkeys correlates strongly with the representations of the same scenes in humans measured with functional magnetic resonance imaging (fMRI). Our results reveal that contextual influences rapidly alter V1 spiking activity in monkeys over distances of several degrees in the visual field, carry information about individual scenes, and resemble those in human V1.
Chapter
How visual content is represented in neuronal population codes and how to analyze such codes with multivariate techniques. Vision is a massively parallel computational process, in which the retinal image is transformed over a sequence of stages so as to emphasize behaviorally relevant information (such as object category and identity) and deemphasize other information (such as viewpoint and lighting). The processes behind vision operate by concurrent computation and message passing among neurons within a visual area and between different areas. The theoretical concept of "population code" encapsulates the idea that visual content is represented at each stage by the pattern of activity across the local population of neurons. Understanding visual population codes ultimately requires multichannel measurement and multivariate analysis of activity patterns. Over the past decade, the multivariate approach has gained significant momentum in vision research. Functional imaging and cell recording measure brain activity in fundamentally different ways, but they now use similar theoretical concepts and mathematical tools in their modeling and analyses. With a focus on the ventral processing stream thought to underlie object recognition, this book presents recent advances in our understanding of visual population codes, novel multivariate pattern-information analysis techniques, and the beginnings of a unified perspective for cell recording and functional imaging. It serves as an introduction, overview, and reference for scientists and students across disciplines who are interested in human and primate vision and, more generally, in understanding how the brain represents and processes information.
Article
Optogenetics allows precise temporal control of neuronal activity in the brain. Engineered viral vectors are routinely used to transduce neurons with light-sensitive opsins. However, reliable virus injection and light delivery in animals with large brains, such as nonhuman primates, has proven challenging. The Opto-Array is a novel yet simple device that is used to deliver light to extended regions of the cortex surface for high-throughput behavioral optogenetics in large brains. Here we present protocols for surgical delivery of virus (Basic Protocol 1) and implantation of the Opto-Array (Basic Protocol 2) in two separate surgeries in a rhesus monkey's inferior temporal cortex. As a proof of concept, we measured the behavioral performance of an animal detecting cortical optogenetic stimulations (Basic Protocol 3) with different illumination power and duration using the Opto-Array. The animal was able to detect the optogenetic stimulation for all tested illumination powers and durations. Regression analysis also showed both power and duration of illumination significantly modulate the detectability of the optogenetic stimulation. The outcome of this approach is superior to the standard practice of injecting and recording through a chamber for large areas of the cortex surface. Moreover, the chronic nature of the Opto-Array allows perturbation of neuronal activity of the same site across multiple sessions because it is highly stable; thus, data can be pooled over months. The detailed surgical method presented here makes it possible to use optogenetics to modulate neuronal activity across large regions of the cortex surface in the nonhuman primate brain. This method also lays the groundwork for future attempts to use optogenetics to restore vision in humans. © 2023 The Authors. Current Protocols published by Wiley Periodicals LLC. Basic Protocol 1: Virus injection surgery Basic Protocol 2: Opto-Array implantation surgery Basic Protocol 3: Cortical Perturbation Detection (CPD) task behavioral training.