PatentPDF Available

Abstract

An interactive and responsive presentation of spatially associated information related to a user ' s environment based on the user ' s context is provided. Aspects of a 3D audio cloud system extract information about entities in the user ' s environment , and generate and provide the user with a 3D audio cloud that presents a spatial audio representation of entities in the user ' s environment that gives the user cues as to where the entities are located with respect to the user. An entity can be represented in a descriptive manner that provides the user with various types of information about the entity (e. g. , shape , size , color , others ' opinions , actions , emotions). Further , aspects of the 3D audio cloud system dynamically update the 3D audio cloud based on the user ' s location , position , or orientation , receive selection input from the user associated with the 3D audio cloud , and retrieve and present additional information as requested by the user. (51) Publication Classification Int. Ci .
US
20180217804A1
(
19
)
United
States
(
12
)
Patent
Application
Publication
(
10
)
Pub
.
No
. :
US
2018
/
0217804
A1
Manohar
et al
.
(
43
)
Pub
.
Date
:
Aug
. 2 ,
2018
(
54
)
RESPONSIVE
SPATIAL
AUDIO
CLOUD
(
71
)
Applicant
:
Microsoft
Technology
Licensing
,
LLC
,
Redmond
,
WA
(
US
)
(
52
) U . S .
CI
.
???
.
GO6F
3 / 16 (
2013
.
01
) ;
GIOL
13
/
00
(
2013
.
01
) ;
H04W
4 /
008
(
2013
.
01
) ;
G06K
9 /
00671
(
2013
.
01
) ;
G06F
3704842
(
2013
.
01
)
(
57
)
ABSTRACT
(
72
)
Inventors
:
Swaminathan
Manohar
,
Bangalore
(
IN
) ;
Swapna
C .
Joshi
,
Bloomington
,
IN
(
US
) ;
Sujeath
Pareddy
,
Hyderabad
( IN ) ;
Abhay
Kumar
Agarwal
,
Bangalore
(
IN
)
(
73
)
Assignee
:
Microsoft
Technology
Licensing
,
LLC
,
Redmond
,
WA
(
US
)
(
21
)
Appl
.
No
. : 15 /
423
,
281
(
22
)
Filed
:
Feb
. 2 ,
2017
An
interactive
and
responsive
presentation
of
spatially
associated
information
related
to a
user
' s
environment
based
on
the
user
' s
context
is
provided
.
Aspects
of
a
3D
audio
cloud
system
extract
information
about
entities
in
the
user
' s
environment
,
and
generate
and
provide
the
user
with
a
3D
audio
cloud
that
presents
a
spatial
audio
representation
of
entities
in
the
user
' s
environment
that
gives
the
user
cues
as
to
where
the
entities
are
located
with
respect
to
the
user
.
An
entity
can
be
represented
in
a
descriptive
manner
that
provides
the
user
with
various
types
of
information
about
the
entity
( e . g . ,
shape
,
size
,
color
,
others
'
opinions
,
actions
,
emotions
) .
Further
,
aspects
of
the
3D
audio
cloud
system
dynamically
update
the
3D
audio
cloud
based
on
the
user
' s
location
,
position
,
or
orientation
,
receive
selection
input
from
the
user
associated
with
the
3D
audio
cloud
,
and
retrieve
and
present
additional
information
as
requested
by
the
user
.
(
51
)
Publication
Classification
Int
.
Ci
.
GO6F
3 /
16
(
2006
.
01
)
GIOL
13
/
00
(
2006
.
01
)
G06F
3 /
0484
(
2006
.
01
)
G06K
9 /
00
(
2006
.
01
)
106
0
5 .
102
. .
-
104
Ca
108c
108a
108n
110a
1101
110b
. . .
1100
110c
110n
Patent
Application
Publication
Aug
. 2 ,
2018
Sheet
lof9
US
2018
/
0217804
A1
6
??
,
102
* * * * * * * * . * * . . . . .
•••
• •
104
1083
187
LELIEEETLASS
-
SELEMEIELTANTALSE
+ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - +
110a
|
130b
|
110c
0
FIG
.
1A
Patent
Application
Publication
Aug
. 2 ,
2018
Sheet
2
of
9
US
2018
/
0217804
A1
gestellt
[ { "
name
" : { "
wall
) "
confidence
" : 0 .
9889965057373047
) ,
"
name
" : "
indoor
confidence
" : 0 .
9854915738105774
) : 1
"
name
" : "
desk
" , "
confidence
" : 0 .
8862359523773193
) ,
"
name
" : "
office
" , "
conlidence
" : 0 .
5645751357078552
) : 1
"
name
" : "
duttered
" , "
confidence
" : 0 .
138368651270866411
13
114
112
desktop
wall
de
conference
lamp
Sitting
table
artwork
sing
indoor
clock
Shelt
indoor
floor
group
White
table
cluttered
room
OKK
shelf
black
people
gou
stop
many
office
116
* * *
san
FIG
.
1B
Patent
Application
Publication
Aug
. 2 ,
2018
Sheet
3
of
9
US
2018
/
0217804
A1
106
116
102
FIG
.
1C
p
Environmental
capture
device
104
Coordinate
applier
218
Patent
Application
Publication
Video
-
to
-
frames
converter
210
3D
audio cloud
generator
220
Image
-
to
-
text
converter
212
Selection
receiver
222
Word
cloud
generator
214
Web
information
downloader
224
iirin
Aug
. 2 ,
2018
Sound
properties
converter
216
Navigation
support
generator
226
Sheet
4
of
9
204
3D
Audio Cloud
System
206
102
US
2018
/
0217804
A1
FIG
. 2
Patent
Application
Publication
Aug
. 2 ,
2018
Sheet
5
of
9
US
2018
/
0217804
A1
300
Start
-
302
-
304
Obtain
information
about
space
around
user
-
306
Generate
presentation
of
information
p
308
Provide
information
to
user
310
NO
Changes
detected
?
Yes
Dynamically
modify
information
based
on
user
' s
position
314
Receive
a
selection
?
316
Retrieve
additional
Yested
information
based
on
selection
No
game
398
End
FIG
. 3
Patent
Application
Publication
Aug
. 2 ,
2018
Sheet
6
of
9
US
2018
/
0217804
A1
400
p
Computing
Device
408
409
we
were
we
were
w e
weer
ever
we
were
so
w e
were
we
ever
we
were
ever
se
w
we
-
404
w w
Removable
Storage
System
Memory
a
s
w w
410
-
405
w
Operating
System
-
w
-
w
Non
-
Removable
Storage
w w
-
w
-
w w
406
412
402
w
Program
Modules
w
-
w
Input
Device
( s )
-
w w
-
-
w
-
w
3D
Audio
Cloud
System
206
414
-
w w
Diiiiiiiiiii
-
w
w
w
-
Output
Device
( s )
-
w
Processing
Unit
winnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
-
w
-
w
-
w
416
-
w
-
-
w
*
vi
w
Communication
Connections
-
www
-
-
wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww
me
a
-
-
w
-
ww
w
w w w . - - - - -
418
Other
Computing
Devices
FIG
. 4
US
2018
/
0217804
A1
00S
* * * * * * * * * *
-
OIS
-
* *
* * * * * * * *
*
*
*
*
mm
-
OTS
- -
. . .
www
* * *
www
* * * *
war
Sheet
7
of
9
.
- - - - - - - -
- -
530
-
MOMO
17
" 0 "
OOOOOOOO
DDDDDDDD
MOBILE
COMPUTING
DEVICE
FIG
.
5A
-
- -
*
-
-
-
- -
- - -
-
* * * * * * * *
- - - -
meinme
map
wewe
ava
www
.
oec
. -
Sevme
doma
*
*
* * *
* * * *
* * *
* * * </