Conference PaperPDF Available

VR is on the Edge: How to Deliver 360° Videos in Mobile Networks

Conference Paper

VR is on the Edge: How to Deliver 360° Videos in Mobile Networks

Abstract and Figures

VR/AR is rapidly progressing towards enterprise and end customers with the promise of bringing immersive experience to numerous applications. Soon it will target smartphones from the cloud and 360° video delivery will need unprecedented requirements for ultra-low latency and ultra-high throughput to mobile networks. Latest developments in NFV and Mobile Edge Computing reveal already the potential to enable VR streaming in cellular networks and to pave the way towards 5G and next stages in VR technology. In this paper we present a Field Of View (FOV) rendering solution at the edge of a mobile network, designed to optimize the bandwidth and latency required by VR 360° video streaming. Preliminary test results show the immediate benefits in bandwidth saving this approach can provide and generate new directions for VR/AR network research.
Content may be subject to copyright.
VR is on the Edge: How to Deliver 360°Videos in Mobile
Networks
Simone Mangiante
Vodafone Group R&D
Newbury, UK
simone.mangiante@vodafone.com
Guenter Klas
Vodafone Group R&D
Newbury, UK
guenter.klas@vodafone.com
Amit Navon
Huawei, Network Technology Lab
Israel
amit.navon@huawei.com
Zhuang GuanHua
Huawei, Network Technology Lab
Nanjing, China
zhuangguanhua@huawei.com
Ju Ran
Huawei, Network Technology Lab
Nanjing, China
juran@huawei.com
Marco Dias Silva
Vodafone Group R&D
Newbury, UK
marco.silva1@vodafone.com
ABSTRACT
VR/AR is rapidly progressing towards enterprise and end customers
with the promise of bringing immersive experience to numerous
applications. Soon it will target smartphones from the cloud and
360
°
video delivery will need unprecedented requirements for ultra-
low latency and ultra-high throughput to mobile networks. Latest
developments in NFV and Mobile Edge Computing reveal already
the potential to enable VR streaming in cellular networks and to
pave the way towards 5G and next stages in VR technology.
In this paper we present a Field Of View (FOV) rendering solution
at the edge of a mobile network, designed to optimize the bandwidth
and latency required by VR 360°video streaming.
Preliminary test results show the immediate benets in band-
width saving this approach can provide and generate new directions
for VR/AR network research.
CCS CONCEPTS
Networks Mobile networks
;Logical / virtual topologies;
Cloud computing;
Human-centered computing Virtual re-
ality;
KEYWORDS
360
°
video delivery, Field of view, Mobile network, Edge computing
ACM Reference format:
Simone Mangiante, Guenter Klas, Amit Navon, Zhuang GuanHua, Ju Ran,
and Marco Dias Silva. 2017. VR is on the Edge: How to Deliver 360
°
Videos
in Mobile Networks. In Proceedings of VR/AR Network ’17, Los Angeles, CA,
USA, August 25, 2017, 6 pages.
https://doi.org/10.1145/3097895.3097901
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specic permission
and/or a fee. Request permissions from permissions@acm.org.
VR/AR Network ’17, August 25, 2017, Los Angeles, CA, USA
©
2017 Copyright held by the owner/author(s). Publication rights licensed to Associa-
tion for Computing Machinery.
ACM ISBN ISBN 978-1-4503-5055-6/17/08.. . $15.00
https://doi.org/10.1145/3097895.3097901
1 INTRODUCTION
In recent years, virtual and augmented reality (VR/AR) applica-
tions delivered over wireless networks have attracted interest from
academia and industry. They have been often identied as killer
use cases of the 5G ecosystem and been showcased at industrial
and standardization events [14].
For enterprise users and vertical markets such as manufactur-
ing and design, health care, transportation, and retail, VR/AR is
expected to raise productivity, allowing workers to visually interact
with data (e.g. remote maintenance). For consumers, VR/AR will
provide immersive experiences and personalized content. Content
providers are already developing solutions to enrich knowledge
by combining video and augmented information for training and
education, sports, tourism, remote diagnostics and surgery. Addi-
tionally, gaming on VR/AR will put players into interactive scenes
and social networks will encourage users to share those experi-
ences.
However, a real immersive VR experience is yet to come due
to several technical challenges [
3
], one of them being its delivery
through a mobile network. Today, virtual reality services are mainly
consumed statically by nature, using powerful, customized, heavy
head mounted displays (HMDs) like Oculus
1
and HTC Vive
2
. We
envision that the current localized VR/AR applications will move
to the cloud and be delivered to smartphones, as their operating
systems and apps will include VR capabilities3.
360
°
multi-perspective immersive videos have a key role in en-
abling VR/AR applications. VR trac quadrupled in 2015, from 4.2
PB per month in 2014 to 17.9 PB per month in 2015. Globally, VR
trac will increase 61-fold between 2015 and 2020 [
7
]. In 2016 we
observed an increased spread of 360
°
videos: more than 8000 new
videos on YouTube were watched more than 250,000 times daily;
more than 1000 new videos were created for the Oculus platform;
worldwide popular events such as NBA games and the US Open
golf tournament were streamed live using 360°video technology.
The development of VR 360
°
videos focuses on user immersive
experience and evolves from single view to 3D/multi-view and
from no interaction to full interaction, which will bring the network
tremendous challenges in allowing ultra-high throughput and ultra-
low latency. Furthermore, additional challenges for VR are posed
1https://www.oculus.com/
2https://www.vive.com
3Google Daydream, https://vr.google.com/daydream/
VR/AR Network ’17, August 25, 2017, Los Angeles, CA, USA S. Mangiante, G. Klas, A. Navon et al.
by end devices that must be lightweight but at the same time have
high processing power to handle VR video processing.
After presenting VR network requirements in Section 2, we show
in Section 3 how current developments in mobile network can be
exploited and propose a new solution in Section 4. We analyze
results from our preliminary tests in Section 5 before concluding
with planned further work in Section 6.
2 NETWORK REQUIREMENTS
Latency is the most important quality parameter of VR/AR ap-
plications. VR/AR is latency sensitive because human perception
requires accurate and smooth movements in vision. Large amounts
of latency can lead to a detached VR experience and can contribute
to a motion sickness sensation. VR/AR developers and industries
agree that application round-trip latency should be less than 20
ms [
5
] in order for the Motion-To-Photon latency (MTP) [
11
] to
become imperceptible. Other researches indicate that 15 ms should
be the required MTP threshold, and even 7 ms would be ideal
4
. Cur-
rent local VR/AR systems use customized and tuned technologies
in their HMDs to meet the 20 ms threshold. In network-based/cloud
VR/AR applications, the actual latency budget for the transport net-
work can be derived by estimating the latency of all the components
involved in the process, as we did in Table 2, and happens to be
extremely low. When streaming video, caching can help reducing
latency, but interactive VR/AR user actions are less predictable than
static content.
The network bandwidth for VR/AR is the throughput required
to stream the related 360
°
video to a user and it is another critical
parameter for the quality of VR/AR application. The resolution
of a VR immersive video viewed using a HMD positioned few
centimeters from the user’s eyes needs to approximate the amount
of detail the human retina can perceive. Thus 8K quality and above
is necessary for VR, as a 4K VR 360
°
video only has 10 pixels per
degree, equivalent to a 240p on a TV screen. We believe that the
VR 360
°
video experience may evolve through the stages listed in
Table 1, requiring a network throughput of 400 Mbps and above,
more than 100 times higher than the current throughput supporting
HD video services.
2.1 Bandwidth, Compute and Latency Tradeo
Display technology requires very high refresh rate to provide im-
mersive experiences. Switching views in VR is similar to switching
channels in IPTV, whose latency is about 700 ms using FCC (fast
channel change). Taking into account the MTP latency threshold of
20 ms and considering that a smooth VR playback typically requires
throughput to be 1.5 times the average video bitrate, we deduce the
network burst with fast-ush performance should be more than 50
times better than current IPTV.
Current VR 360
°
video services are in early-stage and delivered
using a wasteful and still limited technology [
8
], since the full
360
°
video is transmitted over the network and the user device has
to process it according to mouse or head movements in order to
switch to a new eld of view. Figure 1 shows the complete process
of generating and consuming such a video:
(1) Multi camera array is used to capture video
4http://blogs.valvesoftware.com/abrash/latency-the-sine-qua- non-of- ar- and-vr/
Figure 1: VR 360°video producing and viewing
(2)
Images are then projected on a sphere and stitched together
to obtain a spherical video
(3)
The spherical video is unfolded to 2D plane using the equirect-
angular projection
(4)
Viewer can see any view in every direction: given a specic
view point
(θ0,ϕ0)
, the 2D plane image is projected into a
sphere and then rendered on the display.
Massive compute power is required by the end user device to
process and render the complex equirectangular video quickly
enough to ensure minimal MTP latency. Therefore, a
bandwidth/compute/latency tradeo should be considered as a new
media network model (Figure 2), improved from the traditional
store and forward model, for delivery optimization and adaptive
services for dierent users. Table 1 summarizes our vision regarding
bandwidth and latency requirements for network VR/AR applica-
tions, related to dierent VR experience stages [
10
]. The network
bandwidth requirement is estimated based on 1.5 times the bit rate.
The latency value is determined by MTP latency requirement and
VR development stage forecast.
3 MOBILE NETWORK ENVIRONMENT
A telecommunication operator’s 4G/LTE network is generally com-
posed of a backhaul divided into hierarchical levels of aggregation
that we call POC (Point Of Concentration). From a base station
Figure 2: BCL model
VR is on the Edge: How to Deliver 360°Videos in Mobile Networks VR/AR Network ’17, August 25, 2017, Los Angeles, CA, USA
Table 1: VR network requirements (bandwidth and latency)
VR resolution Equivalent TV res. Bandwidth Latency
Early stage VR (current) 1K*1K@visual eld
2D_30fps_8bit_4K
240P 25 Mbps 40 ms
Entry level VR 2K*2K@visual eld
2D_30fps_8bit_8K
SD 100 Mbps 30 ms
Advanced VR 4K*4K@visual eld
2D_60fps_10bit_12K
HD 400 Mbps 20 ms
Extreme VR 8K*8K@visual eld
3D_120fps_12bit_24K
4K 1 Gbps (smooth play)
2.35 Gbps (interactive)
10 ms
(eNodeB) to the telecom operator’s core network (Evolved Packet
Core or EPC in 3GPP terminology [
1
]), traditionally hosting LTE
control and user plane components (e.g. S-GW, P-GW as dened
by 3GPP), trac typically hits 3 POCs along the 3GPP S1 interface:
POC3. the aggregation point closest to the devices. Instances
deployed in a large national network: around few thousands,
aggregating multiple (order of magnitude of 10) base stations;
POC2: an intermediate aggregation point. Instances deployed
in a large national network: from several hundreds to a few
thousands;
POC1: the aggregation point closest to the EPC components.
Instances deployed in a large national network: up to very
few hundreds.
In this typical topology, the average network roundtrip latency
is distributed as shown in Figure 3.
3.1 Network Function Virtualization (NFV)
In the last years, the rising paradigms of Software Dened Networks
(SDN) and Network Function Virtualization (NFV) are transform-
ing telecom operators’ networks, making them more exible and
responsive. NFV leverages IT technologies (e.g. virtualization, stan-
dard servers, open software) to virtualize network functions in
order to create better managed services and dynamically chain
them. Virtualization and convergence of xed and mobile trans-
port generated other initiatives like Cloud-RAN [
6
] and CORD
5
,
aiming to achieve the freedom to move and scale network func-
tions throughout the whole network. Developments towards 5G [
2
]
added more features and use cases, promising that 5G networks will
be able to support communications for some special scenarios not
currently supported by 4G networks. 5G will play an instrumental
role in enabling low latency and high throughput networks for a
smooth VR/AR experience.
The current process of “cloudication” of telecom operator net-
works led to the introduction of MEC (Mobile Edge Computing, to
be expanded to Multi-access Edge Computing) [4, 12].
3.2 Mobile Edge Computing (MEC)
MEC provides distributed cloud-computing capabilities at the edge
of the mobile network, within the Radio Access Network (RAN)
and in close proximity to customers. The aim is to reduce network
5Central Oce Re-architected as a Datacenter, http://opencord.org/
congestion and response time, achieve highly ecient network
operation, and oer a better user experience. Leveraging IT tech-
nologies and APIs, MEC also allows mobile operators to open their
network to authorized application developers and content providers,
providing direct access to real-time information from the underly-
ing radio transport (e.g. an API to the Radio Network Information
Service which provides real-time details about the device’s radio
access bearer). Moving workloads to the edge is then the argument
for enabling highly responsive services, supported by smaller and
slimmer devices, with improved user QoE (Quality of Experience)
in 4G networks immediately, without waiting for 5G enhancements.
Thanks to NFV elasticity and the increasing availability of com-
pute, storage, and network resources at many locations within an
operator network, MEC platforms can be deployed in principle in
any location: at base stations, co-hosted with LTE small cells [
13
],
at any POC aggregation hub (e.g. next to a router, provided that
sucient resources are available), at Cloud-RAN sites, and in the
main datacenters.
Any deployment strategies will depend on a number of factors,
such as level of new CAPEX costs involved, opportunities for shar-
ing of infrastructure between MEC software platform and other
network functions (e.g. distributed 3GPP functions), and use case
requirements. MEC platforms, as stated by ETSI MEC ISG, have to
leverage management and orchestration functionality from existing
NFV environments as much as possible to achieve easy deployment
and integration in the mobile network [
9
]. Therefore, MEC has
already the potential to enable high quality 360
°
video delivery and
obtain other signicant benets like saving backhaul bandwidth.
We envision that 360
°
video delivery components (e.g. streaming en-
gine, transcoding, caching) hosted in a MEC platform at a location
providing the right latency, combined with forthcoming display and
Figure 3: Example of a mobile network topology
VR/AR Network ’17, August 25, 2017, Los Angeles, CA, USA S. Mangiante, G. Klas, A. Navon et al.
computing technologies, will achieve a total latency comparable to
current local VR scenarios, as shown in Table 2.
4 PROPOSED SOLUTION
4.1 Edge FOV Rendering
In order to address the challenges presented in Section 2, we propose
an edge processing technology that will be able to perform Field Of
View (FOV) rendering of 360
°
VR videos [
15
], optimize the required
bandwidth and reduce the processing power requirements on end
devices, also improving their battery utilization. Ultra-low network
latency must be guaranteed in order to achieve eective real-time
FOV rendering as the experience becomes interactive and upstream
control indications are sent from end devices to the network.
Optionally, motion prediction can be performed at the edge in
order to anticipate the FOV requested by the user. This component
can loosen the VR latency requirement where the network cannot
support it. We believe that the transmission delay can be relaxed to
a few hundred milliseconds at a low cost of bandwidth.
The overall ow is specied in Figure 4:
(1)
A 360
°
video service (e.g. generating content from a public
cloud and/or from cameras at an event venue) produces
a 360
°
video stream at the video hub, which is delivered
to a
µ
Cloud.
µ
Cloud is what we call an either virtualized
or physical deployment, designed to tackle specic heavy
and real-time calculation like video transcoding, co-hosted
within a MEC platform or directly attached to it through a
fast network.
(2)
The
µ
Cloud processes the 360
°
video and performs optional
motion prediction, FOV extraction, and transcoding and
optimization for mobile devices. The computed FOV is then
streamed to the end user.
(3)
FOV control indications, representing the angle the user is
currently looking at, are sent upstream from the user to the
µ
Cloud, which in turns computes and streams the requested
FOV.
Optionally, a low-resolution full view stream can be delivered to
end users’ devices together with FOV. The low-resolution stream
acts as a fallback option to ensure a minimal user experience in
bad network situations with little extra cost. Should the FOV data
fail to be delivered in time, the user device loads and displays the
low-resolution stream. When the network recovers, the user device
may switch back to the FOV input stream.
Figure 4: Edge FOV rendering ow
Table 2: Latency of VR components in dierent approaches
(RTT in ms)
Component
Local VR
Current online
VR
Future online
VR with MEC
Sensor 1 1 1
Transport 2 (USB and HDMI) 40 (network to cloud)
5
(network to MEC
location)
Computing
35100+ 7
Display 10 (refresh) 15 (decode and refresh)
5
(decode and dynamic
refresh)
Total 18 150+ 18
4.2 Eective Resolution Vs. Delivered
Resolution
When the spherical video is unfolded to a 2D plane using the
equirectangular projection, 4K 360
°
source video quality means
that a user actually enjoys a much lower eective resolution (~1K)
at any selected FOV. Typically, the FOV is 90
°
wide horizontally and
approximately 60
°
wide vertically. The ratio between the FOV and
the source video is 1:3 on the vertical axis and 1:4 on the horizontal
axis, accounting to 1:12 ratio in the total area. Translating this into
pixels, in order to provide a 4K video at the user device equipped
with a 16:9 display, the source equirectangular projection video
delivered over the network will have to be at a 16K resolution.
One of the main advantages of edge FOV rendering technology
for VR is actually to enable a 4K eective resolution quality that
cannot be achieved today with traditional streaming mechanism
in a mobile network. Besides improving latency to end users and
saving bandwidth by reducing the amount of data to be transmitted,
applying FOV rendering technologies to a MEC node/platform o-
loads the computation power from the end device to the edge of
the network and enables thin clients to support high quality VR
experience.
5 PRELIMINARY EVALUATION
To assess the role and value of edge computing to reduce trac
sent over an operator’s radio access network for VR 360
°
live and
on-demand video streaming, we deployed a 4G LTE test lab envi-
ronment whose logical network topology is depicted in Figure 5.
The user device (laptop or mobile) is connected to a 4G/LTE small
cell. A MEC platform/node is deployed transparently between the
small cell and the EPC and it can be enabled by conguration to
intercept and process/redirect the trac. The
µ
Cloud FOV enhance-
ment is connected to both the MEC node and the EPC to enable
two dierent scenarios (long-dashed and dotted path in Figure 5):
FOV rendering hosted at the edge of the network and FOV render-
ing hosted in the core of the network. We can simulate dierent
network segments’ conditions (e.g. MEC location, EPC location,
the public internet) using impairment nodes. The central cloud is
connected to an Eyesir 360
°
4K source camera
6
and hosts a media
streaming server. The solid path represents the scenario where the
full 360°video is delivered to the end user.
6http://www.perfant.com/en/product.html
VR is on the Edge: How to Deliver 360°Videos in Mobile Networks VR/AR Network ’17, August 25, 2017, Los Angeles, CA, USA
Figure 5: Logical network topology of the test environment
Table 3 lists the main metrics characterizing the lab setup for
the preliminary tests. The values represent the currently available
baseline using the equipment in our controlled lab environment.
They reproduce a realistic scenario where the
µ
Cloud is deployed
either in POC1 or POC2 and they may be further tuned as explained
in Section 6.
We sequentially tested two delivery options, targeting two users
through the long-dashed (User #1) and solid (User #2) path in Fig-
ure 5. The trac ow is as follows:
(1)
A live stream from the source camera or a 4K VR le (on-
demand) is published to the Central Cloud.
(2)
The stream destined to User #1 is routed via the
µ
Cloud. The
same stream is delivered directly to User #2.
(3)
For User #1 FOV rendering is applied on the
µ
Cloud and the
selected FOV is sent to the user in 4K. For User #2 the full
360°video is delivered.
In this rst stage of evaluation, end-to-end live streams have
been successfully transmitted in our lab environment, and have
been tested with and without MEC. A static FOV cropping and
rendering (without real-time FOV control indicators from the user)
is applied in the
µ
Cloud without any special hardware acceleration,
then the FOV is transmitted via the mobile network and played by
Potplayer
7
in a PC client. The resulting actual delay between frame
output at source and playback at the user device (including total net-
work latency and other delays) is 859 ms, which illustrates that we
could achieve sub-second delay, while real network environments
settle at several seconds at present.
The resolution of the original full view at the source is
3840x1920
pixels. The
µ
Cloud computes a predened FOV resolution of
1280x1024
pixels, representing a horizontal view angle of 120
°
and a
vertical view angle of 96
°
. This cropped portion safely approximates
what a user typically sees (stated in Section 4.2) and it is an accu-
rate FOV to be delivered to the client device. Such computed area
watched by the end user occupies 17.8% of the original full view,
which means the bandwidth saving could theoretically reach 82.2%.
7http://potplayer.daum.net/
Table 3: Test lab setup metrics
Metric Value
Available bandwidth between central cloud and µCloud 1 Gbps
Available bandwidth between µCloud and user 22 Mbps
Network latency (RTT) between central cloud and user 30 ms
Network latency (RTT) between µCloud and user 13 ms
Table 4: Collected metrics from preliminary tests
Metric No µCloud
(User #2)
µ
Cloud & FOV
(User #1)
Frames per second (FPS) during play 18 29
Throughput between source camera
and µCloud
N.A. 30 Mbps
Throughput observed at the user device
22 Mbps 5.85 Mbps
Trac savings in core and radio access
80.5%
The bandwidth saving observed in our tests could reach 80.5%. The
rst subset of collected data is shown in Table 4.
In order to get a smooth playback, the typical throughput is
recommended to be 1.5 times the average video bitrate. In our
experiment, the available bandwidth at the air interface (the bot-
tleneck within the access network) is 22 Mbps, not even matching
the average bitrate of the full 4K video stream. That causes frame
loss and results in lower FPS for User #2, while User #1 needs less
throughput for the FOV reduced resolution and experiences higher
FPS. FOV processing and rendering at a MEC location reduces the
mobile trac in dierent segments of an operator network: the
radio access, because a FOV stream consumes a considerably (more
than 80%) less amount of bandwidth; the backhaul and core, because
the full stream ows once from the central cloud to the
µ
Cloud
(where it can be cached for on-demand usage) and it is not repli-
cated as many times as users requesting it. Given the rst promising
results, we believe that leveraging MEC infrastructure in VR video
delivery leads to signicant benets for dierent stakeholders: mo-
bile network operators, who own the enabling infrastructure to
provide new VR services and save bandwidth in the backhaul/core;
VR broadcasters and video content providers, who can reach a mas-
sive amount of mobile users; and end users, who can experience
good quality VR services from early stage.
6 CONCLUSION AND FUTURE WORK
In this paper we presented a FOV rendering application at the
edge of a mobile network enabling improved VR 360
°
live video
delivery to mobile devices. Considering our decomposition of future
online VR latency requirements in Table 1, many development
opportunities exist in each network VR processing stage. More
research and validation over dierent types of access networks,
with dierent user equipment and multiple device connectivity,
will be conducted. We plan to exploit our modular and congurable
test lab to get more data by testing more scenarios. Leveraging
various equipment and impairment behaviors, we can simulate
realistic trac conditions varying throughout the day (e.g. heavy
congestion and higher latency at peak times). As an example, we will
compare MEC platform deployments at various locations providing
dierent resources and dierent values of latency to the end users,
in order to nd the sweet spot in the bandwidth/compute/latency
model introduced in Section 2 and to identify the business impact.
User mobility was not in the scope of this work, but it will be
investigated in the future. It will become a requirement as supported
devices get lighter and more portable, and more complex mobility-
aware applications arise. Users can switch to a base station served
VR/AR Network ’17, August 25, 2017, Los Angeles, CA, USA S. Mangiante, G. Klas, A. Navon et al.
by the same MEC node or by a dierent MEC node: both scenarios
will be considered in order to develop the best mechanisms to cope
with mobility.
Hardware and software components of the
µ
Cloud will be tar-
get of future research, with the aim of understanding the tradeo
between bespoke hardware features designed to enhance video
applications (e.g. GPU and custom processors) and smarter soft-
ware counterparts executable on commercial o-the-shelf machines.
This work will be assessing also the eciency of FOV rendering at
the edge considering the broad variety of equipment available at
dierent locations within mobile networks.
More research will be done on VR value-added services (e.g.
content delivery network optimized for VR/AR) in edge clouds
inside or close to the access network, testing a pre-commercial
solution articulating the benets for dierent stakeholders for live
and on-demand scenarios.
ACKNOWLEDGMENTS
We thank Rame Canolli, Nick Edwards from Vodafone Innovation
Lab, Han Lei, Li Feng, Li Jin, Zhu JiRong, Dr. Itsik Dvir, Amiram Al-
louch and Guy Almog from Network Architecture Technology Lab
of Huawei Technologies, for all the valuable guidance and support
in every stage of the proposal and the evaluation deployment.
REFERENCES
[1]
3GPP. 2016. 3GPP Evolved Universal Terrestrial Radio Access (EUTRA) and Evolved
Universal Terrestrial Radio Access Network (E-UTRAN) - Overall Description, Stage
2. TS 36.300 Release 13. 3GPP.
[2]
Jerey G Andrews, Stefano Buzzi, Wan Choi, Stephen V Hanly, Angel Lozano,
Anthony CK Soong, and Jianzhong Charlie Zhang. 2014. What will 5G be? IEEE
Journal on selected areas in communications 32, 6 (2014), 1065–1082.
[3]
Ejder Baştuğ, Mehdi Bennis, Muriel Médard, and Mérouane Debbah. 2016. To-
wards Interconnected Virtual Reality: Opportunities, Challenges and Enablers.
arXiv preprint arXiv:1611.05356 (2016).
[4]
Michael Till Beck, Martin Werner, Sebastian Feld, and S Schimper. 2014. Mobile
edge computing: A taxonomy. In Proc. of the Sixth International Conference on
Advances in Future Internet. Citeseer.
[5]
John Carmack. 2013. Latency Mitigation Strategies. (2013). Retrieved 2017-05-26
from https://www.twentymilliseconds.com/post/latency-mitigation-strategies/
[6]
Aleksandra Checko, Henrik L Christiansen, Ying Yan, Lara Scolari, Georgios
Kardaras, Michael S Berger, and Lars Dittmann. 2015. Cloud RAN for mobile
networks - A technology overview. IEEE Communications surveys & tutorials 17,
1 (2015), 405–426.
[7]
Cisco. 2016. Cisco Visual Networking Index: Global Mobile Data Trac Forecast
Update, 2015-2020. (2016). Retrieved 2017-05-26 from http://www.cisco.com/c/
en/us/solutions/service-provider/visual- networking-index- vni/index.html
[8]
Mohammad Hosseini and Viswanathan Swaminathan. 2016. Adaptive 360 VR
Video Streaming: Divide and Conquer! arXiv preprint arXiv:1609.08729 (2016).
[9]
Yun Chao Hu, Milan Patel, Dario Sabella, Nurit Sprecher, and Valerie Young. 2015.
Mobile edge computing - A key technology towards 5G. ETSI White Paper 11
(2015).
[10]
Huawei. 2016. Whitepaper on the VR-Oriented Bearer Net-
work Requirement. (2016). Retrieved 2017-05-26 from http:
//www-le.huawei.com/~/media/CORPORATE/PDF/white%20paper/
whitepaper-on- the-vr- oriented- bearer-network- requirement-en.pdf
[11]
Katerina Mania, Bernard D Adelstein, Stephen R Ellis, and Michael I Hill. 2004.
Perceptual sensitivity to head tracking latency in virtual environments with
varying degrees of scene complexity. In Proceedings of the 1st Symposium on
Applied perception in graphics and visualization. ACM, 39–47.
[12]
Yuyi Mao, Changsheng You, Jun Zhang, Kaibin Huang, and Khaled B Letaief.
2017. Mobile Edge Computing: Survey and Research Outlook. arXiv preprint
arXiv:1701.01090 (2017).
[13]
Takehiro Nakamura, Satoshi Nagata, Anass Benjebbour, Yoshihisa Kishiyama,
Tang Hai, Shen Xiaodong, Yang Ning, and Li Nan. 2013. Trends in small cell
enhancements in LTE advanced. IEEE Communications Magazine 51, 2 (2013),
98–105.
[14]
Jason Orlosky, Kiyoshi Kiyokawa, and Haruo Takemura. 2017. Virtual and
Augmented Reality on the 5G Highway. Journal of Information Processing 25
(2017), 133–141.
[15]
Feng Qian, Lusheng Ji, Bo Han, and Vijay Gopalakrishnan. 2016. Optimizing 360
video delivery over cellular networks. In Proceedings of the 5th Workshop on All
Things Cellular: Operations, Applications and Challenges. ACM, 1–6.
... In recent years, with the development of wireless networks and the popularity of smart mobile devices, mobile applications such as augmented reality (AR), virtual reality (VR), and facial recognition payment have grown exponentially [1,2]. These applications tend to be computation intensive and require low latency, but the battery capacities, computation resources, and storage capacities of mobile user equipment (UE) are very limited. ...
... The computing task on TD is divided into three parts, which are computed on a local, edge cloud, and D2D RD, respectively. x ij ∈ {0, 1}, ∀i ∈ U , ∀j ∈ K/k 0 is the user association between TD i and RD j, x ij = 1 indicates that TD i offloads part of the computing task to D2D RD j, and otherwise, x ij = 0. Since a TD selects, at most, one D2D RD for computational offloading, there are constraints: ∑ 1], i ∈ U denote the proportion of a computing task on TD i that is offloaded to the edge cloud and D2D RD, respectively. Since the locally computed ratio should be non-negative, α i and β i should satisfy the constraint: 0 ≤ α i + β i ≤ 1. ...
... In Figure 3, we show the number of supported (or unexecuted) TDs versus the total number of TDs in the system. The computing capacity of the MEC server is 50 Mcycles/s, and the number of D2D RDs is 1 2 of the number of TDs. Since the scenarios we study mainly concern computationally intensive tasks, none of the tasks are computed locally. ...
Article
Full-text available
Mobile edge computing (MEC) and device-to-device (D2D) communication can alleviate the resource constraints of mobile devices and reduce communication latency. In this paper, we construct a D2D-MEC framework and study the multi-user cooperative partial offloading and computing resource allocation. We maximize the number of devices under the maximum delay constraints of the application and the limited computing resources. In the considered system, each user can offload its tasks to an edge server and a nearby D2D device. We first formulate the optimization problem as an NP-hard problem and then decouple it into two subproblems. The convex optimization method is used to solve the first subproblem, and the second subproblem is defined as a Markov decision process (MDP). A deep reinforcement learning algorithm based on a deep Q network (DQN) is developed to maximize the amount of tasks that the system can compute. Extensive simulation results demonstrate the effectiveness and superiority of the proposed scheme.
... The motion-to-photon latency includes any delay incurred by motion capture, encoding, communication, sensor fusion, processing, actuator control, rendering and decoding of each frame. The different tasks that lie on the critical path of the motion-to-photon latency, and their associated timings include [45], [46]: ...
Preprint
Full-text available
We present a new approach to Extended Reality (XR), denoted as iCOPYWAVES, which seeks to offer naturally low-latency operation and cost-effectiveness, overcoming the critical scalability issues faced by existing solutions. iCOPYWAVES is enabled by emerging PWEs, a recently proposed technology in wireless communications. Empowered by intelligent (meta)surfaces, PWEs transform the wave propagation phenomenon into a software-defined process. We leverage PWEs to i) create, and then ii) selectively copy the scattered RF wavefront of an object from one location in space to another, where a machine learning module, accelerated by FPGAs, translates it to visual input for an XR headset using PWEdriven, RF imaging principles (XR-RF). This makes for an XR system whose operation is bounded in the physical layer and, hence, has the prospects for minimal end-to-end latency. Over large distances, RF-to-fiber/fiber-to-RF is employed to provide intermediate connectivity. The paper provides a tutorial on the iCOPYWAVES system architecture and workflow. A proof-of-concept implementation via simulations is provided, demonstrating the reconstruction of challenging objects in iCOPYWAVES produced computer graphics.
... All these drawbacks contribute to the user's perception of the experience in the metaverse. Excessive latency, for example, over 30ms in AR applications and over 20ms in VR applications [20], can cause dizziness and nausea for the users. ...
Conference Paper
With the advances of the Internet of Things (IoT) and 5G/6G wireless communications, the paradigms of mobile computing have developed dramatically in recent years, from centralized mobile cloud computing to distributed fog computing and mobile edge computing (MEC). MEC pushes compute-intensive assignments to the edge of the network and brings resources as close to the endpoints as possible, addressing the shortcomings of mobile devices with regard to storage space, resource optimisation, computational performance and efficiency. Compared to cloud computing, as the distributed and closer infrastructure, the convergence of MEC with other emerging technologies, including the Metaverse, 6G wireless communications , artificial intelligence (AI), and blockchain, also solves the problems of network resource allocation, more network load as well as latency requirements. Accordingly, this paper investigates the computational paradigms used to meet the stringent requirements of modern applications. The application scenarios of MEC in mobile augmented reality (MAR) are provided. Furthermore, this survey presents the motivation of MEC-based Metaverse and introduces the applications of MEC to the Metaverse. Particular emphasis is given on a set of technical fusions mentioned above, e.g., 6G with MEC paradigm, MEC strengthened by blockchain, etc.
... All these drawbacks contribute to the user's perception of the experience in the metaverse. Excessive latency, for example, over 30ms in AR applications and over 20ms in VR applications [20], can cause dizziness and nausea for the users. ...
Conference Paper
With the advances of the Internet of Things (IoT) and 5G/6G wireless communications, the paradigms of mobile computing have developed dramatically in recent years, from centralized mobile cloud computing to distributed fog computing and mobile edge computing (MEC). MEC pushes compute-intensive assignments to the edge of the network and brings resources as close to the endpoints as possible, addressing the shortcomings of mobile devices with regard to storage space, resource optimisation, computational performance and efficiency. Compared to cloud computing, as the distributed and closer infrastructure, the convergence of MEC with other emerging technologies, including the Metaverse, 6G wireless communications , artificial intelligence (AI), and blockchain, also solves the problems of network resource allocation, more network load as well as latency requirements. Accordingly, this paper investigates the computational paradigms used to meet the stringent requirements of modern applications. The application scenarios of MEC in mobile augmented reality (MAR) are provided. Furthermore, this survey presents the motivation of MEC-based Metaverse and introduces the applications of MEC to the Metaverse. Particular emphasis is given on a set of technical fusions mentioned above, e.g., 6G with MEC paradigm, MEC strengthened by blockchain, etc.
... All these drawbacks contribute to the user's perception of the experience in the metaverse. Excessive latency, for example, over 30ms in AR applications and over 20ms in VR applications [20], can cause dizziness and nausea for the users. ...
Preprint
Full-text available
With the advances of the Internet of Things (IoT) and 5G/6G wireless communications, the paradigms of mobile computing have developed dramatically in recent years, from centralized mobile cloud computing to distributed fog computing and mobile edge computing (MEC). MEC pushes compute-intensive assignments to the edge of the network and brings resources as close to the endpoints as possible, addressing the shortcomings of mobile devices with regard to storage space, resource optimisation, computational performance and efficiency. Compared to cloud computing, as the distributed and closer infrastructure, the convergence of MEC with other emerging technologies, including the Metaverse, 6G wireless communications, artificial intelligence (AI), and blockchain, also solves the problems of network resource allocation, more network load as well as latency requirements. Accordingly, this paper investigates the computational paradigms used to meet the stringent requirements of modern applications. The application scenarios of MEC in mobile augmented reality (MAR) are provided. Furthermore, this survey presents the motivation of MEC-based Metaverse and introduces the applications of MEC to the Metaverse. Particular emphasis is given on a set of technical fusions mentioned above, e.g., 6G with MEC paradigm, MEC strengthened by blockchain, etc.
... mmWave meets MEC for Transmission Efficiency in Wireless VR: Furthermore, several studies have indicated that introducing FOV into 360 • video will reduce up to 80% of the bandwidth requirements compared to delivering 360 • video, hence lowering the overall necessary transmission data rate [177], [178]. For example, authors in [179] analyze the tradeoff between homogeneous and heterogeneous FOVs for a MEC-based mobile VR delivery model regarding computations and caching tasks. ...
Preprint
Full-text available
Since Facebook was renamed Meta, a lot of attention, debate, and exploration have intensified about what the Metaverse is, how it works, and the possible ways to exploit it. It is anticipated that Metaverse will be a continuum of rapidly emerging technologies, usecases, capabilities, and experiences that will make it up for the next evolution of the Internet. Several researchers have already surveyed the literature on artificial intelligence (AI) and wireless communications in realizing the Metaverse. However, due to the rapid emergence of technologies, there is a need for a comprehensive and in-depth review of the role of AI, 6G, and the nexus of both in realizing the immersive experiences of Metaverse. Therefore, in this survey, we first introduce the background and ongoing progress in augmented reality (AR), virtual reality (VR), mixed reality (MR) and spatial computing, followed by the technical aspects of AI and 6G. Then, we survey the role of AI in the Metaverse by reviewing the state-of-the-art in deep learning, computer vision, and edge AI. Next, we investigate the promising services of B5G/6G towards Metaverse, followed by identifying the role of AI in 6G networks and 6G networks for AI in support of Metaverse applications. Finally, we enlist the existing and potential applications, usecases, and projects to highlight the importance of progress in the Metaverse. Moreover, in order to provide potential research directions to researchers, we enlist the challenges, research gaps, and lessons learned identified from the literature review of the aforementioned technologies.
... Specifically, such a deployment is expected to require a multi-gigabit link, capable of delivering video content at extremely low latency. When streaming content per-frame, one video-frame must be delivered fully within 7 ms to maintain an optimal Quality of Ex-perience (QoE) [5]. Furthermore, the network must be highly reliable, as even modest packet loss is detrimental to the QoE. ...
Conference Paper
Full-text available
Achieving extremely high-quality and truly immer-sive interactive Virtual Reality (VR) is expected to require a wireless link to the cloud, providing multi-gigabit throughput and extremely low latency. A prime candidate for fulfilling these requirements is millimeter-wave (mmWave) communications, operating in the 30 to 300 GHz bands, rather than the traditional sub-6 GHz. Evaluations with first-generation mmWave Wi-Fi hardware, based on the IEEE 802.11ad standard, have so far largely remained limited to lower-layer metrics. In this work, we present the first experimental analysis of the capabilities of mmWave for streaming VR content, using a novel testbed capable of repeatably creating blockage through mobility. Using this testbed, we show that (a) motion may briefly interrupt transmission, (b) a broken line of sight may degrade throughput unpredictably, and (c) TCP-based streaming frameworks need careful tuning to behave well over mmWave.
Article
Virtual reality (VR) users interact with virtual objects using motion-tracked controllers. While many devices utilise abstract button pushes for interactions, some allow for limited finger tracking by estimating finger positions based on sensors. In this study, the Vive Wands and the Valve Index controllers were compared in three tasks: direct interaction with objects (throwing), tool usage (bow) and indirect control of a character (remote-control). Forty-four participants completed each task with both devices and rated the usability of the device after each task. Results showed only differences in preference for the remote-control task. Some participants noted that using the thumbstick of the Index instead of the touchpad of the Wands controller felt more natural in this task. However, performance did not differ between devices in any task. Therefore, future research should not only compare designs of controllers but also consider assets and interactions, as there may be preference and performance differences for certain combinations.
Article
Full-text available
In recent years, virtual and augmented reality have begun to take advantage of the high speed capabilities of data streaming technologies and wireless networks. However, limitations like bandwidth and latency still prevent us from achieving high fidelity telepresence and collaborative virtual and augmented reality applications. Fortunately, both researchers and engineers are aware of these problems and have set out to design 5G networks to help us to move to the next generation of virtual interfaces. This paper reviews state of the art virtual and augmented reality communications technology and outlines current efforts to design an effective, ubiquitous 5G network to help to adapt to virtual application demands. We discuss application needs in domains like telepresence, education, healthcare, streaming media, and haptics, and provide guidelines and future directions for growth based on this new network infrastructure.
Article
Full-text available
Just recently, the concept of augmented and virtual reality (AR/VR) over wireless has taken the entire 5G ecosystem by storm spurring an unprecedented interest from both academia, industry and others. Yet, the success of an immersive VR experience hinges on solving a plethora of grand challenges cutting across multiple disciplines. This article underscores the importance of VR technology as a disruptive use case of 5G (and beyond) harnessing the latest development of storage/memory, fog/edge computing, computer vision, artificial intelligence and others. In particular, the main requirements of wireless interconnected VR are described followed by a selection of key enablers, then, research avenues and their underlying grand challenges are presented. Furthermore, we examine three VR case studies and provide numerical results under various storage, computing and network configurations. Finally, this article exposes the limitations of current networks and makes the case for more theory, and innovations to spearhead VR for the masses.
Article
Full-text available
Cloud Radio Access Network (C-RAN) is a novel mobile network architecture which can address a number of challenges the operators face while trying to support growing end-user's needs. The main idea behind C-RAN is to pool the Baseband Units (BBUs) from multiple base stations into centralized BBU Pool for statistical multiplexing gain, while shifting the burden to the high-speed wireline transmission of In-phase and Quadrature (IQ) data. C-RAN enables energy efficient network operation and possible cost savings on baseband resources. Furthermore, it improves network capacity by performing load balancing and cooperative processing of signals originating from several base stations. This paper surveys the state-of-the-art literature on C-RAN. It can serve as a starting point for anyone willing to understand C-RAN architecture and advance the research on C-RAN.
Article
Full-text available
What will 5G be? What it will not be is an incremental advance on 4G. The previous four generations of cellular technology have each been a major paradigm shift that has broken backwards compatibility. And indeed, 5G will need to be a paradigm shift that includes very high carrier frequencies with massive bandwidths, extreme base station and device densities and unprecedented numbers of antennas. But unlike the previous four generations, it will also be highly integrative: tying any new 5G air interface and spectrum together with LTE and WiFi to provide universal high-rate coverage and a seamless user experience. To support this, the core network will also have to reach unprecedented levels of flexibility and intelligence, spectrum regulation will need to be rethought and improved, and energy and cost efficiencies will become even more critical considerations. This paper discusses all of these topics, identifying key challenges for future research and preliminary 5G standardization activities, while providing a comprehensive overview of the current literature, and in particular of the papers appearing in this special issue.
Conference Paper
Full-text available
System latency (time delay) and its visible consequences are fundamental virtual environment (VE) deficiencies that can hamper user perception and performance. The aim of this research is to quantify the role of VE scene content and resultant relative object motion on perceptual sensitivity to VE latency. Latency detection was examined by presenting observers in a head-tracked, stereoscopic head mounted display with environments having differing levels of complexity ranging from simple geometrical objects to a radiosity-rendered scene of two interconnected rooms. Latency discrimination was compared with results from a previous study in which only simple geometrical objects, without radiosity rendering or a 'real-world' setting, were used. From the results of these two studies, it can be inferred that the Just Noticeable Difference (JND) for latency discrimination by trained observers averages ~15 ms or less, independent of scene complexity and real-world meaning. Such knowledge will help elucidate latency perception mechanisms and, in turn, guide VE designers in the development of latency countermeasures.
Conference Paper
As an important component of the virtual reality (VR) technology, 360-degree videos provide users with panoramic view and allow them to freely control their viewing direction during video playback. Usually, a player displays only the visible portion of a 360 video. Thus, fetching the entire raw video frame wastes bandwidth. In this paper, we consider the problem of optimizing 360 video delivery over cellular networks. We first conduct a measurement study on commercial 360 video platforms. We then propose a cellular-friendly streaming scheme that delivers only 360 videos' visible portion based on head movement prediction. Using viewing data collected from real users, we demonstrate the feasibility of our approach, which can reduce bandwidth consumption by up to 80% based on a trace-driven simulation.
Conference Paper
Mobile Edge Computing proposes co-locating computing and storage resources at base stations of cellular networks. It is seen as a promising technique to alleviate utilization of the mobile core and to reduce latency for mobile end users. Due to the fact that Mobile Edge Computing is a novel approach not yet deployed in real-life networks, recent work discusses merely general and non-technical ideas and concepts. This paper introduces a taxonomy for Mobile Edge Computing applications and analyzes chances and limitations from a technical point of view. Application types which profit from edge deployment are identified and discussed. Furthermore, these applications are systematically classified based on technical metrics.
Article
3GPP LTE, or Long Term Evolution, the fourth generation wireless access technology, is being rolled out by many operators worldwide. Since LTE Release 10, network densification using small cells has been an important evolution direction in 3GPP to provide the necessary means to accommodate the anticipated huge traffic growth, especially for hotspot areas. Recently, LTE Release 12 has been started with more focus on small cell enhancements. This article provides the design principles and introduces the ongoing discussions on small cell enhancements in LTE Release 12, and provides views from two active operators in this area, CMCC and NTT DOCOMO.