Data pruning-based compression using high order edge-directed interpolation.
ABSTRACT This paper proposes a data pruning-based compression scheme to improve the rate-distortion relation of compressed images and video sequences. The original frames are pruned to a smaller size before compression. After decoding, they are interpolated to their original size by an edge-directed interpolation. The data pruning is optimized to obtain the minimal distortion in the interpolation phase. Furthermore, a novel high order interpolation is proposed to adapt the interpolation to many edge directions. This high order filtering uses extra surrounding pixels and achieves more robust edge-directed image interpolation. Simulation results are shown for both image interpolation and coding applications.
- SourceAvailable from: ucsd.edu
Conference Proceeding: Markov Random Field Model-Based Edge-Directed Image Interpolation.[show abstract] [hide abstract]
ABSTRACT: This paper presents an edge-directed image interpolation algorithm. In the proposed algorithm, the edge directions are implicitly estimated with a statistical-based approach. Consequently, the local edge directions are represented by length-16 vectors, which are denoted as weight vectors. The weight vectors are used to formulate geometric regularity constraint, which is imposed on the interpolated image through the Markov Random Field (MRF) model. Furthermore, the interpolation problem is formulated as a Maximum A Posterior (MAP)-MRF problem and, under the MAP-MRF framework, the desired interpolated image corresponds to the minimal energy state of a two-dimensional random held. Simulated Annealing method is used to search for the minimal energy state from a reasonable large state space. Simulation and comparison results show that the proposed MRF model-based edge-directed interpolation method produces edges with strong geometric regularity.Proceedings of the International Conference on Image Processing, ICIP 2007, September 16-19, 2007, San Antonio, Texas, USA; 01/2007
Conference Proceeding: POCS based adaptive image magnification[show abstract] [hide abstract]
ABSTRACT: We tackle the problem of magnifying an image without incurring blurring, ringing or other artifacts common to classical schemes. The proposed iterative scheme starts with an initial magnified image generated by a process of selective interpolation. By placing suitable constraints on the final magnified image, which are convex in nature, we show that magnification can be posed as a problem of finding a solution which lies at the intersection of convex sets. By avoiding explicit high frequency enhancing assumptions in the iterative process, we avoid edge enhancement artifacts in the magnified imageImage Processing, 1998. ICIP 98. Proceedings. 1998 International Conference on; 11/1998
- [show abstract] [hide abstract]
ABSTRACT: With the ever increasing computational power of modern day processors, it has become feasible to use more robust and computationally complex algorithms that increase the resolution of images without distorting edges and contours. We present a novel image interpolation algorithm that uses the new contourlet transform to improve the regularity of object boundaries in the generated images. By using a simple wavelet-based linear interpolation scheme as our initial estimate, we use an iterative projection process based on two constraints to drive our solution towards an improved high-resolution image. Our experimental results show that our new algorithm significantly outperforms linear interpolation in subjective quality, and in most cases, in terms of PSNR as well.Proc SPIE 01/2007; 6498.
DATA PRUNING-BASED COMPRESSION
USING HIGH ORDER EDGE-DIRECTED INTERPOLATION
D˜ ung Trung V˜ o1∗, Joel Sol´ e2, Peng Yin2, Cristina Gomila2, Truong Nguyen1
1Video Processing Laboratory, UC San Diego. 9300 Gilman Drive, La Jolla, CA 92092.
2Thomson Corporate Research. 2 Independence Way, Princeton, NJ 08540.
This paper proposes a data pruning-based compression scheme to
improve the rate-distortion relation of compressed images and video
sequences. The original frames are pruned to a smaller size before
compression. After decoding, they are interpolated to their original
sizebyan edge-directed interpolation. Thedatapruning isoptimized
to obtain the minimal distortion in the interpolation phase. Further-
more, a novel high order interpolation is proposed to adapt the inter-
polation to many edge directions. This high order ?ltering uses extra
surrounding pixels and achieves more robust edge-directed image
interpolation. Simulation results are shown for both image interpo-
lation and coding applications.
IndexTerms— interpolation, datapruning, compression, spa-
tial ?ltering, edge-directed interpolation, video coding.
Nowadays, the request for higher quality video is emerging very
fast. Video tends to higher resolution, higher frame-rate and higher
bit-depth. New technologies to further reduce bit-rate are strongly
demanded to combat the bit-rate increase of this high de?nition
video, especially to meet the network and communication transmis-
sion constraints. In video coding, there are two main directions to
reduce compression bit-rate. One is to improve the compression
technology and the other one is to perform a preprocessing before
The ?rst direction can be seen from the development of the
MPEG video coding standard, from MPEG-1 to H.264/MPEG-4
AVC. For most video coding standards, increasing quantization step
size is used to reduce bit-rate . However, this technique can result
in blocky artifacts and other coding artifacts due to the loss of high
frequency details. In the second direction, common techniques are
low-pass ?ltering or downsampling (which can be seen as a ?ltering
process) followed by reconstructing or upsampling at the decoder.
For example, low-pass ?lters were adaptively used based on Hu-
man Visual System to eliminate high frequency information in 
or to simplify the contextual information in . Also, to reduce
the bit-rate, some digital television systems uniformly downsized
the original sequence and upsized it after decoding. The recon-
structed video applying these techniques looked blur because they
were designed to eliminate high-frequency information with the
anti-aliasing ?lter before downsizing or with the low-pass ?lter in
the preprocessing step.
∗This work is done while D˜ ung Trung V˜ o was with Thomson Corporate
This paper proposes a novel data pruning-based compression
scheme to reduce the bit-rate while still keeping a high quality re-
constructed frame. The original frames are ?rst optimally pruned to
a smaller size by adaptively dropping rows or columns prior to en-
coding. At the ?nal stage, an interpolation phase is implemented to
reconstruct the decoded frames to their original size. By avoiding
?ltering the remaining rows and columns, the reconstructed frames
can achieve high quality from a lower bit-rate.
Main applications of interpolation are upsampling, demosaick-
ing and displaying in different video formats. A wide range of in-
terpolation methods has been discussed, starting from conventional
bilinear and bicubic interpolations to sophisticated iterative interpo-
lations such as projection onto convex sets (POCS)  and noncon-
vex nonlinear partial differential equations . Another group of in-
terpolation algorithms predicted the ?ne structure of the high resolu-
tion (HR) image from its low resolution (LR) version using different
kinds of transform such as wavelet  or contourlet transform .
To avoid the jerkiness artifacts occurring along edges, edge-oriented
interpolation methods were performed using Markov random ?eld
 or the LR image covariance .
All the above methods are for upsampling the same ratio in both
horizontal and vertical directions. However, when interpolation is
used along with data pruning, the method needs to adapt to the way
of pruning the data and to the structure of surrounding pixels. For
instance, thereare pruning cases inwhich only rows or only columns
are dropped and upsampling in only one direction is required. This
paper develops a high order edge-directed interpolation scheme to
deal with these cases. Another algorithm is also considered for the
cases of dropping both rows and columns. The paper is organized as
follows. Section II introduces the data pruning-based compression
method and derives an optimal data pruning algorithm. Section III
describes the high order edge-directed interpolation method. Results
for interpolation and coding applications are presented in Section IV.
Finally, Section V gives the concluding remarks.
2. OPTIMAL DATA PRUNING
The block diagram of the data pruning-based compression is shown
in Fig. 1. Assume that the original frame I of size M×N is pruned
to frame P of smaller size (M −Mp)×(N −Np), where Mp and
Np are the number of dropped rows and columns, respectively. The
purpose of data pruning is to reduce the number of bits representing
the stored or compressed frame P?. Then, I?of the original size is
obtained by interpolating P?.
In this paper, only the even rows and columns may be discarded,
while the odd rows and columns are always kept for later interpola-
tion. The block diagram of the data pruning phase is shown in Fig. 2.
997 978-1-4244-2354-5/09/$25.00 ©2009 IEEEICASSP 2009
( M− Mp)
× ( N− Np)
( M− Mp)
× ( N− Np)
Fig. 1. Block diagram of the data pruned-based compression.
Fig. 2. Block diagram of the data pruning phase.
To simplify the analysis, the compression stage in Fig. 1 is ignored.
In this phase, the original frame I is selectively decimated to the LR
frame Ilfor cases of dropping all even rows, all even columns and
all even rows and columns. Then, Ilis interpolated to the HR frame
Ih based on remaining odd rows and/ or columns after the selec-
tively decimate phase. Finally, Ihis compared to I to decide which
even rows and columns are dropped before compression. The mean
squared error (MSE) between Ihand I is de?ned as
Given a target MSEmax, the data pruning is optimized to discard
the maximum number of pixels while keeping the overall MSE of
Ihhaving dropped Mprows and Npcolumns less than MSEmax
MSE(Mp,Np) ≤ MSEmax
The location of the dropped rows and columns is indicated by
αm and αn, respectively. If the ktheven column is dropped, then
αn(k) = 1, otherwise αn(k) = 0. The similar algorithm is applied
to rows. These indicators are stored as side information in the coded
bitstream and are used for reconstructing the decoded frame. The
linemean square error (LMSE) for one dropped column isde?ned as
m = 1
with smaller LMSE have higher priority to be dropped than lines
with higher LMSE. Assume that the Mprows and Npcolumns with
smallest LMSE are dropped and the maximum LMSE of these lines
is LMSEmax. Then, the overall MSE in (1) becomes the averaged
MSE of all dropped pixels
n = 1
m = 1
n = 1
`I(m,n)−I?(m,n)´2and similarly for rows. From (2), lines
m = 1
n = 1
m = 1
Therefore, the condition in (2) can be tightened to
wherePSNRm i n is the target minimalPSNRthat the reconstructed
frame has to achieve. An example of the proposed optimal data
pruning is shown in Fig. 3 for the 1stframe of the sequence Akiyo.
In Fig. 3a, the white lines indicate the dropped lines. With target
PSNRm i n = 50 dB, the frame size is reduced from 720×480 to
464×320. The data pruned frame in Fig. 3b is more compact and
requires smaller compressed bitstream than the original frame. Most
dropped lines locate in ?at areas where aliasing does not happen.
LMSEm a x≤MSEm a x=
PSNRm i n
(a) Lines indicated for pruning
Fig. 3. Data pruning for the 1stframe of Akiyo sequence.
Fig. 4. Model parameters of high order edge-directed interpolation.
(b) Pruned frame
( i , 2 j )
( i?, 2 j?− 1 )
3. HIGH ORDER EDGE-DIRECTED INTERPOLATION
This section proposes a high order edge-directed interpolation
method to upsize the downsized frames Il in Fig. 1 and the data
pruned frames P?in Fig. 2. In , the fourth-order new edge-
directed interpolation (NEDI-4) is used to upsize only for the 2×2
ratio. This interpolation can only orient to edges in 2 directions and
causes some artifacts in the intersections of more than 2 edges. The
proposed method is a higher order interpolation that can adapt to
more edge directions. The sixth-order edge-directed interpolation
and eighth-order interpolation are developed for upsizing the cases
with ratio 1×2 (dropping only rows or only columns) and ratio 2×2
(dropping both rows and columns), respectively.
3.1. Sixth-order Edge-directed Interpolation (NEDI-6)
The algorithm for dropping only columns is presented here. A sim-
ilar algorithm is applied for the case of dropping only rows. The
block diagram of NEDI-6 is shown in Fig. 5. First, P?is expanded
to P??of size M ×N by inserting columns of zeros at the kthcol-
umn of P?if the column indicator αn(k)=1. P??is downsampled
by 1×2 ratio to form I?
mapped to the odd columns of the HR frame P?hof size M×N by
P?h(i,2j−1) = P?l(i,j). The even columns of P?
from the odd columns by a sixth-order interpolation
lof size M ×N
2. Then, columns of P?
k= − 1
l = 0
3l + k+ 1P?
where h6is the vector of sixth-order model parameters and P?
the vector of 6-neighboring pixels of P?h(i,2j) as shown in Fig. 4a.
Assuming that h6is nearly constant in a local window W, the op-
timal h6minimizing the MSE between the interpolatedˆP?h(i,2j)
and original pixels in W can be calculated by
h(i,2j) − h6P?
( M− Mp)
× ( N− Np)
Fig. 5. Block diagram of the NEDI-6.
The geometric duality assumption states that the model vector
h6can be considered constant for different scales and so, it can be
estimated from the LR pixels by
( i?,2 j?− 1 ) ∈W
is the LR model parameter vector, as shown in Fig. 4a. h6?is the
edge-directed weight at LR scale and is applied to HR scale for in-
terpolation. The optimal minimum MSE linear h is then obtained by
and A6is a 6×K matrix. The elements of the kthcolumn of A6are
the 6-neighboring pixels of LR pixels. Finally, the column indicator
αmdetermines whether the columns in the ?nal reconstructed frame
are selected from the interpolated or from the data prunted frame
3.2. Eighth-order Edge-directed Interpolation (NEDI-8)
This section develops an algorithm to deal with upsampling ratio
2×2. Similar to NEDI-6, the pixels in P?corresponding to the LR
pixels downsampling by 2×2 in I are extracted to form the LR frame
4 as in  for the ?rst round and NEDI-8 for the second round.
Using the quincunx sublattice, 2 passes are performed in the ?rst
round. In the ?rst pass, NEDI-4 is used to interpolate type 1 pixels
(squares with lines in Fig. 4b) from the LR pixels (solid circles). In
the second pass, type 2 pixels (squares) and type 3 pixels (circles)
are interpolated from type 1 pixels and LR pixels.
Having an initial estimation of all 8-neighboring pixels, NEDI-8
is implemented to get extra information from 4 directions in the sec-
ond round. The model parameters can be directly estimated from its
HR pixels in thisround. Therefore, the over?ttingproblem of NEDI-
4 is lessened while considering more edge orientations. For the sake
of interpolation consistency, NEDI-8 is applied to the pixels of type
3, 2, and 1 as in this order. The fourth-order model parameters h4
and eighth-order model parameters h8for HR scale are shown in
Fig. 4b. The optimal h8is similarly calculated by (8) where y is the
vector of all HR pixels in W and A8is a 8 ×K matrix whose kth
column is composed of the 8-neighboring pixels of HR pixels. The
?nal reconstructed frame is formed by
4. SIMULATION RESULTS
l6are 6-neighboring LR pixels of P?h(i?,2j?−1) and h6?
wherey isthevector of allK mapped LRpixelsP?h(i?,2j?−1)inW
2. The interpolation is performed using NEDI-
if αm(m)=0 and αn(n)=0
4.1. High order edge directed interpolation
Simulationsareperformed tocompare the proposed highorder edge-
directed interpolation with other interpolation methods for a wide
range of data in different formats. For the NEDI-6 case, original
frames are downsampled by 2 in the horizontal direction. The down-
sized frames are then interpolated using bicubic, sinc, and the pro-
posed NEDI-6 interpolation. Note that other interpolation methods,
(b) Bicubic (c) Sinc(d) Nedi-6
Fig. 6. Comparison of NEDI-6 to other methods.
Fig. 7. Comparison of NEDI-8 to other methods.
such as , can only be applied in cases of downsampling by ratio
of 2 in both directions. A particular result is shown in Fig. 6 for a
zoomed part of the 1stframe of the Foreman sequence. The PSNR
values of the interpolated frames using bicubic, sinc and NEDI-6
interpolation are 38.62 dB, 38.56 dB and 39.22 dB, respectively.
These results validate the effectiveness of NEDI-6 for edge-directed
interpolation, since less jerkiness and higher PSNR is attained com-
pared to the other methods.
For the NEDI-8 case, the comparison is performed for bicubic,
sinc, NEDI-4 and the proposed NEDI-8. To enhance the pixels near
the frame borders in the proposed NEDI-8, the frame is expanded by
re?ecting these pixels over the borders. PSNR values are shown in
Table 1 for sequences in different formats. To perform a fair com-
parison to other methods that use bilinear interpolation for pixels
near the borders, pixels at 5 lines or fewer away from the border
are not counted for the PSNR computation. The visual results for a
selected part of Foreman sequence are shown in Fig. 7. As shown
in Fig. 7b, the result using the sinc-based interpolation has a lot of
jerkiness. While the NEDI-4 interpolation has signi?cant less jerki-
ness, the interpolated frame in Fig. 7c still shows jerkiness along the
strong edges. Because NEDI-4 only uses pixels of 2 directions, arti-
facts can be observed at the intersections of more than 2 edges. On
the other hand, the NEDI-8 interpolated frame in Fig. 7d achieves
the best quality with least jerkiness. Using pixels in 4 directions,
the NEDI-8 interpolation also has less artifacts at the intersection of
more than 2 edges. With respect to objective quality, the proposed
NEDI-8 has the highest PSNR values for all the sequences across
4.2. Data pruning-based compression
The data pruning approach is applied to video compression. An ex-
periment is performed in which a GOP of 15 frames of Akiyo are
pruned with the target PSNRmin= 45 dB. As a consequence, the
frame size is reduced from 720×480 to 480×320 lines. An H.264
codec is applied with the IPPP GOP structure and QI =QP −1=
[12,40]. The LMSE is averaged over the whole GOP, so that the
Table 1. PSNR comparison (in dB)
Sequences FormatBicubicSinc Nedi-4
29.63 29.37 29.60
34.70 34.59 35.15
32.262 32.118 32.584
0 5001000 1500
Sinc data pruning?based
Optimal data pruning?based
(a) Whole R-D curves
4042 44 46 48
50 52 54 5658
Sinc data pruning?based
Optimal data pruning?based
(b) One zoomed in part
Fig. 8. Comparison results for R-D curves.
same lines are dropped for all the frames. In this way, the side in-
formation to determine the dropped lines is greatly reduced. The
extra bit-rate is 1.2 Kbps for the whole GOP, which is very small
compared to the total bit-rate of the compressed bitstream. For com-
parison, the data pruning scheme is applied to the sequence down-
and up-sized by 2×2 with the uniform sinc interpolation.
The rate-distortion (R-D) curves are shown in Fig. 8a (the whole
curves) and in Fig. 8b (one zoomed in part). The results show that
the R-D curve of the sinc data-pruned method is lower than the
R-D curve of the optimal data pruning-based compressed sequence.
Comparing to the R-D curve of the H.264/AVC compressed se-
quence, the one of the proposed method is better in the range of
PSNR 32-37.5 dB. In this range, the PSNR improvement at the
same bit-rate is around 0.3-0.7 dB.
Even having the same bit-rate and PSNR values, the recon-
structed frames have less artifacts because they are compressed
with smaller quantization step level QI and QP. Fig. 9 shows
the comparison between the H.264/AVC compressed frame and the
optimal data pruning-based compressed frame at the quantization
level of 35 and 32, respectively. These sequences have nearly same
bit-rate of 92 Kbps and 94 Kbps and same PSNR of 37.83 dB and
37.81 dB respectively for the H.264/AVC and the proposed data
pruning-based compressed sequences. Results show that the pro-
posed data pruning-based compressed frame in Fig. 9b has higher
visual quality and less artifacts than the H.264/AVC compressed
frame in Fig. 9a. This merit can be explained by the interpolation
phase, which helps reducing the blocking and ringing artifacts, and
the smaller quantization step level. The percentage of bit-rate saving
of the optimal data pruning-based compressed sequence is 23-36%
comparing to the H.264/AVC compressed sequence for the cases of
using the same quantization step size.
Both PSNR curve and visual results validate the effectiveness of
the proposed data pruning-based compression. The proposed algo-
tion phases, so the complexity of data pruning-based compression is
higher than the normal compression. But the coding and decoding
time of the proposed method decreases proportionally to the size re-
(b) Optimal data pruning-based
Fig. 9. Comparison for H.264/AVC compression and optimal data
pruning-based compression with same bit-rate and PSNR values.
duction of thedata pruned frame. All simulation resultscan be found
The paper proposed anovel data pruning-based compression method
to reduce the bit-rate. High order edge-directed interpolations are
also discussed to include more surrounding pixels and to adapt to
different data pruning schemes. The results show that these high
order edge-directed interpolation methods help to reduce the jerk-
iness along strong edges and the artifacts at the intersection areas.
The NEDI-6 for upsamling only rows can be also applied for de-
interleaving. In a future work, instead of using only the pixels at
odd indices, high order edge-directed interpolation methods may use
more available pixels to estimate more accurately the model param-
eters. Additionally, the objective function in (2) may be extended
to consider the coding ef?ciency of dropping these pixels to further
improve the R-D curve.
 “Advanced Video Coding for Generic Audiovisual Services,”
 N. Vasconcelos and F. Dufaux, “Pre and Post-?ltering for Low
Bit-rate Video Coding,” IEEE Conf. on Image Process., vol. 1,
pp. 291–294, Oct. 1997.
 A. Cavallaro, O. Steiger, and T. Ebrahimi, “Perceptual Pre?lter-
ing for Video Coding,” IEEE Int. Symposium Intel. Multimedia,
Video and Speech Processing, pp. 510–513, Oct. 2004.
 K. Ratakonda and N. Ahuja, “POCS Based Adaptive Image
Magni?cation,” IEEE Conf. on Image Process., vol. 3, pp. 203–
207, Oct. 1998.
 Y. Cha and S. Kim, “Edge-forming Methods for Color Image
Zooming,” IEEETrans. ImageProcess., vol. 15, pp. 2315–2323,
 N. Mueller, Y. Lu, and M. N. Do, “Image Interpolation using
Multiscale Geometric Representations,” Proc. of SPIE Conf. on
Electronic Imaging, vol. 6498, Feb. 2007.
 N. Mueller and T. Q. Nguyen, “Image Interpolation using Clas-
si?cation and Stitching,” IEEE Conf. on Image Process., Oct.
 M. Li and T. Q. Nguyen, “Markov Random Field Model-based
Edge-directed Image Interpolation,” IEEE Trans. Image Pro-
cess., vol. 17, pp. 1121–1128, July 2008.
 X. Li and M. T. Orchard, “New Edge-directed Interpolation,”
IEEE Trans. Image Process., vol. 10, pp. 1521–1527, Oct. 2001.