ChapterPDF Available

Seam Carving Detection and Localization Using Two-Stage Deep Neural Networks

Authors:

Abstract and Figures

Seam carving is a method to resize an image in a content-aware fashion. However, this method can also be used to carve out objects from images. In this paper, we propose a two-step method to detect and localize seam carved images. First, we build a detector to detect small patches in an image that has been seam carved. Next, we compute a heatmap on an image based on the patch detector’s output. Using these heatmaps, we build another detector to detect if a whole image is seam carved or not. Our experimental results show that our approach is effective in detecting and localizing seam carved images.
Content may be subject to copyright.
Seam Carving Detection and Localization using
Two-Stage Deep Neural Networks
Lakshmanan Nataraj, Chandrakanth Gudavalli, Tajuddin Manhar Mohammed,
Shivkumar Chandrasekaran, and B.S. Manjunath
Mayachitra Inc., Santa Barbara, CA, USA,
https://mayachitra.com/
Abstract. Seam carving is a method to resize an image in a content
aware fashion. However, this method can also be used to carve out objects
from images. In this paper, we propose a two-step method to detect
and localize seam carved images. First, we build a detector to detect
small patches in an image that has been seam carved. Next, we compute
a heatmap on an image based on the patch detector’s output. Using
these heatmaps, we build another detector to detect if a whole image is
seam carved or not. Our experimental results show that our approach is
effective in detecting and localizing seam carved images.
Keywords: image forensics, seam carving detection, fake images, object
removal
1 Introduction
With new cameras, mobile phones and digital tablets, the amount of digital
images has had an exponential increase. Social media platforms have also con-
tributed to their increased distribution. At the same time, software for manipu-
lating these digital images have also significantly evolved. These software tools
make it trivial for people to manipulate these digital images. The objective of Me-
dia Forensics is to identify these manipulations and detect these doctored images.
Over the years, many techniques have been proposed to identify image manip-
ulations. These include digital artificats based on camera forensics, resampling
characteristics, compression, and others. A common operation in image tamper-
ing is removing certain image regions in a “content-aware” way. In this regard,
seam carving is a popular technique for “content-aware” image resizing [1, 39] In
seam carving, the “important content” in an image is left unaffected when the
image is resized and it is generally assumed that the “important content” is not
characterized by the low energy pixels. Since seam carving based object removal
involves non-traditional ways of removing objects, it is a challenge to detect
doctored images that have been seam carved. In this paper, we propose a novel
method to detect and localize seam carved images using two stages of convolu-
tional neural networks (CNNs): one for detection and one for localization. First,
we train a CNN to identify patches that have been seam carved. For every pixel
in an image, we then compute the detection score which results in a heatmap
2 Lakshmanan Nataraj et al.
Fig. 1: Illustration of seam carving detection and localization: (a) Original im-
age, (b) Object marked in red to be removed and Object marked in green to
be preserved, (c) Seam carved image with object removed, (d) Seam carving
detection heatmap using proposed approach (red pixels are areas where seams
were likely removed).
for the whole image, that can be used for localization. Finally, we train another
CNN on the heatmaps which gives a score at the image level to determine if an
image has been seam carved or not. Fig. 1 illustrates the proposed approach.
The rest of the paper is organized as follows. Sec. 2 presents the related
work in this area and Sec. 3 introduces seam carving and seam insertion on
images. The methodology to detect seam carving is presented in Sec. 4 while the
experiments are detailed in Sec. 5. Finally, the conclusion is presented in Sec. 6.
2 Related Work
There have been several works proposed to detect digital image manipulations.
These include detection of splicing, morphing, resampling artifacts, copy-move,
seam carving, computer-generated (CG) images, JPEG artifacts, inpainting,
compression artifacts, to name a few. Many methods have been proposed to de-
tect copy-move [21, 11], resampling [34, 29, 20, 33, 36, 15, 6, 8], splicing [18, 2, 37],
and inpainting based object removal [44,23]. Other approaches exploit JPEG
compression artifacts [13, 24, 28, 7] or artifacts arising from artificial intelligence
(AI) generated images [30, 48, 16, 3]. In recent years, deep learning based meth-
ods have shown better performance in detecting image manipulations [5, 4, 35,
8].
Several methods have been proposed over the past decade to detect seam
carving based manipulations [38, 27, 26, 9, 43, 25, 42, 14, 40, 46, 41, 19, 17, 47, 22].
These include methods using steganalysis [38], hashing [27, 14], local binary pat-
tern [46, 47], and deep learning based methods [45, 10, 31, 32]. In this paper, our
approach to detect seam carving based manipulations is also based on deep
learning.
Seam Carving Detection 3
a1,1 a1,2 a1,5
a2,1 a2,5
a4,1 a4,2 a4,5
a2,2
seam carving path
b1,1 b1,2
b2,1
b4,1 b4,2
b2,2
b4,3
b3,3
b4,4
b1,3
b2,4
b1,4
b3,4
(i) before seam carving (ii) after seam carving
a1,3
a2,4
a3,3
b2,3
At these locations, ai,j = bi,j At these locations, ai,j+1 = bi,j
Fig. 2: Example of Seam Carving when a 4 ×5 matrix ais seam carved and a
4×4 matrix bresults due to the removal of a single seam.
3 Seam Carving and Seam Insertion
A seam is defined as an optimal 8-connected path of pixels on an image either
from top-to-bottom or left-to-right. In seam carving, the seams are removed
and the image dimension is reduced by a column or a row. In seam insertion, a
seam is first removed and two pixels are inserted at the position where the seam
was removed. Fig. 2 and 3 illustrates the processes of seam carving and seam
insertion. An energy function computed for all points along a seam is considered
for the optimality criterion for seam selection. This choice of seams helps in
maintaining the image quality during the resizing process. We consider the seam
carved/inserted image as a tampered image because the image dimensions and
it’s content are altered.. Hence, the problem of detecting seam carving/insertion
is important from an image forensics perspective. Interpolation kernel based
methods for re-sampling detection will fail when the resizing in the doctored
image is done using seam carving/insertion. Though it was initially proposed
for automatic image resizing while maintaining a good perceptual quality of
a1,1 a1,2 a1,5
a2,1 a2,5
a4,1 a4,2 a4,5
a2,2
seam insertion path
b1,1 b1,2 b1,6
b2,1 b2,6
b4,1 b4,2 b4,6
b2,2
pixels are changed on either side of seam
insertion path
b4,3
b3,3
b4,4
b1,3
b2,4
b1,4
b2,5
b3,4
(i) before seam insertion (ii) after seam insertion
a1,3
a2,4
a3,3
Fig. 3: Example of Seam Insertion: (i) aand (ii) bare the 4 ×5 and the 4 ×6
image matrices before and after seam insertion, respectively. For points along
the seam, the values are modified as shown for the first row: b1,1=a1,1, b1,2=
a1,2, b1,3=round(a1,2+a1,3
2), b1,4=round(a1,3+a1,4
2), b1,5=a1,4, b1,6=a1,5.
4 Lakshmanan Nataraj et al.
(a) Stage 1
(b) Stage 2
Fig. 4: Two-stage approach
the resized image, seam carving has also been used for removal of certain image
regions. It is to be noted that seam carving can discard and retain certain regions,
depending on the weight we assign to certain regions. E.g. for an object removal
problem, we may need to ensure that certain image regions are left unaffected
as distorting them may cause significant perceptual distortion. We first explain
how seam carving is used for object removal and then discuss the interesting
problems involved.
4 Detection of Seam Carving
In order to detect and localize seam carving in images, we propose a two-stage
detection approach: one for detection of seam carved patches and the other for
localizing seam carving in an image by generating a heatmap. First, we train a
deep neural network to identify whether patches in an image have been seam
carved or not. We then divide an image into patches and for every patch, we
Seam Carving Detection 5
Fig. 5: Screenshot of: (a) Non-seam carved patches, (b) Seam carved patches
compute the detection score which results in a heatmap for the whole image.
This heatmap can be used for localization of seam carving. Finally, we train
another deep neural network with the heatmaps as input which gives a score at
the image level to determine whether an image has been seam carved or not.
The entire block schematic is shown in Fig. 4
5 Experiments
5.1 Experimental Setup
We first extract 64 ×128 patches from images belonging to RAISE dataset [12].
From these patches, we form two classes of image patches: first class where the
patches are further cropped to 64 ×64, and the second class where the patches
are seam carved horizontally by 50% to obtain 64 ×64 seam carved patches. In
this way we obtained 16,000 patches from the RAISE dataset (8000 in each class)
and 40,000 patches from the Dresden dataset (20,000 in each class). These were
further randomly divided into 80% training, 10% testing and 10% validation.
5.2 Learning
The patches are trained using a multi layer deep convolutional neural network
which consists of convolution layer with 32 3x3 convs, followed by ReLu layer,
convolution layer with 32 5x5 convs followed by max pooling layer, convolu-
tion layer with 64 3x3 convs followed by ReLu layer, convolution layer with 64
5x5 convs followed by max pooling layer, convolution layer with 128 3x3 convs
followed by ReLu layer, convolution layer with 128 5x5 convs followed by max
pooling layer, and finally a 256 dense layer followed by a 256 dense layer and a
sigmoid layer. We train this model till a high training accuracy and validation
accuracy are obtained.
6 Lakshmanan Nataraj et al.
Fig. 6: Detection heatmaps on images that have (a) not been seam carved, and
(b) seam carved
5.3 Detection Heatmaps
Using the trained model on the patches, the probability of a pixel being seam
carved or not is computed on overlapping patches in an image. Fig. 6 show
the heatmaps on non-seam carved and seam carved images. As we can see, the
heatmaps on the seam carved images have more red regions than the images on
non-seam carved images. Even for an image that has the blue sky, the heatmaps
can be clearly identified for seam carved image and the non-seam carved image.
This motivated us to train the heatmaps with another CNN which takes the
heatmaps as input (Fig. 4(b)) and outputs the probability whether an image
has been seam carved or not. As we can see from Fig. 7, we obtained high
accuracy when trained on the heatmaps.
5.4 Robustness to Percentage of seams removed
In this experiment, we varied the percentage of seams removed in the testing set
and evaluated the model which was trained with 50% seams removed, in order to
check the robustness of the model for different amounts of seams removed. The
Area Under the Curve (AUC) is the evaluation metric for varying percentage of
seams removed. The results are tabulated in Tab. 1. We observe that the AUC
is very high for percentages around 50% and decreases for lower percentages of
seams removed. This shows that the model is generalizable for most percentages
of seams removed. In future, we will train another model for lower percentages.
Seam Carving Detection 7
Fig. 7: ROC curve of seam carving detection on the model trained on the
heatmaps
Table 1: Robustness to Percentage of Seams Removed
Percentage Area under the curve (AUC)
1 0.6464
2 0.7838
5 0.9274
8 0.9540
10 0.9724
20 0.9866
30 0.9919
40 0.9937
50 0.9916
60 0.9502
70 0.9150
80 0.8670
90 0.8223
5.5 Robustness to JPEG compression
In this experiment, we evaluated the robustness of our proposed against JPEG
compression. We varied the JPEG Quality Factors (QFs) of test images from
100 to 50. The model was trained on seam carved and non-seam carved patches
and images, which were also JPEG compressed between the Quality Factors of
70-100. The Area Under the Curve (AUC) is chosen as the evaluation metric.
The results are tabulated in Tab. 2. We observe that the AUC is high when
the QF is high (compression is low) and the AUC reduces as the QF decreases
(compression increases). However, even at a QF of 50, the AUC is still reasonably
high.
8 Lakshmanan Nataraj et al.
Fig. 8: Detection heatmaps on images where objects have been removed using
seam carving: (a) original image, (b) heatmap computed on original image, (c)
object marked for removal in red, (d) image with object removed using seam
carving, (e) heatmap computed on object removed image showing the possible
seam paths
5.6 Explainability on Object Removed Images
Here, we evaluate our approach in a practical scenario where objects are removed
in images using seam carving. We chose an object or a region in an image that
has to be removed. The weights of this region are set to a low value such that
the seam carving algorithm is forced to pass through this region, thus removing
the object from the image. When our approach was evaluated on these images,
we observe that our model is able to localize the region that was removed as well
as the paths taken by the seam carving algorithm as shown in Fig. 8.
The detection heatmaps also exhibit explainability as shown in Fig. 9 where
an object is marked for removal in red. While this object is removed successfully,
a person’s leg in the foreground also gets removed (top row). To prevent this,
another area is marked in green (bottom row) by giving high weights so that the
person’s legs are not removed. As we can see from the heatmaps computed on
the seam carved images (top and bottom row), the path showing the possible
seams also changes near the person’s legs, thus exhibiting explainability.
Seam Carving Detection 9
Fig. 9: Explainability in the heatmaps: (a) original image, (b) heatmap computed
on original image, (c) object marked for removal in red and area preserved in
green (bottom row) (d) image with object removed using seam carving (in the
top row - person’s leg is removed while preserved in the bottom row), (e) heatmap
computed on object removed image showing the possible seam paths with ex-
plainability. The seam paths change on the top row and bottom row near the
person’s leg.
5.7 Extension to Seam Insertion Detection
Finally, we also extend the seam carving detection approach to detecting seam
insertion. We first extract 64 ×64 patches from images belonging to RAISE
dataset [12]. From these patches, we form two classes of image patches: first
class where the patches are further cropped to 64 ×64, and the second class
where the patches are seam inserted from 64 ×32 dimensions to 64 ×64 seam
dimensions. In this way we obtained 16,000 patches from the RAISE dataset
(8000 in each class). These were further randomly divided into 80% training,
10% testing and 10% validation. The patches are trained using a multi layer
convolutional neural network as explained in Sec. 5.2. We train this model till a
high training accuracy and validation accuracy are obtained. Using the trained
model on the patches, the probability of a pixel being seam inserted or not is
computed on overlapping patches in an image to produce a heatmap. Another
model is trained on the heatmaps to determine if an image has seam insertions
or not. As we can see from Fig. 10, we obtained high accuracy when trained on
the heatmaps.
10 Lakshmanan Nataraj et al.
Fig. 10: ROC curves at image level to detect seam inserted images
6 Conclusion and Future Work
In this paper, we presented an approach to detect seam carved images. Using
two stages of CNNs, we detect and localize areas in an image that have been
seam carved. In future, we will focus on making our detections more robust,
combining seam carving and insertions, and also extend to other object removal
methods such as inpainting.
7 Acknowledgements
This research was developed with funding from the Defense Advanced Research
Projects Agency (DARPA). The views, opinions and/or findings expressed are
those of the author and should not be interpreted as representing the official
views or policies of the Department of Defense or the U.S. Government. The
paper is approved for public release, distribution unlimited.
Table 2: Robustness to JPEG Compression
JPEG Quality Factor (QF) Area under the curve (AUC)
100 0.9376
90 0.9160
80 0.8578
70 0.8027
60 0.7658
50 0.7332
Seam Carving Detection 11
References
1. Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. ACM
Trans. Graph. 26(3), 10 (2007)
2. Bappy, J.H., Simons, C., Nataraj, L., Manjunath, B., Roy-Chowdhury, A.K.: Hy-
brid lstm and encoder–decoder architecture for detection of image forgeries. IEEE
Transactions on Image Processing 28(7), 3286–3300 (2019)
3. Barni, M., Kallas, K., Nowroozi, E., Tondi, B.: Cnn detection of gan-
generated face images based on cross-band co-occurrences analysis. arXiv preprint
arXiv:2007.12909 (2020)
4. Bayar, B., Stamm, M.C.: Design principles of convolutional neural networks for
multimedia forensics. In: The 2017 IS&T International Symposium on Electronic
Imaging: Media Watermarking, Security, and Forensics. IS&T Electronic Imaging
5. Bayar, B., Stamm, M.C.: A deep learning approach to universal image manipula-
tion detection using a new convolutional layer. In: Proceedings of the 4th ACM
Workshop on Information Hiding and Multimedia Security. pp. 5–10 (2016)
6. Bayar, B., Stamm, M.C.: On the robustness of constrained convolutional neural
networks to jpeg post-compression for image resampling detection. In: The 42nd
IEEE International Conference on Acoustics, Speech and Signal Processing (2017)
7. Bianchi, T., De Rosa, A., Piva, A.: Improved dct coefficient analysis for forgery
localization in jpeg images. In: Acoustics, Speech and Signal Processing (ICASSP),
2011 IEEE International Conference on. pp. 2444–2447. IEEE (2011)
8. Bunk, J., Bappy, J.H., Mohammed, T.M., Nataraj, L., Flenner, A., Manjunath, B.,
Chandrasekaran, S., Roy-Chowdhury, A.K., Peterson, L.: Detection and localiza-
tion of image forgeries using resampling features and deep learning. In: Computer
Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on.
pp. 1881–1889 (2017)
9. Chang, W.L., Shih, T.K., Hsu, H.H.: Detection of seam carving in jpeg images.
In: 2013 International Joint Conference on Awareness Science and Technology &
Ubi-Media Computing (iCAST 2013 & UMEDIA 2013). pp. 632–638. IEEE (2013)
10. Cieslak, L.F.S., Da Costa, K.A., PauloPapa, J.: Seam carving detection using con-
volutional neural networks. In: 2018 IEEE 12th International Symposium on Ap-
plied Computational Intelligence and Informatics (SACI). pp. 000195–000200. Ieee
(2018)
11. Cozzolino, D., Poggi, G., Verdoliva, L.: Efficient dense-field copy–move forgery
detection. IEEE Transactions on Information Forensics and Security 10(11), 2284–
2297 (2015)
12. Dang-Nguyen, D.T., Pasquini, C., Conotter, V., Boato, G.: Raise: a raw images
dataset for digital image forensics. In: Proceedings of the 6th ACM Multimedia
Systems Conference. pp. 219–224. ACM (2015)
13. Farid, H.: Exposing digital forgeries from jpeg ghosts. IEEE transactions on infor-
mation forensics and security 4(1), 154–160 (2009)
14. Fei, W., Gaobo, Y., Leida, L., Ming, X., Dengyong, Z.: Detection of seam carving-
based video retargeting using forensics hash. Security and Communication Net-
works 8(12), 2102–2113 (2015)
15. Feng, X., Cox, I.J., Doerr, G.: Normalized energy density-based forensic detection
of resampled images. IEEE Transactions on Multimedia 14(3), 536–545 (2012)
16. Goebel, M., Nataraj, L., Nanjundaswamy, T., Mohammed, T.M., Chandrasekaran,
S., Manjunath, B.: Detection, attribution and localization of gan generated images.
arXiv preprint arXiv:2007.10466 (2020)
12 Lakshmanan Nataraj et al.
17. Gong, Q., Shan, Q., Ke, Y., Guo, J.: Detecting the location of seam and recovering
image for seam inserted image. Journal of Computational Methods in Sciences and
Engineering 18(2), 499–509 (2018)
18. Guillemot, C., Le Meur, O.: Image inpainting: Overview and recent advances. Sig-
nal processing magazine 31(1), 127–144 (2014)
19. Han, R., Ke, Y., Du, L., Qin, F., Guo, J.: Exploring the location of object deleted
by seam-carving. Expert Systems with Applications 95, 162–171 (2018)
20. Kirchner, M.: On the detectability of local resampling in digital images. Security,
Forensics, Steganography, and Watermarking of Multimedia Contents X 6819(1),
68190F (2008), http://link.aip.org/link/?PSI/6819/68190F/1
21. Li, J., Li, X., Yang, B., Sun, X.: Segmentation-based image copy-move forgery
detection scheme. IEEE Transactions on Information Forensics and Security 10(3),
507–518 (2015)
22. Li, Y., Xia, M., Liu, X., Yang, G.: Identification of various image retargeting tech-
niques using hybrid features. Journal of Information Security and Applications 51,
102459 (2020)
23. Liang, Z., Yang, G., Ding, X., Li, L.: An efficient forgery detection algorithm for
object removal by exemplar-based image inpainting. Journal of Visual Communi-
cation and Image Representation 30, 75–85 (2015)
24. Lin, Z., He, J., Tang, X., Tang, C.K.: Fast, automatic and fine-grained tampered
jpeg image detection via dct coefficient analysis. Pattern Recognition 42(11), 2492–
2501 (2009)
25. Liu, Q., Chen, Z.: Improved approaches with calibrated neighboring joint density to
steganalysis and seam-carved forgery detection in jpeg images. ACM Transactions
on Intelligent Systems and Technology (TIST) 5(4), 1–30 (2014)
26. Liu, Q., Cooper, P.A., Zhou, B.: An improved approach to detecting content-
aware scaling-based tampering in jpeg images. In: 2013 IEEE China Summit and
International Conference on Signal and Information Processing. pp. 432–436. IEEE
(2013)
27. Lu, W., Wu, M.: Seam carving estimation using forensic hash. In: Proceedings of
the thirteenth ACM multimedia workshop on Multimedia and security. pp. 9–14
(2011)
28. Luo, W., Huang, J., Qiu, G.: Jpeg error analysis and its applications to digital
image forensics. IEEE Transactions on Information Forensics and Security 5(3),
480–491 (2010)
29. Mahdian, B., Saic, S.: Blind authentication using periodic properties of interpo-
lation. Information Forensics and Security, IEEE Transactions on 3(3), 529–538
(Sept 2008)
30. Marra, F., Gragnaniello, D., Cozzolino, D., Verdoliva, L.: Detection of gan-
generated fake images over social networks. In: 2018 IEEE Conference on Mul-
timedia Information Processing and Retrieval (MIPR). pp. 384–389. IEEE (2018)
31. Nam, S.H., Ahn, W., Mun, S.M., Park, J., Kim, D., Yu, I.J., Lee, H.K.: Content-
aware image resizing detection using deep neural network. In: 2019 IEEE Interna-
tional Conference on Image Processing (ICIP). pp. 106–110. IEEE (2019)
32. Nam, S.H., Ahn, W., Yu, I.J., Kwon, M.J., Son, M., Lee, H.K.: Deep con-
volutional neural network for identifying seam-carving forgery. arXiv preprint
arXiv:2007.02393 (2020)
33. Nataraj, L., Sarkar, A., Manjunath, B.S.: Improving re-sampling detection by
adding noise. In: SPIE, Media Forensics and Security. vol. 7541 (2010), http:
//vision.ece.ucsb.edu/publications/lakshman spie 2010.pdf
Seam Carving Detection 13
34. Popescu, A.C., Farid, H.: Exposing digital forgeries by detecting traces of resam-
pling. IEEE Transactions on signal processing 53(2), 758–767 (2005)
35. Rao, Y., Ni, J.: A deep learning approach to detection of splicing and copy-move
forgeries in images. In: Information Forensics and Security (WIFS), 2016 IEEE
International Workshop on. pp. 1–6. IEEE (2016)
36. Ryu, S.J., Lee, H.K.: Estimation of linear transformation by analyzing the period-
icity of interpolation. Pattern Recognition Letters 36, 89–99 (2014)
37. Salloum, R., Ren, Y., Kuo, C.C.J.: Image splicing localization using a multi-task
fully convolutional network (mfcn). Journal of Visual Communication and Image
Representation 51, 201–209 (2018)
38. Sarkar, A., Nataraj, L., Manjunath, B.S.: Detection of seam carving and localiza-
tion of seam insertions in digital images. In: Proceedings of the 11th ACM workshop
on Multimedia and security. pp. 107–116. ACM (2009)
39. Shamir, A., Avidan, S.: Seam carving for media retargeting. Communications of
the ACM 52(1), 77–85 (2009)
40. Sheng, G., Gao, T.: Detection of seam-carving image based on benford’s law for
forensic applications. International Journal of Digital Crime and Forensics (IJDCF)
8(1), 51–61 (2016)
41. Sheng, G., Li, T., Su, Q., Chen, B., Tang, Y.: Detection of content-aware image
resizing based on benford’s law. Soft Computing 21(19), 5693–5701 (2017)
42. Wattanachote, K., Shih, T.K., Chang, W.L., Chang, H.H.: Tamper detection of
jpeg image due to seam modifications. IEEE transactions on information forensics
and security 10(12), 2477–2491 (2015)
43. Wei, J.D., Lin, Y.J., Wu, Y.J., Kang, L.W.: A patch analysis approach for seam-
carved image detection. In: ACM SIGGRAPH 2013 Posters, pp. 1–1 (2013)
44. Wu, Q., Sun, S.J., Zhu, W., Li, G.H., Tu, D.: Detection of digital doctoring in
exemplar-based inpainted images. In: Machine Learning and Cybernetics, 2008
International Conference on. vol. 3, pp. 1222–1226. IEEE (2008)
45. Ye, J., Shi, Y., Xu, G., Shi, Y.Q.: A convolutional neural network based seam carv-
ing detection scheme for uncompressed digital images. In: International Workshop
on Digital Watermarking. pp. 3–13. Springer (2018)
46. Zhang, D., Li, Q., Yang, G., Li, L., Sun, X.: Detection of image seam carving
by using weber local descriptor and local binary patterns. Journal of information
security and applications 36, 135–144 (2017)
47. Zhang, D., Yang, G., Li, F., Wang, J., Sangaiah, A.K.: Detecting seam carved
images using uniform local binary patterns. Multimedia Tools and Applications
79(13), 8415–8430 (2020)
48. Zhang, X., Karaman, S., Chang, S.F.: Detecting and simulating artifacts in gan
fake images. In: 2019 IEEE International Workshop on Information Forensics and
Security (WIFS). pp. 1–6. IEEE (2019)
... One may reapply seam carving to enlarge the image to its original size. Seam carving may highlight some texture artifacts in the resulting image as well [10,15,16], which can be considered for further automatic detection. Although these artifacts are mostly imperceptible to the human eye, they can be detected by computer vision techniques. ...
Chapter
Seam carving is a computational method capable of resizing images for both reduction and expansion based on its content, instead of the image geometry. Although the technique is mostly employed to deal with redundant information, i.e., regions composed of pixels with similar intensity, it can also be used for tampering images by inserting or removing relevant objects. Therefore, detecting such a process is of extreme importance regarding the image security domain. However, recognizing seam-carved images does not represent a straightforward task even for human eyes, and robust computation tools capable of identifying such alterations are very desirable. In this paper, we propose an end-to-end approach to cope with the problem of automatic seam carving detection that can obtain state-of-the-art results. Experiments conducted over public and private datasets with several tampering configurations evidence the suitability of the proposed model. KeywordsSeam carvingConvolutional neural networksImage security
... One may reapply seam carving to enlarge the image to its original size. Seam carving may highlight some texture artifacts in the resulting image as well [16,10,15], which can be considered for further automatic detection. Although these artifacts are mostly imperceptible to the human eye, they can be detected by computer vision techniques. ...
Preprint
Full-text available
Seam carving is a computational method capable of resizing images for both reduction and expansion based on its content, instead of the image geometry. Although the technique is mostly employed to deal with redundant information, i.e., regions composed of pixels with similar intensity, it can also be used for tampering images by inserting or removing relevant objects. Therefore, detecting such a process is of extreme importance regarding the image security domain. However, recognizing seam-carved images does not represent a straightforward task even for human eyes, and robust computation tools capable of identifying such alterations are very desirable. In this paper, we propose an end-to-end approach to cope with the problem of automatic seam carving detection that can obtain state-of-the-art results. Experiments conducted over public and private datasets with several tampering configurations evidence the suitability of the proposed model.
... One may reapply seam carving to enlarge the image to its original size. Seam carving may highlight some texture artifacts in the resulting image as well [16,10,15], which can be considered for further automatic detection. Although these artifacts are mostly imperceptible to the human eye, they can be detected by computer vision techniques. ...
Conference Paper
Seam carving is a computational method capable of resizing images for both reduction and expansion based on its content, instead of the image geometry. Although the technique is mostly employed to deal with redundant information, i.e., regions composed of pixels with similar intensity, it can also be used for tampering images by inserting or removing relevant objects. Therefore, detecting such a process is of extreme importance regarding the image security domain. However, recognizing seam-carved images does not represent a straightforward task even for human eyes, and robust computation tools capable of identifying such alterations are very desirable. In this paper, we propose an end-to-end approach to cope with the problem of automatic seam carving detection that can obtain state-of-the-art results. Experiments conducted over public and private datasets with several tampering configurations evidence the suitability of the proposed model.
... There is extensive recent work on image forensics, including techniques to detect image splicing based forgeries ( [5], [6]), copy-move forgeries ( [7], [8], [1]), image retouching ( [9], [10]), seam carving ( [11], [12]) and image resampling ( [13], [14]). Other common manipulations include machine learning based forgeries, typically manipulated using Generative Adversarial Networks (GANs) ( [15], [16]). ...
Preprint
Full-text available
Recent advances in machine learning and computer vision have made it simple to manipulate a variety of images, including satellite images. Most of the commercially available satellite images go through the process of orthorectification to remove potential distortions due to terrain variations. This orthorectification process typically involves the use of rational polynomial coefficients (RPC) that geometrically remap the pixels in the original image to the rectified image. This paper proposes a new method to verify the authenticity of these orthorectified images with respect to the associated RPC metadata. The steps include calculating the Residual Discrete Fourier Transform (DFT) pattern from the image using a linear predictor based residual spectral analysis and comparing with expected residual DFT pattern using the RPC metadata associated with the image. If the metadata associated with an orthorectified image is the correct one, then both the DFT patterns should have high structural similarity. We use SSIM (Structural Similarity Index Metric) to quantify the similarity and thereby verify if the data has been tampered or not. Detailed experimental results are presented to demonstrate the high accuracy of the proposed method in detecting manipulations.<br
... There is extensive recent work on image forensics, including techniques to detect image splicing based forgeries ( [5], [6]), copy-move forgeries ( [7], [8], [1]), image retouching ( [9], [10]), seam carving ( [11], [12]) and image resampling ( [13], [14]). Other common manipulations include machine learning based forgeries, typically manipulated using Generative Adversarial Networks (GANs) ( [15], [16]). ...
Preprint
Full-text available
Recent advances in machine learning and computer vision have made it simple to manipulate a variety of images, including satellite images. Most of the commercially available satellite images go through the process of orthorectification to remove potential distortions due to terrain variations. This orthorectification process typically involves the use of rational polynomial coefficients (RPC) that geometrically remap the pixels in the original image to the rectified image. This paper proposes a new method to verify the authenticity of these orthorectified images with respect to the associated RPC metadata. The steps include calculating the Residual Discrete Fourier Transform (DFT) pattern from the image using a linear predictor based residual spectral analysis and comparing with expected residual DFT pattern using the RPC metadata associated with the image. If the metadata associated with an orthorectified image is the correct one, then both the DFT patterns should have high structural similarity. We use SSIM (Structural Similarity Index Metric) to quantify the similarity and thereby verify if the data has been tampered or not. Detailed experimental results are presented to demonstrate the high accuracy of the proposed method in detecting manipulations.<br
... The authors in [38] improved their network design with a new architecture, ILFNet [39]. The authors in [40] proposes a two stage model where stage-1 performs patch level localization and stage-2 performs image level classification. The method proposed in this paper localizes seams at a pixel level as opposed to the patch level strategies in prior work and focuses on satellite imagery, which is more resistant to seam carving artifacts than consumer images. ...
Preprint
Full-text available
Seam carving is a popular technique for content aware image retargeting. It can be used to deliberately manipulate images, for example, change the GPS locations of a building or insert/remove roads in a satellite image. This paper proposes a novel approach for detecting and localizing seams in such images. While there are methods to detect seam carving based manipulations, this is the first time that robust localization and detection of seam carving forgery is made possible. We also propose a seam localization score (SLS) metric to evaluate the effectiveness of localization. The proposed method is evaluated extensively on a large collection of images from different sources, demonstrating a high level of detection and localization performance across these datasets. The datasets curated during this work will be released to the public.
Preprint
Full-text available
Development and exploitation of technology have led to the further expansion and complexity of digital crimes. On the other hand, the growing volume of data and, subsequently, evidence is a severe challenge in digital forensics. In recent years, the application of machine learning techniques to identify and analyze evidence has been on the rise in different digital forensics domains. This paper offers a systematic literature review of the research published in major academic databases from January 2010 to December 2021 on the application of machine learning in digital forensics, which was not presented yet to the best of our knowledge as comprehensive as this. The review also identifies the domains of digital forensics and machine learning methods that have received the most attention in the previous papers and finally introduces remaining research gaps. Our findings demonstrate that image forensics has obtained the greatest benefit from using machine learning methods, compared to other forensic domains. Moreover, CNN-based models are the most important machine learning methods that are increasingly being used in digital forensics. We present a comprehensive mind map to provide a proper perspective for valuable analytical results. Furthermore, visual analysis has been conducted based on the keywords of the papers, providing different thematic relevance topics. This research will give digital forensics investigators, machine learning developers, security researchers, and enthusiasts a broad view of the application of machine learning in digital forensics.
Article
Full-text available
Recent advances in Generative Adversarial Networks (GANs) have led to the creation of realistic-looking digital images that pose a major challenge to their detection by humans or computers. GANs are used in a wide range of tasks, from modifying small attributes of an image (StarGAN [14]), transferring attributes between image pairs (CycleGAN [92]), as well as generating entirely new images (ProGAN [37], StyleGAN [38], SPADE/GauGAN [65]). In this paper, we propose a novel approach to detect, attribute and localize GAN generated images that combines image features with deep learning methods. For every image, co-occurrence matrices are computed on neighborhood pixels of RGB channels in different directions (horizontal, vertical and diagonal). A deep learning network is then trained on these features to detect, attribute and localize these GAN generated/manipulated images. A large scale evaluation of our approach on 5 GAN datasets comprising over 2.76 million images (ProGAN, StarGAN, CycleGAN, StyleGAN and SPADE/GauGAN) shows promising results in detecting GAN generated images.
Conference Paper
Full-text available
Last-generation GAN models allow to generate synthetic images which are visually indistinguishable from natural ones, raising the need to develop tools to distinguish fake and natural images thus contributing to preserve the trustworthiness of digital images. While modern GAN models can generate very high-quality images with no visibile spatial artifacts, reconstruction of consistent relationships among colour channels is expectedly more difficult. In this paper, we propose a method for distinguishing GAN-generated from natural images by exploiting inconsistencies among spectral bands, with specific focus on the generation of synthetic face images. Specifically, we use cross-band co-occurrence matrices, in addition to spatial co-occurrence matrices, as input to a CNN model, which is trained to distinguish between real and synthetic faces. The results of our experiments confirm the goodness of our approach which outperforms a similar detection technique based on intra-band spatial co-occurrences only. The performance gain is particularly significant with regard to robustness against post-processing, like geometric transformations, filtering and contrast manipulations.
Article
Full-text available
With advanced image journaling tools, one can easily alter the semantic meaning of an image by exploiting certain manipulation techniques such as copy-clone, object splicing, and removal, which mislead the viewers. In contrast, the identification of these manipulations becomes a very challenging task as manipulated regions are not visually apparent. This paper proposes a high-confidence manipulation localization architecture which utilizes resampling features, Long-Short Term Memory (LSTM) cells, and encoder-decoder network to segment out manipulated regions from non-manipulated ones. Resampling features are used to capture artifacts like JPEG quality loss, upsampling, downsampling, rotation, and shearing. The proposed network exploits larger receptive fields (spatial maps) and frequency domain correlation to analyze the discriminative characteristics between manipulated and non-manipulated regions by incorporating encoder and LSTM network. Finally, decoder network learns the mapping from low-resolution feature maps to pixel-wise predictions for image tamper localization. With predicted mask provided by final layer (softmax) of the proposed architecture, end-to-end training is performed to learn the network parameters through back-propagation using ground-truth masks. Furthermore, a large image splicing dataset is introduced to guide the training process. The proposed method is capable of localizing image manipulations at pixel level with high precision, which is demonstrated through rigorous experimentation on three diverse datasets. Code and data are available in the following link-https://github.com/jawadbappy/forgery_localization_HLED
Article
Full-text available
Seam carving is widely used excellent content-aware image scaling method. When an image is processed by seam carving, its local texture changes. Local binary patterns is an excellent local descriptor for describing the local texture of an image. In this paper, a blind detection based uniform local binary patterns(ULBP) is proposed to detect seam-carved image. Firstly, the image is transformed into gray-scale image. Then the ULBP histogram features and seam features are extracted from the gray-scale image. Finally support vector machine (SVM) is adopted as classifier to train and test those features to identify whether an image is subjected to seam carving or not. Wei et al. (Pattern Recogn Lett 36:100–106 2014) method and Ryu et al. (IEICE Trans Inf Syst 97(5):1304–1311 2014) method are selected as the benchmark. Extensive compared experiments are conducted by the three methods, respectively. Experimental results show that the proposed method has the best performance among the three methods under a variety of setting.
Article
Seam carving is a representative content-aware image retargeting approach to adjust the size of an image. To preserve visually prominent content, seam-carving algorithms first calculate the connected path of pixels, referred to as the seam, according to a defined cost function and then adjust the size of an image by removing or duplicating repeatedly calculated seams. Seam carving is actively exploited to overcome diversity in the resolution of images between applications and devices; hence, detecting the distortion caused by seam carving has become important in image forensics. In this paper, we propose a convolutional neural network (CNN)-based approach to classifying seam-carving forgery. To attain the ability to learn low-level features, we designed a convolutional neural network (CNN) architecture comprising five types of network blocks specialized in capturing local artifacts caused by seam carving. An ensemble module is further adopted to both enhance performance and comprehensively analyze the features in the local areas. To validate the effectiveness of our work, extensive experiments based on various CNN-based baselines were conducted. Compared to the baselines, our work exhibits state-of-the-art performance in terms of three-class classification (original, seam inserted, and seam removed). The experimental results also demonstrate that our model with the ensemble module is robust for various unseen cases.
Article
Image retargeting refers to manipulating an input image to adapt its display on diverse screens with distinct spatial resolutions and aspect ratios. However, it might also be used for malicious image forgeries such as object removal. There exist several types of image retargeting techniques, yet most existing image retargeting detection approaches expose only single type of image retargeting. This paper proposes a blind forensics approach to identify three image retargeting techniques including seam carving, scaling and scale-and-stretch. Since image retargeting usually leads to local texture distortion and global structure deformation, a 512-dimensional feature set, which is made up of LBPCO (co-occurrence of local binary pattern) and BSIF (binarized statistical image features), is designed to expose image retargeting artifacts. Support vector machine (SVM) is exploited as pattern classifier to identify three image retarget-ing techniques. Compared with existing works, the proposed approach not only detects image retargeting forgery, but also identifies specific image retargeting technique.
Conference Paper
Deep Learning techniques have been widely used in the recent years, primarily because of their efficiency in several applications, such as engineering, medicine, and data security. Seam carving is a content-aware image resizing method that can also be used for image tampering, being not straightforward to be identified. In this paper, we combine Convolutional Neural Networks and Local Binary Patterns to recognize whether an image has been modified automatically or not by seam carving. The experimental results show that the proposed approach can achieve accuracies within the range [81% − 98%] depending on the severity of the tampering procedure.