Research Article
BibTex RIS Cite

A New Approach to Improve Visual Quality for Single Image Super Resolution Applications

Year 2023, Volume: 6 Issue: 1, 52 - 58, 15.03.2023
https://doi.org/10.38016/jista.1183882

Abstract

The single image super resolution problem has been extensively studied in the literature using various deep learning-based techniques. Super resolution based on deep convolutional networks has become a rapidly growing area of interest with many practical applications. However, the first studies based on deep learning were based on convolutional neural networks and peak signal-to-noise ratio (PSNR) oriented. Thanks to the models developed based on generative adversarial networks (GAN) in recent years, it has been determined as the main objective to increase the visual quality; however, this is not seen when the image quality metrics are examined. In this study, both mean square error and perceptual loss values were used for the network loss used during the training of the network. Also, the combination of three different training datasets was used as a new training dataset. As a result of these factors, both the visual quality has been increased and a significant increase has been achieved in the image quality metric values. In addition, batch normalization layers are not included in the network architecture and the training speed of the deep network architecture is increased by using the skip connection technique. The success performance of the proposed model was compared with the state-of-the-art models in the literature. Here, the peak signal to noise ratio (PSNR) and structural similarity index measure (SSIM) values were calculated and evaluated separately for three different test data sets commonly used in the literature. When the obtained results are evaluated, it is seen that the proposed model is more successful than other models and generates higher quality images. When all the findings are evaluated, it is seen that the proposed model is a more efficient model in terms of both success and training speed compared to state-of-the-art models.

References

  • Anwar, S., Khan, S., Barnes, N., 2020. A deep journey into super-resolution: A Survey. ACM Computing Surveys 53:1-34. http://dx.doi.org/10.1145/3390462
  • Chudasama, V., Patel, H., Prajapati, K., Upla, K., Ramachandra, R., Raja, K., Busch, C., 2020. TherISuRNet- A computationally efficient thermal image super-resolution network. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp 388-397. http://dx.doi.org/10.1109/CVPRW50498.2020.00051
  • Dong, C., Loy, C.C., He, K., Tan, X., 2015. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38:295-307. http://dx.doi.org/10.1109/TPAMI.2015.2439281
  • Dong, C., Loy, C.C., Tang, X., 2016. Accelerating the Super-Resolution Convolutional Neural Network. Computer Vision ECCV 2016 Lecture Notes in Computer Science Springer 9906. https://doi.org/10.1007/978-3-319-46475-6_25
  • Dosovitskiy, A., Brox, T., 2016. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems (NIPS). pp 658–666. https://doi.org/10.48550/arXiv.1602.02644
  • Glasner, D., Bagon, S., Irani, M., 2009. Super-resolution from a single image. IEEE 12th International Conference on Computer Vision. pp 349-356. http://dx.doi.org/10.1109/ICCV.2009.5459271
  • Goodfellow, I., vd., 2014. Generative adversarial networks. In Advances in Neural Information Processing Systems (NIPS). pp 2672–2680. http://dx.doi.org/10.1145/3422622
  • Gu, Y., vd., 2020. MedSRGAN: medical images super-resolution using generative adversarial networks. Multimedia Tools and Applications 79:21815–21840. http://dx.doi.org/10.1007/s11042-020-08980-w
  • Javaid, H., Babar, T.K., Rasool, A., Saghir, R.U., 2013. Video colour variation detection and motion magnification to observe subtle changes. M.Sc. Thesis, Blekinge Institute of Technology, Faisalabad, Pakistan.
  • Johnson, J., Alahi, A., Li, F., 2016. Perceptual losses for real-time style transfer and super resolution. In European Conference on Computer Vision (ECCV) Springer. pp 694–711. http://dx.doi.org/10.1007/978-3-319-46475-6_43
  • Kim, J., Lee, J.K., Lee, K.M., 2016. Accurate image super-resolution using very deep convolutional networks. IEEE CVPR. pp 1646–1654. http://dx.doi.org/10.1109/CVPR.2016.182
  • Ledig, C., vd., 2017. Photo-realistic single image super-resolution using a generative adversarial network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Honolulu. pp 105-114. http://dx.doi.org/10.1109/CVPR.2017.19
  • Radford, A., Metz, L., Chintala, S., 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
  • Senalp, F.M., Ceylan, M., 2020. Enhancement of low resolution thermal face image resolution using deep learning. European Journal of Science & Tech. pp 131-135. https://doi.org/10.31590/ejosat.802174
  • Senalp, F.M., Ceylan, M., 2021. Deep learning based super resolution and classification applications for neonatal thermal images. Traitement du Signal 38(5):1361-1368. https://doi.org/10.18280/ts.380511
  • Toyran, M., 2008. Reconstructing super resolution images from low resolution images. M.Sc. Thesis, Institute of science, Istanbul.
  • Wang, M., Chen, Z., Wu, Q.M.J., Jian, M., 2020. Improved face super-resolution generative adversarial networks. Machine Vision and Applications 31:22. https://doi.org/10.1007/s00138-020-01073-6
  • Yan, R., Yang, K., Wang, K., 2021. NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation across RGB-Depth, Polarization, and Thermal Images. 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp 1129-1135. http://dx.doi.org/10.1109/ROBIO54168.2021.9739390
  • Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L., 2018. Image super-resolution:the techniques, applications, and future. Signal Processing 128:389-408. http://dx.doi.org/10.1016/j.sigpro.2016.05.002
  • Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y., 2018. Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision. pp 286–301. http://dx.doi.org/10.1007/978-3-030-01234-2_18

Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım

Year 2023, Volume: 6 Issue: 1, 52 - 58, 15.03.2023
https://doi.org/10.38016/jista.1183882

Abstract

Tek görüntü süper çözünürlük problemi, literatürde çeşitli derin öğrenme tabanlı teknikler kullanılarak kapsamlı çalışmalar yapılmıştır. Derin evrişimli ağlar tabanlı süper çözünürlük, çok sayıda pratik uygulama ile beraber hızla büyüyen bir ilgi alanı haline gelmiştir. Bununla birlikte derin öğrenme tabanlı ilk çalışmalar evrişimli sinir ağları tabanlı olup, tepe sinyal gürültü oranı odaklı çalışmalardır. Son yıllardaki çekişmeli üretici ağlar tabanlı geliştirilen modeller sayesinde görsel kaliteyi artırmak esas amaç olarak belirlenmiştir; fakat bu durum görüntü kalite metrikleri incelendiğinde görülmemektedir. Bu çalışmada ise ağın eğitimi sırasında kullanılan ağ kaybı için hem ortalama kare hata hem de algısal kayıp değerlerinden faydalanılmıştır. Ayrıca, üç farklı eğitim veri setinin birleşimi yeni bir eğitim veri seti olarak kullanılmıştır. Bu etmenlerin sonucunda hem görsel kalite artırılmış hem de görüntü kalite metrik değerlerinde ciddi bir artış yakalanmıştır. Ek olarak, yığın normalleştirme katmanları ağ mimarisine dahil edilmemiş ve bağlantı atlama tekniği kullanılarak derin ağ mimarisinin eğitim hızı artırılmıştır. Önerilen modelin başarı performansı literatürde yer alan önemli modeller ile karşılaştırılmıştır. Burada, tepe sinyal gürültü oranı ve yapısal benzerlik indeksi değerleri literatürde yaygın kullanılan üç farklı test veri seti için ayrı ayrı hesaplanmış ve değerlendirilmiştir. Elde edilen sonuçlar değerlendirildiğinde önerilen modelin diğer modellere göre daha başarılı olduğu ve daha kaliteli görüntüler oluşturduğu görülmektedir. Tüm bulgular değerlendirildiğinde önerilen modelin diğer modellere kıyasla hem başarı hem de eğitim hızı bakımından daha verimli bir model olduğu görülmektedir.

References

  • Anwar, S., Khan, S., Barnes, N., 2020. A deep journey into super-resolution: A Survey. ACM Computing Surveys 53:1-34. http://dx.doi.org/10.1145/3390462
  • Chudasama, V., Patel, H., Prajapati, K., Upla, K., Ramachandra, R., Raja, K., Busch, C., 2020. TherISuRNet- A computationally efficient thermal image super-resolution network. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp 388-397. http://dx.doi.org/10.1109/CVPRW50498.2020.00051
  • Dong, C., Loy, C.C., He, K., Tan, X., 2015. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38:295-307. http://dx.doi.org/10.1109/TPAMI.2015.2439281
  • Dong, C., Loy, C.C., Tang, X., 2016. Accelerating the Super-Resolution Convolutional Neural Network. Computer Vision ECCV 2016 Lecture Notes in Computer Science Springer 9906. https://doi.org/10.1007/978-3-319-46475-6_25
  • Dosovitskiy, A., Brox, T., 2016. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems (NIPS). pp 658–666. https://doi.org/10.48550/arXiv.1602.02644
  • Glasner, D., Bagon, S., Irani, M., 2009. Super-resolution from a single image. IEEE 12th International Conference on Computer Vision. pp 349-356. http://dx.doi.org/10.1109/ICCV.2009.5459271
  • Goodfellow, I., vd., 2014. Generative adversarial networks. In Advances in Neural Information Processing Systems (NIPS). pp 2672–2680. http://dx.doi.org/10.1145/3422622
  • Gu, Y., vd., 2020. MedSRGAN: medical images super-resolution using generative adversarial networks. Multimedia Tools and Applications 79:21815–21840. http://dx.doi.org/10.1007/s11042-020-08980-w
  • Javaid, H., Babar, T.K., Rasool, A., Saghir, R.U., 2013. Video colour variation detection and motion magnification to observe subtle changes. M.Sc. Thesis, Blekinge Institute of Technology, Faisalabad, Pakistan.
  • Johnson, J., Alahi, A., Li, F., 2016. Perceptual losses for real-time style transfer and super resolution. In European Conference on Computer Vision (ECCV) Springer. pp 694–711. http://dx.doi.org/10.1007/978-3-319-46475-6_43
  • Kim, J., Lee, J.K., Lee, K.M., 2016. Accurate image super-resolution using very deep convolutional networks. IEEE CVPR. pp 1646–1654. http://dx.doi.org/10.1109/CVPR.2016.182
  • Ledig, C., vd., 2017. Photo-realistic single image super-resolution using a generative adversarial network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Honolulu. pp 105-114. http://dx.doi.org/10.1109/CVPR.2017.19
  • Radford, A., Metz, L., Chintala, S., 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
  • Senalp, F.M., Ceylan, M., 2020. Enhancement of low resolution thermal face image resolution using deep learning. European Journal of Science & Tech. pp 131-135. https://doi.org/10.31590/ejosat.802174
  • Senalp, F.M., Ceylan, M., 2021. Deep learning based super resolution and classification applications for neonatal thermal images. Traitement du Signal 38(5):1361-1368. https://doi.org/10.18280/ts.380511
  • Toyran, M., 2008. Reconstructing super resolution images from low resolution images. M.Sc. Thesis, Institute of science, Istanbul.
  • Wang, M., Chen, Z., Wu, Q.M.J., Jian, M., 2020. Improved face super-resolution generative adversarial networks. Machine Vision and Applications 31:22. https://doi.org/10.1007/s00138-020-01073-6
  • Yan, R., Yang, K., Wang, K., 2021. NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation across RGB-Depth, Polarization, and Thermal Images. 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp 1129-1135. http://dx.doi.org/10.1109/ROBIO54168.2021.9739390
  • Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L., 2018. Image super-resolution:the techniques, applications, and future. Signal Processing 128:389-408. http://dx.doi.org/10.1016/j.sigpro.2016.05.002
  • Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y., 2018. Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision. pp 286–301. http://dx.doi.org/10.1007/978-3-030-01234-2_18
There are 20 citations in total.

Details

Primary Language Turkish
Subjects Artificial Intelligence
Journal Section Research Articles
Authors

Fatih Mehmet Şenalp 0000-0001-7831-6724

Early Pub Date December 27, 2022
Publication Date March 15, 2023
Submission Date October 3, 2022
Published in Issue Year 2023 Volume: 6 Issue: 1

Cite

APA Şenalp, F. M. (2023). Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım. Journal of Intelligent Systems: Theory and Applications, 6(1), 52-58. https://doi.org/10.38016/jista.1183882
AMA Şenalp FM. Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım. JISTA. March 2023;6(1):52-58. doi:10.38016/jista.1183882
Chicago Şenalp, Fatih Mehmet. “Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım”. Journal of Intelligent Systems: Theory and Applications 6, no. 1 (March 2023): 52-58. https://doi.org/10.38016/jista.1183882.
EndNote Şenalp FM (March 1, 2023) Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım. Journal of Intelligent Systems: Theory and Applications 6 1 52–58.
IEEE F. M. Şenalp, “Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım”, JISTA, vol. 6, no. 1, pp. 52–58, 2023, doi: 10.38016/jista.1183882.
ISNAD Şenalp, Fatih Mehmet. “Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım”. Journal of Intelligent Systems: Theory and Applications 6/1 (March 2023), 52-58. https://doi.org/10.38016/jista.1183882.
JAMA Şenalp FM. Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım. JISTA. 2023;6:52–58.
MLA Şenalp, Fatih Mehmet. “Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım”. Journal of Intelligent Systems: Theory and Applications, vol. 6, no. 1, 2023, pp. 52-58, doi:10.38016/jista.1183882.
Vancouver Şenalp FM. Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım. JISTA. 2023;6(1):52-8.

Journal of Intelligent Systems: Theory and Applications