Charles B. Collins
ABSTRACT
We describe a deep learning convolutional neural network (CNN) for enhancing low resolution multispectral satellite imagery without the use of a panchromatic image. For training, low resolution images are used as input and corresponding high resolution images are used as the target output (label). The CNN learns to automatically extract hierarchical features that can be used to enhance low resolution imagery. The trained network can then be effectively used for super-resolution enhancement of low resolution multispectral images where no corresponding high resolution image is available. The CNN enhances all four spectral bands of the low resolution image simultaneously and adjusts pixel values of the low resolution to match the dynamic range of the high resolution image. The CNN yields higher quality images than standard image resampling methods.
- University of Twenty Faculty of Geo-Information and Earth Observation (ITC) ITC's database of satellites and sensors. 2017.Google Scholar
- LeCun, Y., Bengio, Y. and Hinton, G. Deep learning. Nature, 521, 7553 (2015), 436--444.Google ScholarCross Ref
- Nielsen, M. A. Neural Networks and Deep Learning. Determination Press, 2015.Google Scholar
- Badrinarayanan, V., Handa, A. and Cipolla, R. Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling. arXiv preprint arXiv:1505.07293 (2015).Google Scholar
- Basu, S., Ganguly, S., Mukhopadhyay, S., DiBiano, R., Karki, M. and Nemani, R. Deepsat: a learning framework for satellite imagery. In Proceedings of the Proceedings of the 23rd SIGSPATIAL International Conference on Advances in Geographic Information Systems (Seattle, WA, 2015). ACM. Google ScholarDigital Library
- Farabet, C., Couprie, C., Najman, L. and LeCun, Y. Learning hierarchical features for scene labeling. IEEE transactions on pattern analysis and machine intelligence, 35, 8 (2013), 1915--1929. Google ScholarDigital Library
- Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A. Going deeper with convolutions. In Proceedings of the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Boston, MA, 2015).Google ScholarCross Ref
- Tompson, J. J., Jain, A., LeCun, Y. and Bregler, C. Joint training of a convolutional network and a graphical model for human pose estimation. City, 2014.Google Scholar
- Chen, S., Wang, H., Xu, F. and Jin, Y.-Q. Target Classification Using the Deep Convolutional Networks for SAR Images. IEEE Transactions on Geoscience and Remote Sensing, 54, 8 (2016), 4806--4817.Google Scholar
- Krizhevsky, A., Sutskever, I. and Hinton, G. E. Imagenet classification with deep convolutional neural networks. NIPS, 2012. Google ScholarDigital Library
- Dong, C., Loy, C. C., He, K. and Tang, X. Learning a deep convolutional network for image super-resolution. Springer, 2014.Google ScholarCross Ref
- Zhong, J., Yang, B., Huang, G., Zhong, F. and Chen, Z. Remote sensing image fusion with convolutional neural network. Sensing and Imaging, 17, 1 (2016), 1--16.Google ScholarCross Ref
- Radhika, K., Rao, C. and Prasad, V. K. Enhancement of AWiFS Spatial Resolution with SVM Learning. In Proceedings of the 2016 IEEE 6th International Conference on Advanced Computing (IACC) (Bhimavaram, India, (2016). IEEE.Google ScholarCross Ref
- Klinkenberg, L. Image Fusion.Google Scholar
- Saraf, A. IRS-1C-LISS-III and PAN data fusion: an approach to improve remote sensing based mapping techniques. International Journal of Remote Sensing, 20, 10 (1999), 1929--1934.Google ScholarCross Ref
- Saroglu, E., Bektas, F., Musaoglu, N. and Goksel, C. Fusion of multisensor remote sensing data: assessing the quality of resulting images. Int. Arch. Photogram. Rem. Sens. Spatial. Inform. Sci, 35 (2004), 575--579.Google Scholar
- Venkataraman, G., Mahato, B. C., Ravi, S., Rao, Y. and Mathur, P. Fusion of Optical and Microwave Remote Sensing data for Snow Cover Mapping. In Proceedings of the Geoscience and Remote Sensing Symposium, 2004. IGARSS'04. Proceedings. 2004 IEEE International (Anchorage, AK, 2004). IEEE.Google ScholarCross Ref
- Pal, S., Majumdar, T. and Bhattacharya, A. K. ERS-2 SAR and IRS-1C LISS III data fusion: A PCA approach to improve remote sensing based geological interpretation. ISPRS journal of photogrammetry and remote sensing, 61, 5 (2007), 281--297.Google Scholar
- Harris, J. R., Murray, R. and Hirose, T. IHS transform for the integration of radar imagery with other remotely sensed data. Photogrammetric Engineering and Remote Sensing, 56, 12 (1990), 1631--1641.Google Scholar
- Yonghong, J. Fusion of landsat TM and SAR image based on principal component analysis. Remote Sensing Technology and Application, 13, 1 (1998), 46--49.Google Scholar
- Zhou, J., Civco, D. and Silander, J. A wavelet transform method to merge Landsat TM and SPOT panchromatic data. International Journal of Remote Sensing, 19, 4 (1998), 743--757.Google ScholarCross Ref
- Huang, D. and Yang, M. SAR and Multi-spectral images fusion based on Contourlet and HIS Transform. In Proceedings of the Wireless Communications Networking and Mobile Computing (WiCOM), 2010 6th International Conference on (Chengdu, China, 2010). IEEE.Google ScholarCross Ref
- Bannari, A., El-Battay, A., Saquaque, A. and Miri, A. PALSAR-FBS L-HH Mode and Landsat-TM Data Fusion for Geological Mapping. Advances in Remote Sensing, 5, 04 (2016), 246.Google Scholar
- Rao, C., Malleswara Rao, J., Senthil Kumar, A. and Dadhwal, V. Fast spatiotemporal data fusion: merging LISS III with AWiFS sensor data. International Journal of Remote Sensing, 35, 24 (2014), 8323--8344. Google ScholarDigital Library
- Li, S., Yin, H. and Fang, L. Remote sensing image fusion via sparse representations over learned dictionaries. IEEE Transactions on Geoscience and Remote Sensing, 51, 9 (2013), 4779--4789.Google ScholarCross Ref
- Rao, C., Malleswara Rao, J., Senthil Kumar, A., Lakshmi, B. and Dadhwal, V. Expansion of LISS III swath using AWiFS wider swath data and contourlet coefficients learning. GIScience & Remote Sensing, 52, 1 (2015), 78--93.Google ScholarCross Ref
- Patwary, M. A. A., Champati ray, P. K. and Parvaiz, I. IRS-LISS-III and PAN data analysis for landslide susceptibility mapping using heuristic approach in active tectonic region of Himalaya. Journal of the Indian Society of Remote Sensing, 37, 3 (2009), 493--509.Google ScholarCross Ref
- Dong, C., Loy, C. C. and Tang, X. Accelerating the super-resolution convolutional neural network. In Proceedings of the European Conference on Computer Vision (Amsterdam, The Netherlands, 2016). Springer.Google ScholarCross Ref
- Kim, J., Kwon Lee, J. and Mu Lee, K. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Seattle, WA, 2016).Google Scholar
- Wang, L., Lin, Z., Gao, J., Deng, X. and An, W. Fast single image super-resolution based on sigmoid transformation. arXiv preprint arXiv:1708.07029 (2017).Google Scholar
- Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J. and Wang, Z. Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint arXiv:1609.04802 (2016).Google Scholar
- Masi, G., Cozzolino, D., Verdoliva, L. and Scarpa, G. Pansharpening by convolutional neural networks. Remote Sensing, 8, 7 (2016), 594.Google ScholarCross Ref
- Palsson, F., Sveinsson, J. R. and Ulfarsson, M. O. Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network. IEEE Geoscience and Remote Sensing Letters, 14, 5 (2017), 639--643.Google ScholarCross Ref
- Seshadri, K., Rao, M., Jayaraman, V., Thyagarajan, K. and Murthi, K. S. Resourcesat-1: A global multi-observation mission for resources monitoring. Acta Astronautica, 57, 2 (2005), 534--539.Google ScholarCross Ref
- Pandya, M., Murali, K. and Kirankumar, A. Quantification and comparison of spectral characteristics of sensors on board Resourcesat-1 and Resourcesat-2 satellites. Remote sensing letters, 4, 3 (2013), 306--314.Google Scholar
- Clevert, D.-A., Unterthiner, T. and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015).Google Scholar
- Wang, Z., Bovik, A. C., Sheikh, H. R. and Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13, 4 (2004), 600--612. Google ScholarDigital Library
- Huynh-Thu, Q. and Ghanbari, M. Scope of validity of PSNR in image/video quality assessment. Electronics letters, 44, 13 (2008), 800--801.Google Scholar
- GDAL Development Team GDAL - Geospatial Data Abstraction Library, Version 1.10. Open Source Geospatial Foundation, 2017.Google Scholar
- Kingma, D. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
Index Terms
- Deep learning for multisensor image resolution enhancement
Recommendations
Convolutional neural networks for wavelet domain super resolution
Proposed a super resolution method with higher reconstruction accuracy than before.Cast super resolution as a problem of estimating sparse wavelet detail coefficients.Estimated sparse wavelet coefficients using a convolutional neural network (CNN)...
Microscopic image super resolution using deep convolutional neural networks
AbstractRecently, deep convolutional neural networks (CNNs) have achieved excellent results in single image super resolution (SISR). Owing to the strength of deep CNNs, it gives promising results compared to state-of-the-art learning based models on ...
Modified Back Projection Kernel Based Image Super Resolution
AIMS '14: Proceedings of the 2014 2nd International Conference on Artificial Intelligence, Modelling and SimulationIn this paper, we propose a new super resolution technique based on iterative interpolation followed by registering them using back projection (BP). Firstly the low resolution image is interpolated and then decimated to four low resolution images. The ...
Comments