miun.sePublikasjoner
Endre søk
Link to record
Permanent link

Direct link
BETA
Li, Yongwei
Publikasjoner (7 av 7) Visa alla publikasjoner
Li, Y. (2020). Computational Light Field Photography: Depth Estimation, Demosaicing, and Super-Resolution. (Doctoral dissertation). Sundsvall: Mid Sweden University
Åpne denne publikasjonen i ny fane eller vindu >>Computational Light Field Photography: Depth Estimation, Demosaicing, and Super-Resolution
2020 (engelsk)Doktoravhandling, med artikler (Annet vitenskapelig)
Abstract [en]

The transition of camera technology from film-based cameras to digital cameras has been witnessed in the past twenty years, along with impressive technological advances in processing massively digitized media content. Today, a new evolution emerged -- the migration from 2D content to immersive perception. This rising trend has a profound and long-term impact to our society, fostering technologies such as teleconferencing and remote surgery. The trend is also reflected in the scientific research community, and more intention has been drawn to the light field and its applications.

 

The purpose of this dissertation is to develop a better understanding of light field structure by analyzing its sampling behavior and to addresses three problems concerning the light field processing pipeline: 1) How to address the depth estimation problem when there is limited color and texture information. 2) How to improve the rendered image quality by using the inherent depth information. 3) How to solve the interdependence conflict of demosaicing and depth estimation.

 

The first problem is solved by a hybrid depth estimation approach that combines advantages of correspondence matching and depth-from-focus, where occlusion is handled by involving multiple depth maps in a voting scheme. The second problem is divided into two specific tasks -- demosaicing and super-resolution, where depth-assisted light field analysis is employed to surpass the competence of traditional image processing. The third problem is tackled with an inferential graph model that encodes the connections between demosaicing and depth estimation explicitly, and jointly performs a global optimization for both tasks.

 

The proposed depth estimation approach shows a noticeable improvement in point clouds and depth maps, compared with references methods. Furthermore, the objective metrics and visual quality are compared with classical sensor-based demosaicing and multi-image super-resolution to show the effectiveness of the proposed depth-assisted light field processing methods. Finally, a multi-task graph model is proposed to challenge the performance of the sequential light field image processing pipeline. The proposed method is validated with various kinds of light fields, and outperforms the state-of-the-art in both demosaicing and depth estimation tasks.

 

The works presented in this dissertation raise a novel view of the light field data structure in general, and provide tools to solve image processing problems in specific. The impact of the outcome can be manifold: To support scientific research with light field microscopes, to stabilize the performance of range cameras for industrial applications, as well as to provide individuals with a high-quality immersive experience.

Abstract [sv]

Under de senaste tjugo åren har det skett en övergång från filmbaserad till digital kamerateknik, parallellt med en imponerande teknisk utveckling inom bearbetning av omfattande digitaliserat medieinnehåll. På senare tid även en ny utvecklingslinje – övergången från 2D-innehåll till omslutande perception. Detta är en utveckling som har långtgående och långvarig påverkan på samhället och främjar arbetsmetoder såsom telekonferens och fjärrstyrd kirurgi. Den här utvecklingst trenden återspeglas också i det vetenskapliga forskningssamhället, och mer uppmärksamhet har lagts på light field och dess olika tillämpningsområden.

Syftet med avhandlingen är att nå en bättre förståelse av strukturen i light field genom att analysera hur light field samplas, och att lösa tre problem inom behandlingsprocessen av light field: 1) Hur problemet med djupestimering kan lösas med begränsad information om färg och textur. 2) Hur renderad bildkvalitet kan förbättras genom att utnyttja den inneboende djupinformationen. 3) Hur beroendekonflikten mellan demosaicing (färgfiltrering) och djupestimering kan lösas.

Det första problemet har lösts genom en hybridmetod för djupestimering, som kombinerar styrkorna med korrespondensmatchning och djup från fokus, där ocklusion hanteras genom att använda flera djupkartor i ett röstningssystem. Det andra problemet delas upp i två separata moment – demosaicing och superupplösning, där djupassisterad analys av light field används för att överträffa kapaciteten för traditionell bildbehandling. Det tredje problemet har angripits med en inferentiell grafmodell som explicit kopplar samman demosaicing och djupestimering, och samfällt utför en global optimering för båda dessa processteg.

Den metod för djupestimering som föreslås producerar visuellt tilltalande punktmoln och djupkartor, jämfört med andra referensmetoder. Objektiva mätvärden och visuell kvalitet jämförs vidare med klassisk sensorbaserad demosaicing och superupplösning från multipla bilder, för att visa effektiviteten hos de föreslagna metoderna för djupassisterad behandling av light field. En multitaskande grafmodell föreslås även för att matcha och överträffa prestandan hos sekventiell light field-baserad bildbehandling. Den metod som föreslås valideras med olika sorters light fields och överträffar de bästa existerande metoderna inom både demosaicing och djupestimering.

De arbeten som presenteras i avhandlingen utgör ett nytt sätt att betrakta den generella datastrukturen hos light field, och tillhandahåller verktyg för att lösa specifika bildbehandlingsproblem. Effekterna av dessa resultat kan vara många, till exempel som stöd för vetenskaplig forskning om light field-baserade mikroskop, för att förbättra prestandan hos avståndsmätande kameror i industriella tillämpningar, såväl som för att erbjuda högkvalitativa omslutande mediaupplevelser.

sted, utgiver, år, opplag, sider
Sundsvall: Mid Sweden University, 2020. s. 56
Serie
Mid Sweden University doctoral thesis, ISSN 1652-893X ; 327
Emneord
Light field, computational photography, depth estimation, demosaicing, super-resolution
HSV kategori
Identifikatorer
urn:nbn:se:miun:diva-39005 (URN)978-91-88947-57-4 (ISBN)
Disputas
2020-06-10, C312, Holmgatan 10, Sundsvall, 09:00 (engelsk)
Opponent
Veileder
Prosjekter
European Unions Horizon 2020 under the Marie Sklodowska-Curie grant agreement No 676401, European Training Network on Full Parallax Imaging
Merknad

Vid tidpunkten för disputationen var följande delarbeten opublicerade: delarbete 4 manuskript och delarbete 5 inskickat.

At the time of the doctoral defence the following papers were unpublished: paper 4 manuscript and paper 5 submitted.

Tilgjengelig fra: 2020-05-18 Laget: 2020-05-11 Sist oppdatert: 2020-05-13bibliografisk kontrollert
Li, Y. & Sjöström, M. (2019). Depth-Assisted Demosaicing for Light Field Data in Layered Object Space. In: 2019 IEEE International Conference on Image Processing (ICIP): . Paper presented at 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 22-25 September, 2019 (pp. 3746-3750). IEEE, Article ID 8803441.
Åpne denne publikasjonen i ny fane eller vindu >>Depth-Assisted Demosaicing for Light Field Data in Layered Object Space
2019 (engelsk)Inngår i: 2019 IEEE International Conference on Image Processing (ICIP), IEEE, 2019, s. 3746-3750, artikkel-id 8803441Konferansepaper, Publicerat paper (Fagfellevurdert)
Abstract [en]

Light field technology, which emerged as a solution to the increasing demands of visually immersive experience, has shown its extraordinary potential for scene content representation and reconstruction. Unlike conventional photography that maps the 3D scenery onto a 2D plane by a projective transformation, light field preserves both the spatial and angular information, enabling further processing steps such as computational refocusing and image-based rendering. However, there are still gaps that have been barely studied, such as the light field demosaicing process. In this paper, we propose a depth-assisted demosaicing method for light field data. First, we exploit the sampling geometry of the light field data with respect to the scene content using the ray-tracing technique and develop a sampling model of light field capture. Then we carry out the demosaicing process in a layered object space with object-space sampling adjacencies rather than pixel placement. Finally, we compare our results with state-of-art approaches and discuss about the potential research directions of the proposed sampling model to show the significance of our approach.

sted, utgiver, år, opplag, sider
IEEE, 2019
Emneord
Lenses, Cameras, Image color analysis, Three-dimensional displays, Microoptics, Interpolation, Two dimensional displays, Light field, demosaicing, object space, ray-tracing technique
HSV kategori
Identifikatorer
urn:nbn:se:miun:diva-37690 (URN)10.1109/ICIP.2019.8803441 (DOI)000521828603177 ()2-s2.0-85076819023 (Scopus ID)978-1-5386-6249-6 (ISBN)
Konferanse
2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 22-25 September, 2019
Tilgjengelig fra: 2019-11-15 Laget: 2019-11-15 Sist oppdatert: 2020-05-11bibliografisk kontrollert
Li, Y., Olsson, R. & Sjöström, M. (2018). An analysis of demosaicing for plenoptic capture based on ray optics. In: Proceedings of 3DTV Conference 2018: . Paper presented at 3D at any scale and any perspective, 3-5 June 2018, Stockholm – Helsinki – Stockholm. , Article ID 8478476.
Åpne denne publikasjonen i ny fane eller vindu >>An analysis of demosaicing for plenoptic capture based on ray optics
2018 (engelsk)Inngår i: Proceedings of 3DTV Conference 2018, 2018, artikkel-id 8478476Konferansepaper, Publicerat paper (Fagfellevurdert)
Abstract [en]

The plenoptic camera is gaining more and more attention as it capturesthe 4D light field of a scene with a single shot and enablesa wide range of post-processing applications. However, the preprocessing steps for captured raw data, such as demosaicing, have been overlooked. Most existing decoding pipelines for plenoptic cameras still apply demosaicing schemes which are developed for conventional cameras. In this paper, we analyze the sampling pattern of microlens-based plenoptic cameras by ray-tracing techniques and ray phase space analysis. The goal of this work is to demonstrate guidelines and principles for demosaicing the plenoptic captures by taking the unique microlens array design into account. We show that the sampling of the plenoptic camera behaves differently from that of a conventional camera and the desired demosaicing scheme is depth-dependent.

Emneord
Light field, plenoptic camera, depth, image demosaicing
HSV kategori
Identifikatorer
urn:nbn:se:miun:diva-33618 (URN)10.1109/3DTV.2018.8478476 (DOI)000454903900008 ()2-s2.0-85056161198 (Scopus ID)978-1-5386-6125-3 (ISBN)
Konferanse
3D at any scale and any perspective, 3-5 June 2018, Stockholm – Helsinki – Stockholm
Tilgjengelig fra: 2018-05-15 Laget: 2018-05-15 Sist oppdatert: 2020-05-11bibliografisk kontrollert
Li, Y., Scrofani, G., Sjöström, M. & Martinez-Corraly, M. (2018). Area-Based Depth Estimation for Monochromatic Feature-Sparse Orthographic Capture. In: 2018 26th European Signal Processing Conference (EUSIPCO): . Paper presented at EUSIPCO 2018, 26th European Signal Processing Conference, Rome, Italy, September 3-7, 2018 (pp. 206-210). IEEE conference proceedings, Article ID 8553336.
Åpne denne publikasjonen i ny fane eller vindu >>Area-Based Depth Estimation for Monochromatic Feature-Sparse Orthographic Capture
2018 (engelsk)Inngår i: 2018 26th European Signal Processing Conference (EUSIPCO), IEEE conference proceedings, 2018, s. 206-210, artikkel-id 8553336Konferansepaper, Publicerat paper (Fagfellevurdert)
Abstract [en]

With the rapid development of light field technology, depth estimation has been highlighted as one of the critical problems in the field, and a number of approaches have been proposed to extract the depth of the scene. However, depthestimation by stereo matching becomes difficult and unreliable when the captured images lack both color and feature information. In this paper, we propose a scheme that extracts robust depth from monochromatic, feature-sparse scenes recorded in orthographic sub-aperture images. Unlike approaches which relyon the rich color and texture information across the sub-aperture views, our approach is based on depth from focus techniques. First, we superimpose shifted sub-aperture images on top of anarbitrarily chosen central image. To focus on different depths, the shift amount is varied based on the micro-lens array properties. Next, an area-based depth estimation approach is applied tofind the best match among the focal stack and generate the dense depth map. This process is repeated for each sub-aperture image. Finally, occlusions are handled by merging depth maps generated from different central images followed by a voting process. Results show that the proposed scheme is more suitable than conventional depth estimation approaches in the context of orthographic captures that have insufficient color and feature information, such as microscopic fluorescence imaging.

sted, utgiver, år, opplag, sider
IEEE conference proceedings, 2018
Emneord
Depth estimation, integral imaging, orthographic views, depth from focus
HSV kategori
Identifikatorer
urn:nbn:se:miun:diva-34418 (URN)000455614900042 ()2-s2.0-85059811493 (Scopus ID)
Konferanse
EUSIPCO 2018, 26th European Signal Processing Conference, Rome, Italy, September 3-7, 2018
Tilgjengelig fra: 2018-09-14 Laget: 2018-09-14 Sist oppdatert: 2020-05-11bibliografisk kontrollert
Wang, C., Wang, X., Li, Y., Xia, Z. & Zhang, C. (2018). Quaternion polar harmonic Fourier moments for color images. Information Sciences, 450, 141-156
Åpne denne publikasjonen i ny fane eller vindu >>Quaternion polar harmonic Fourier moments for color images
Vise andre…
2018 (engelsk)Inngår i: Information Sciences, ISSN 0020-0255, E-ISSN 1872-6291, Vol. 450, s. 141-156Artikkel i tidsskrift (Fagfellevurdert) Published
Abstract [en]

This paper proposes quaternion polar harmonic Fourier moments (QPHFM) for color image processing and analyzes the properties of QPHFM. After extending Chebyshev–Fourier moments (CHFM) to quaternion Chebyshev-Fourier moments (QCHFM), comparison experiments, including image reconstruction and color image object recognition, on the performance of QPHFM and quaternion Zernike moments (QZM), quaternion pseudo-Zernike moments (QPZM), quaternion orthogonal Fourier-Mellin moments (QOFMM), QCHFM, and quaternion radial harmonic Fourier moments (QRHFM) are carried out. Experimental results show QPHFM can achieve an ideal performance in image reconstruction and invariant object recognition in noise-free and noisy conditions. In addition, this paper discusses the importance of phase information of quaternion orthogonal moments in image reconstruction. 

Emneord
Image reconstruction, Moment invariant, Object recognition, Orthogonal moment, Phase, Quaternion polar harmonic Fourier moments
HSV kategori
Identifikatorer
urn:nbn:se:miun:diva-33500 (URN)10.1016/j.ins.2018.03.040 (DOI)000432646100008 ()2-s2.0-85044451202 (Scopus ID)
Tilgjengelig fra: 2018-04-16 Laget: 2018-04-16 Sist oppdatert: 2018-06-10bibliografisk kontrollert
Li, Y., Pla, F. & Sjöström, M.A Collaborative Graph Model for Light Field Demosaicing and Depth Estimation.
Åpne denne publikasjonen i ny fane eller vindu >>A Collaborative Graph Model for Light Field Demosaicing and Depth Estimation
(engelsk)Manuskript (preprint) (Annet vitenskapelig)
HSV kategori
Identifikatorer
urn:nbn:se:miun:diva-39013 (URN)
Tilgjengelig fra: 2020-05-13 Laget: 2020-05-13 Sist oppdatert: 2020-05-13bibliografisk kontrollert
Li, Y. & Sjöström, M.Depth-Assisted Light Field Super-Resolution in Layered Object Space.
Åpne denne publikasjonen i ny fane eller vindu >>Depth-Assisted Light Field Super-Resolution in Layered Object Space
(engelsk)Manuskript (preprint) (Annet vitenskapelig)
HSV kategori
Identifikatorer
urn:nbn:se:miun:diva-39012 (URN)
Tilgjengelig fra: 2020-05-13 Laget: 2020-05-13 Sist oppdatert: 2020-05-13bibliografisk kontrollert
Organisasjoner