Computational methods in super-resolution microscopy

Abstract

The broad applicability of super-resolution microscopy has been widely demonstrated in various areas and disciplines. The optimization and improvement of algorithms used in super-resolution microscopy are of great importance for achieving optimal quality of super-resolution imaging. In this review, we comprehensively discuss the computational methods in different types of super-resolution microscopy, including deconvolution microscopy, polarization-based super-resolution microscopy, structured illumination microscopy, image scanning microscopy, super-resolution optical fluctuation imaging microscopy, single-molecule localization microscopy, Bayesian super-resolution microscopy, stimulated emission depletion microscopy, and translation microscopy. The development of novel computational methods would greatly benefit super-resolution microscopy and lead to better resolution, improved accuracy, and faster image processing.

1 Introduction

The past few decades have testified to the rapid development of various super-resolution microscopy techniques (Ding et al., 2011; Gu et al., 2014; Gao et al., 2016; Yang et al., 2016a; Yu et al., 2016). Because the process of optical imaging is the convolution of the original object and the point spread function (PSF) of the system, the wave nature of light limits the resolution of conventional optical microscopy. Fortunately, the emergence of super-resolution microscopy has successfully broken the diffraction barrier and brought the spatial resolution of optical imaging down to nanometer scale. Super-resolution microscopy is therefore often termed ‘optical nanoscopy’. Quite a few super-resolution techniques largely rely on computational methods to further enhance the spatial resolution. For instance, deconvolution microscopy can effectively suppress the out-of-focus background and reconstruct the original object through deconvolution (Falk and Lauf, 2001). Fluorescence polarization microscopy (FPM) achieves super-resolution imaging by selective polarization modulation (Rizzo and Piston, 2005). Structured illumination microscopy (SIM) extends the optical transfer function (OTF) in the Fourier domain to enhance the spatial resolution (Gustafsson, 2000). Image scanning microscopy (ISM) replaces conventionally used photomultiplier tubes (PMT) detector with charge-coupled device (CCD), and achieves subdiffraction resolution through deconvolution (Müller and Enderlein, 2010). Super-resolution optical fluctuation imaging (SOFI) employs correlation analysis on the temporal fluctuation of the fluorescence to distinguish signals from independently fluctuating fluorophores (Dertinger et al., 2009). Consequently, the spatial resolution increases linearly with cumulant orders. Single-molecule localization microscopy (SMLM) can accurately localize individual molecules through sequentially switching them between fluorescently on and off states (Betzig et al., 2006; Rust et al., 2006). The achievable resolution of SMLM, e.g., PALM/STORM, can be down to 10–20 nm. As a result, computational methods in super-resolution microscopy are extremely important for achieving high-quality super-resolution imaging.

2 Deconvolution microscopy

Deconvolution microscopy is often used in fluorescence microscopy to remove the out-of-focus background. In an epifluorescence microscope, out-of-focus stacks are blurred, but their signals are still contributed to the detector, resulting in a strong background (Pawley, 2010). The microscopy imaging process could be described as I=O*H+B+N, where I is the detected image, O is the original image, H is the 3D PSF of the optical system, B is the offset from the image background and dark current from the detector, and N is the noise. Shot noise from the detector is a major source of noise, and its intensity is proportional to the square root of the signal level (Mertz, 2011). It is difficult to distinguish the signal from the background when the background is strong.

An intuitive way to remove the background is to solve the reverse problem in the Fourier domain, O=H^TI/(H^TH), but the denominator decays to zero for high frequency, indicating that it is an ill-posed problem. A practical solution is to use the Wiener filter:

$$O(u,\,v,\,\omega ) = {{{H^*}(u,\,v,\,\omega )} \over {|H(u,\,v,\,\omega ){|^2} + K}}I(u,\,v,\,\omega ),$$

((1))

where K is the noise power spectra (Gonzalez and Woods, 2008). This method is fast with reasonable accuracy and also works in problems without nonnegative constraints.

Statistical algorithms take into account the noise and solve the problem as a random process. The maximum likelihood estimation method, for example, uses the image that has the maximum likelihood of estimating the original object. A log-likelihood function is calculated and optimized with iteration (Broxton et al., 2013). Sometimes a penalty function can be added to ensure its convergence.

Constrained iterative algorithms use the constraint that the intensity of each pixel cannot be negative. An initial object function guess is provided. The value of each pixel is updated from the image formation equation on each iteration, and then negative values are set to zero. The Jansson Van-Cittert algorithm (Agard et al., 1989; Jansson, 2014) and the Gold algorithm (Gold, 1964) are two different approaches to updating the object function in each iteration.

Blind deconvolution algorithms assume that both the object and the PSF are unknown, and they use maximum likelihood estimate (MLE) estimation to obtain their approximation. An initial guess of the object and PSF must be provided. First, the object function is updated with the PSF, and then the PSF is updated with the new object function. This process is repeated interactively until a stop criterion is satisfied (Sibarita, 2005; Biggs, 2010; Yassif, 2012). The advantage of blind deconvolution is that the spatial distribution of the PSF of the system is not required, which is difficult or even impossible to obtain in some situations. However, this algorithm has the limitation of unstable and slow divergence.

There are also deconvolution methods applied for 2D images. For example, a variant of the nearest neighborhood method uses the 2D image itself as an estimation of the neighbor stacks. By iteration, it can also enhance the contrast of 2D images. These algorithms feature fast speed, but at the cost of efficiency.

Deconvolution microscopy has better performance for samples with low fluorescence density. Several factors limit its wider application: first, an accurate shift-invariant PSF can hardly be measured (McNally et al., 1999). Second, artifacts may be induced by the ill-conditional nature of the problems. Despite these constraints, deconvolution microscopy is still widely used as an economic solution to improve the resolution/contrast in both wide-field and confocal laser scanning microscopy, even in stimulated emission depletion (STED) super-resolution microscopy (Liu et al., 2012).

3 Polarization-based super-resolution microscopy

Polarization that arises from fluorescent dipole orientation is an important physical property in fluorescence. By polarization modulation of linear dichroism and analysis of fluorescence anisotropy, the fluorescent dipole orientation can be monitored so that the direction of targeted protein may be determined due to a restricted orientation between the fluorophores and their targets. Through specific fluorescent specimen binding to target molecules, fluorescence FPM plays an important role in resolving structural information and visualizing subcellular organelles (Axelrod, 1989; Vrabioiu and Mitchison, 2006; Lazar et al., 2011). Based on the same setup, using a wide-field epifluorescence illumination microscope (Fig. 1), three techniques differing in their computational method are demonstrated as follows.

Fig. 1

Setup and principle of a wide-field epifluorescence illumination microscope

(a) The polarization of the excitation laser is continuously rotated by a half wave plate. Then, it is focused onto the back focal plane of the objective to generate uniform illumination with rotating polarization light. The fluorescent dipole is excited, and the fluorescence image is collected by an electron multiplying charge-coupled device (EMCCD) camera. As shown in the inset (b), the fluorescent molecule (such as green fluorescence protein, GFP) is linked to the target protein via C- and N-terminus. The dipole angle of the fluorophore reflects the orientation of the protein. Reprinted from Zhanghao et al. (2016), Copyright 2016, with permission from CC BY 4.0

Under polarization illumination, the fluorescent molecules with inherent dipole orientations (Fig. 2a) emit periodic signals, which can be described by a cosine-squared function of the rotation angle. Wide-field images are obtained after the polarization modulation is blurred (Fig. 2b), and average dipole orientation is easily deciphered by employing a trigonometric function over the optical diffraction limit. Hafi et al. (2014) developed a novel super-resolution technique, super-resolution by polarization demodulation (SPoD), with polarization demodulation achieved by sparse deconvolution, to yield superresolution information from confocal images (Fig. 2c). Zhanghao et al. (2016) combined the advantage of super-resolution in SPoD with the dipole orientation in FPM and exploited a super-resolution dipole orientation mapping (SDOM) method to resolve effective dipole orientation within the subdiffraction focal area. The SDOM algorithm fully takes advantage of polarization modulation via sparsity-enhanced deconvolution to estimate the intensity signal and least-squares estimation to extract the effective dipole orientation at a subdiffraction scale (Fig. 2d).

Fig. 2

Schematic of three techniques

(a) Two neighboring fluorescent dyes with different dipole orientations labeled in red and green emit periodic signals excited by rotating polarized light denoted by a double arrow. (b) FPM analyzes wide-field polarization modulation signals to decipher average dipole orientation over the optical diffraction limit. (c) Illustration of the SPoD principle. Super-resolution information is retrieved from confocal images by the SPEED algorithm. (d) The pipeline of SDOM. After SDOM reconstruction, the super-resolution image is obtained and the cosine curve is applied to fit the demodulated signal for every pixel to extract the orientation. Arrows indicate the direction of dipole orientation. Scale bar: 200 nm. References to color refer to the online version of this figure. Reprinted from Zhanghao et al. (2016), Copyright 2016, with permission from CC BY 4.0

We denote the polarization angle as α(t) at shift time t and the average orientation of pixel r as α₀(r) at t=0. Then we have α(0)=0 and α₀(r)∈[0, π]. The ith single molecule with orientation α_i emits photons with an intensity M_if₀(α(t)−α_i), where M_i is the maximum number of photons emitted by the ith molecule and f₀(α)=cos²α. Then n molecules at position or pixel r emit photons:

$$\begin{array}{*{20}c}{g(r,\,t) = \sum\limits_{i = 1}^n {{M_i}{f_0}(\alpha (t) - {\alpha_i})} \,\,\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad} \\ { = \sum\limits_{i = 1}^n {{{{M_i}} \over 2}\cos (2\alpha (t) - 2{\alpha_i}) +} \sum\limits_{i = 1}^n {{{{M_i}} \over 2}{.}}} \end{array}$$

((2))

The first sum in Eq. (2) can be regarded as vector addition, so we obtain the following formula:

$$g(r,\,t) = A\cos (2\alpha (t) - 2{\alpha _0}(r)) + B.$$

((3))

Orientational order imaging has made significant advances using polarized fluorescence via wide-field microscopy, and we use Fluorescence LC-PolScope (DeMay et al., 2011) as an example of FPM to present a method for analyzing the anisotropy of fluorescence. p ]Based on the observation whose fluorescence anisotropy is manifested in a sinusoidal variation, the photos I detected by the camera are depicted as a function of the polarization angle α₀:

$$I = {{{I_{\max}} + {I_{\min}} + ({I_{\max}} - {I_{\min}})\cos (2(\alpha (t) - {\alpha_0}(r)))} \over 2}{.}$$

((4))

In Fluorescence LC-PolScope, the intensity is recorded at four specified polarizer angles, 0°, 45°, 90°, and 135°. For each pixel, the anisotropy is calculated using the following expressions:

$$a = {I_0} - {I_{90}},\,b = {I_{45}} - {I_{135}},\,c = {I_0} + {I_{45}} + {I_{90}} + {I_{135}},$$

((5))

$${\rm{Polarization}}\,{\rm{ratio}} = {{{I_{\max}}} \over {{I_{\min}}}} = {{1 + 2\sqrt {{a^2} + {b^2}} /c} \over {1 - 2\sqrt {{a^2} + {b^2}} /c}},$$

((6))

$${\alpha_0}(r) = {1 \over 2}\arctan (b/a){.}$$

((7))

Eq. (7) solves the average fluorescent dipole orientation over diffraction limit spatial resolution. The polarization ratio is applied to interpret the anisotropy of fluorophores within the diffraction-limited volume and their mutual alignment.

In SPoD, the time-dependent fluorescent emission recorded by EMCCD can be written as

$$\begin{array}{*{20}c}{I(r,\,t)\sim {\rm{Poisson}}\left[ {{I_0}(t) \cdot \left( {\int\nolimits_0^T {\int {U(r - r\prime )}}} \right.} \right.\,\quad \quad \quad \quad \quad} \\ {\left. {\left. { \cdot f(t - t\prime )g(r\prime ,\,t\prime ){\rm{d}}r\prime {\rm{d}}t\prime + b(r)} \right)} \right],} \end{array}$$

((8))

where g(r, t) is the real average polarization modulation signal emitted by molecules at position r with time shift t. T corresponds to the period when the polarizer rotates by 180°. At each time point t, g(r, t) is blurred by the PSF U(r) of the microscope system, which can be mathematically expressed as a convolution process.

The cosine-squared function f(t)=cos²(πt/T) is used to depict the factor of the polarization angle in the process of observation. The unmodulated background b(r) is modeled as time-invariant within a short time. I₀(t) is a time-dependent periodic correction factor for the dichroic mirrors and varies somewhat with the polarization angles. The detected image I(r, t) is hence subject to a Poisson distribution with parameter μ(r, t) at each r and t. In SPoD, f(t) is integrated, meaning that the difference of g(r, t) in the temporal dimension is eliminated, so the reconstructed g(r, t) loses the cosine function property, which is of great value in estimating the orientation of position r.

In SDOM, r′ and t′ are assumed to be related instead of independent in Eq. (8) to extract the dipole orientation of the molecules. The observation process is modified as follows:

$$I(r,\,t)\sim {\rm{Poisson}}(\mu (r,\,t)),$$

((9))

where

$$\mu (r,\,t) = {I_0}(t)\left[ {\int {U(r - r\prime )g(r\prime ,t){\rm{d}}r\prime + b(r)}} \right]{.}$$

((10))

Both SPoD and SDOM intend to estimate g(r, t). g(r, t) and $\tilde b(r)$ are both independent and identically distributed, where $\tilde b(r)$ is the cosine transform of b(r) to accelerate computations. Maximum a posteriori (MAP) is applied with

$$\Pr \left( {g(r,\,t),\,\tilde b(r)\vert I(r,\,t)} \right),$$

((11))

and the optimization model $\underset{g,b}{\arg \min} \,L(g,b,I)$ is obtained, where $L(g,b,I) = \int\nolimits_0^T {\int {(\mu - I\log \mu ){\rm{d}}r{\rm{d}}t + {\lambda_1}{{\left\vert g \right\vert}_1} + {\lambda_2}{{\left\vert {\tilde b} \right\vert}_1}{.}}}$ (12)

The fast iterative shrinkage-thresholding algorithm (Beck and Teboulle, 2009) is employed to achieve fast minimization and g(r, t) is recovered from I(r, t). The super-resolution image can be obtained by integrating g(r, t) with respect to t, which is equivalent for SPoD and SDOM.

Beyond super-resolution imaging, SDOM applies least-squares estimation (Eq. (3)) to fit the demodulated polarization intensity signals to extract the dipole orientation, which brings FPM down to super-resolution microscopy. In SDOM, effective dipole orientation within a super-resolved focal volume, as a new dimension, is superimposed onto the intensity super-resolution images to provide more refined insight, so that structural imaging can be reduced to nanometric scales. The orientation uniform factor (OUF=A/B, where A and B are the orientation amplitude and super-resolution translation, respectively) is defined to describe the confidence of local orientation distribution and a higher OUF means a result of fewer heterogeneous fluorescent molecules. SDOM exhibits higher OUF than FPM due to the much smaller number of fluorescent molecules with a subdiffraction area, which makes the orientation mapping more robust and accurate.

Table 1 shows a comparison among FPM, SPoD, and SDOM. Note that these techniques share the same instrumentation that the fluorescent intensities are collected with modulation of excitation polarization.

Table 1 Comparison of the differences between different techniques

In conclusion, FPM is applicable to the analysis of orientational order and has been extensively studied for understanding biological organizations and functions. SDOM and SPoD use different imaging models. Though the sparsity-enhanced deconvolution used in SPoD is implemented as part of the SDOM algorithm, a combination of deconvolution and orientation mapping completes the SDOM algorithm. Only with both steps in the SDOM algorithm, can the polarization modulation information be fully used to obtain super-resolution images of intensity and dipole orientation.

4 Structured illumination microscopy

Whereas most super-resolution techniques, such as STED and PALM/STORM, achieve superresolution from modulations in the spatial domain, the principle of structured illumination microscopy (SIM) lies in the frequency domain (Gustafsson, 2000; Gustafsson et al., 2008; Dong et al., 2015; Yang et al., 2015; Lal et al., 2016; Zhou et al., 2016).

In SIM, the specimen is illuminated by the light of a structured pattern of intensity, and the pattern is typically a 1D sinusoidal function:

$${I_{\theta ,\varphi}}(r) = {I_{\rm{o}}}[1 + m\,\cos (2\pi {p_\theta}r + \varphi )],$$

((13))

where r is the spatial position, I_o is the peak illumination intensity, m is the modulation factor, p_θ is the sinusoidal illumination frequency vector in the reciprocal space, and φ is the phase of the illumination pattern. Subscript θ indicates the orientation of the sinusoidal illumination pattern. Because the high frequency of the specimen is shifted back into the OTF of the optical system, which is similar to the Moiré interference, one can detect a super-resolution image using this approach.

The observed emission distribution through the optical system is

$${D_{\theta ,\varphi}}(r) = [{S_r} \cdot {I_{\theta ,\varphi}}(r)] \otimes H(r) + N(r),$$

((14))

where S(r) denotes the fluorophore density of the specimen, H(r) is the PSF of the optical system, and N(r) is additive Gaussian (white) noise.

The Fourier transform of the observed image is given by

$$\begin{array}{*{20}c}{{{\tilde D}_{\theta ,\varphi}}(k) = [\tilde S(k) \otimes {{\tilde I}_{\theta ,\varphi}}(k)] \cdot \tilde H(k) + \tilde N(k)\quad \quad \quad} \\ { = {{{I_{\rm{o}}}} \over 2}\left[ {\tilde S(k) - {m \over 2}\tilde S(k - {p_\theta}){{\rm{e}}^{ - {\rm{i}}\phi}} - {m \over 2}\tilde S(k + {p_\theta}){{\rm{e}}^{{\rm{i}}\phi}}} \right]} \\ { \cdot \tilde H(k) + \tilde N(k){.}\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad} \end{array}$$

((15))

This equation suggests that the spectrum of the original image is replicated three times with the center on the origin, p_θ, and −p_θ. To separate these spectral components, at least three SIM images with different phase shifts are needed. Typically, φ₁=0°, φ₂=120°, and φ₃=240°.

The ungraded approximations of these spectral components are obtained by Wiener filtering of their corresponding noisy estimates obtained by solving the above system of equations.

Repeating the process with different orientations of the illumination pattern enables computation of the frequency content of a circular region whose radius is twice the radius of the OTF (Fig. 3), which corresponds to a twofold resolution enhancement.

Fig. 3

Image reconstruction principle of structured illumination microscopy

(a) Observable frequency content of a specimen in the reciprocal space is limited by optical system OTF, H(k). (b) Frequency content of a sinusoidally varying intensity pattern (vertical stripes, θ₁=0°) relative to optical system OTF, (c, d, e) Observed frequency content of a structured illuminated specimen is a linear combination of frequency content within three circular regions. Note that frequency contents within crescent-shaped yellow regions are now observable due to the Moiré effect and may be analytically computed. By illuminating the specimen sequentially with a sinusoidally varying illumination pattern at three different angular orientations 0°, 60°, and 120°, the information of specimen’s frequency that is twice that limited by optical system OTF may be obtained. (f) Separately obtained frequency contents are eventually merged and subsequently used to construct a super-resolved image of the specimen. Reprinted from Lal et al. (2016), Copyright 2016, with permission from IEEE. References to color refer to the online version of this figure

Conventionally, 2D SIM employs the previously discussed 9-frame reconstruction; however, SIM reconstructions with a reduced number of raw images have been demonstrated (Orieux et al., 2012; Li et al., 2015).

5 Image scanning microscopy

Confocal fluorescence microscopy is popular among biologists for its superior sectioning ability. In confocal microscopy, a laser is usually used to focus on a detection point, and a pinhole before the detector is conjugated to this point. The point-by-point scanning process is realized by a piezo stage or galvanometer. Theoretically, confocal microscopy could enhance the optical resolution by a factor of $\sqrt 2$. To achieve an acceptable signal level, however, a pinhole with a diameter of 1∼2 Airy units is usually used instead of an infinite small pinhole. Thus, the resolution enhancement is not significant compared to conventional epifluorescence microscopy.

Image scanning microscopy (ISM) (Müller and Enderlein, 2010; Sheppard et al., 2013) is an improved version of conventional confocal microscopy. ISM replaces the PMT pinhole with a CCD and by deconvolution, it could improve the resolution by a factor of 2. The image detected at position s on the CCD and at scanning position r in the sample can be given by

$$I(r,s)\int {{\rm{d}}r\prime} U(r - r\prime + s)E(r - r\prime )c(r\prime ),$$

((16))

where U(r) is the wide-field detection PSF, E(r) is the excitation intensity density, and c(r) is the spatial density of fluorescence emitters. In the step involving the rearrangement of pixels, we use s=r_d and r=r_r−αr_d, and integrate over r_d:

$$\begin{array}{*{20}c}{{I_{{\rm{eff}}}}({r_r}) = \int {{\rm{d}}r\prime} {\rm{d}}{r_d}U({r_r} - (\alpha - 1){r_d} - r\prime )} \\ {\quad \quad \cdot E({r_r} - \alpha {r_d} - r\prime )c(r\prime ),} \end{array}$$

((17))

where subscripts r and d denote the coordinates of the discrete data after scanning the sample.

When α=0, this intensity gives a wide-field image, and when α=1 it gives a nonconfocal scanning image. ISM requires α=0.5, and thus we have

$$\begin{array}{*{20}c}{{I_{{\rm{eff}}}}({r_r}) = \int {{\rm{d}}r\prime c(r\prime )\int {{\rm{d}}{r_d}U}} \left( {{r_r} + {1 \over 2}{r_d} - r\prime} \right)} \\ { \cdot E\left( {{r_r} - {1 \over 2}{r_d} - r\prime} \right){.}\quad} \end{array}$$

((18))

That is, I_eff(r_r)=c(r_r)*U_eff(r_r), where

$$\begin{array}{*{20}c}{{U_{{\rm{eff}}}}({r_r}) = \int {{\rm{d}}{r_d}U({r_r} + {r_d}/2)E({r_r} - {r_d}/2),}} \\ {{\rm{or}}\,{{\tilde U}_{{\rm{eff}}}}(m) = \tilde U(m/2)\tilde E(m/2){.}\,\,} \end{array}$$

((19))

Finally, the spectra of the effective PSF are broadened by a factor of 2, which is similar to structure illumination. A Wiener filter (Gonzalez and Woods, 2008) can be used for deconvolution $\tilde w(k){I_{{\rm{eff}}}}(k)$, where

$$\tilde w(k) = {{U(k/2)} \over { \ni + U(k/2)E(k/2)}},$$

((20))

and $\ni$ denotes a sufficiently small value.

The recent development of ISM includes spinning disk ISM and two-photon ISM. In spinning disk ISM, multiple focal points are scanned in parallel with the help of a microlens array, so the image speed can be significantly enhanced (Schulz et al., 2013). Two-photon ISM uses longer wavelength excitation for better sectioning ability (Ingaramo et al., 2014).

6 Super-resolution optical fluctuation imaging microscopy

Super-resolution optical fluctuation imaging (SOFI) is a technically simple but practically useful technique to achieve super-resolution imaging through analyzing the inherent blinking statistics of fluorophores. Dertinger et al. (2009) first proposed the concept of SOFI and experimentally demonstrated SOFI imaging on cellular structures. The basic principle of SOFI reconstruction is based on temporal correlation of fluctuation signals. As shown in Fig. 4a, we assume a sample consisting of N independently fluctuating fluorophores. The fluorescence intensity at a given position r and time t can be expressed as

$$F(r,t) = \sum\limits_{i = 1}^N {U(r - {r_i}){\varepsilon_i}{s_i}(t),}$$

((21))

where U(r) represents the PSF of the system, ε_i the molecular brightness, and s_i(t) the fluorescence fluctuation of individual fluorophores.

Fig. 4

Schematic of super-resolution optical fluctuation imaging (SOFI) reconstruction (Dertinger et al., 2009)

(a) A sample consisting of N individual fluctuating fluorophores; (b) Image sequence of two nearby fluorophores; (c) Time traces of individual pixels; (d) Temporal correlation functions of the extracted time traces; (e) SOFI-reconstructed image of two nearby fluorophores

Then, the 2nd-order autocorrelation function of the detected fluorescence fluctuation can be described as

$$\begin{array}{*{20}c}{G(r,\tau ) = {{\left\langle {\delta F(r,t + \tau )\delta F(r,t)} \right\rangle}_t}\,\quad \quad \quad \quad \quad \quad \quad \,\,\quad \,\,} \\ { = \sum\limits_i {{U^2}(r - {r_i})\varepsilon_i^2{{\left\langle {\delta {s_i}(t + \tau )\delta {s_i}(t)} \right\rangle}_t}{.}}} \end{array}$$

((22))

In Fig 4b, the image sequence of two nearby fluorophores can be collected by a camera. As can be seen, the two fluorophores cannot be well distinguished in the collected images. The fluctuation traces of individual pixels can be extracted and are shown in Fig. 4c. Subsequently, through pixel-wise analysis of the temporal fluctuations using the correlation function, i.e., Eq. (22), the two nearby fluorophores can be better resolved in the SOFI image shown in Fig. 4e.

As shown in Fig. 5, the SOFI technique can be employed on various platforms, e.g., wide-field, total internal reflection fluorescence (TIRF), and spinning-disk confocal microscopes (Dertinger et al., 2013). Geissbuehler et al. (2012) proposed balanced SOFI (bSOFI) combining multiple SOFI orders to improve spatial resolution and alleviate reconstruction artifacts. Zeng et al. (2015) proposed joint tagging SOFI (JT-SOFI) to achieve fast, high-fidelity super-resolution imaging. In addition, Zhang et al. (2015) synthesized a monomeric switchable fluorescent protein called Skylan-S, featuring high brightness, photostability, and contrast ratio for live-cell SOFI (Fig. 6). Currently, the achievable lateral and axial resolutions for SOFI are ∼50 and ∼300 nm, respectively.

Fig. 5

SOFI employed on different platforms, including wide-field, total internal reflection fluorescence (TIRF), and spinning-disk confocal microscopes

Reprinted from Dertinger et al. (2013), Copyright 2013, with permission from Cambridge University Press. References to color refer to the online version of this figure

Fig. 6

Applications of SOFI in cellular imaging (Geissbuehler et al., 2012; Zhang et al., 2015)

Reprinted from Geissbuehler et al. (2012), Copyright 2012, with permission from CC BY 4.0. References to color refer to the online version of this figure

7 Single-molecule localization microscopy

Single-molecule localization microscopy (SMLM) can achieve very high spatial resolution. (f)PALM/STORM is the typical technique for SMLM. In 2006, Eric Betzig, Samuel Hess, and Xiaowei Zhuang experimentally demonstrated the concept of (f)PALM/STORM (Betzig et al., 2006; Hess et al., 2006; Rust et al., 2006). In PALM/STORM imaging, through sequentially switching the on-off states of individual fluorophores, most of the fluorophores can be spatially isolated. Subsequently, the centers of isolated fluorophores can be determined by 2D Gaussian function fitting. The localization precision can be given by

$${\sigma_{\mu i}} = \sqrt {{{S_i^2} \over N} + {{{a^2}/12} \over N} + {{8\pi S_i^4{b^2}} \over {{a^2}{N^2}}}} ,$$

((23))

where N represents the detected photon number, a is the effective pixel size of the detector, and b describes the background noise. As a result, increasing the detected photon number can significantly improve the localization precision. The final PALM/STORM image can be reconstructed by superimposing all the localized image frames, yielding a super-resolution image. The currently achievable lateral and axial resolutions for PALM/STORM are 10–20 and 50–60 nm, respectively.

8 Bayesian super-resolution microscopy

In localization techniques such as PALM/STORM (Betzig et al., 2006; Rust et al., 2006; Huang et al., 2010; Hao et al., 2013), the fluorescence emission from individual molecule fluorophores should be nonoverlapping. In certain frames, a limited number of fluorescence points can be localized; thus, PALM/STORM needs tens of thousands of frames of raw data or even more (Rust et al., 2006; Huang et al., 2010; Gao et al., 2016). Switching a large number of probes into a nonemitting state is required to achieve the nonoverlapping fluorophore emission necessary for normal localization microscopy analysis methods. Using a near-ultraviolet laser can cause a limited population of fluorophores to generate fluorophores. Also, relatively high intensity illumination (1–10 kW/cm²) at a single imaging wavelength can, under suitable chemical conditions, achieve conventional localization nanoscopy. Both near-ultraviolet and high-intensity illumination can damage the cells, which hindered the application of conventional localization nanoscopy in long-term live cell imaging. One new localization super-resolution model, the Bayesian analysis of blinking and bleaching (3B analysis) method, was used by Cox et al. (2012). 3B nanoscopy models the most likely distribution of molecule locations by studying the fluctuation/blinking and bleaching of the fluorophores in which these fluorophores are overlapping. Compared with PALM/STORM, 3B can allow multiple emitters within the focal volume in the ON state; statistical blinking can be employed constructively to generate the super-resolution image. The 3B method greatly reduces the number of raw data frames required to achieve one resolution-enhanced image. A temporal resolution of 4 frames/s with sub-50-nm resolution can be achieved during dynamic cell processes.

Because the 3B method requires heavy computation to reconstruct a large number of ‘molecules’ from a relatively small image dataset, cloud computing holds the potential to increase the speed of 3B microscopy single-molecule localization and reconstruction (Hu et al., 2013a). Considering the advantage of low phototoxicity and high spatiotemporal resolution, both spatial organization and temporal dynamics of Pol II clusters were observed with Bayesian nanoscopy by Chen et al. (2016). Also, Yang et al. (2016b) performed a 3B subdiffraction imaging measurement (300 frames) of a microtubule sample in a HeLa cell labeled with quantum dots. Furthermore, to enhance the signal-to-noise ratio (SNR) of single-molecule events in one frame, light sheet illumination was employed to enhance the SNR for single-molecule super-resolution imaging in the nucleus. A prism-coupled light sheet Bayesian microscope (Hu et al., 2013b) was set up to render deep-cell subdiffraction imaging of heterochromatin in live embryonic stem cells of a human.

9 Stimulated emission depletion microscopy

Stimulated emission depletion (STED) theoretically achieves diffraction-unlimited super-resolution imaging by virtue of stimulated emission. Hell and Wichmann (1994) proposed the concept of STED fluorescence microscopy. Subsequently, STED was first experimentally demonstrated by Klar et al. (2000). In theory, STED can reconstruct superresolution images without the requirement of additional postprocessing. However, in practice, owing to the noise and image distortions during image acquisition, additional image processing algorithms, e.g., deconvolution and denoising, are usually incorporated to further enhance the spatial resolution and contrast of reconstructed images. In commercial STED microscopes, the Huygens STED deconvolution method is commonly employed as the image postprocessing method (Schoonderwoert et al., 2013). Huygens deconvolution can perform nonlinear iterative deconvolutions on STED raw image data, improving both the spatial resolution and SNR of super-resolution images. The Huygens deconvolution algorithm makes a comparison between the measured image and the estimate of the image, which is the convolution between the estimate of the object and the PSF. Subsequently, the algorithm computes a correction image and the quality measure. After iteratively applying the correction image, the algorithm stops if sufficient quality is reached. As a consequence, the STED deconvolved images present better spatial resolution and contrast compared to their confocal and the conventional STED counterparts.

10 Translation microscopy

Recently, Qiu et al. (2016) presented a novel and technically simple way of achieving super-resolution microscopy, called ‘translation microscopy (TRAM)’. TRAM obtains a super-resolution image through simply translating the specimen/detector during image acquisition. A high-resolution (HR) image can be calculated from a series of low-resolution (LR) images that are translated with one another in the X-Y plane. In theory, TRAM tries to solve an inverse problem by minimizing an energy function. The minimization problem was solved by an iteratively reweighted least-squares (IRLS) method. When the HR image estimations between two adjacent iterations present negligible differences, the iteration stops and the result is regarded as the restored superresolution image. As shown in Fig. 7, the TRAM-restored images exhibit significant background rejection and resolution enhancement. TRAM can achieve up to 7-fold resolution improvement compared to conventional diffraction-limited microscopy. One of the advantages of TRAM is that it can be adapted to most types of conventional microscopes with little or no hardware modification, making TRAM accessible to virtually any laboratory with a conventional microscope configuration.

Fig. 7

Principle and applications of translation microscopy

(a) Illustration of translation microscopy (TRAM); (b) Diffraction-limited image of a cell sample staining actin, microtubules, and nuclei; (c) TRAM-restored image using 60 translational diffraction-limited images; (d) Magnified region of microtubules and DAPI-stained nuclei in (b); (e) TRAM-restored image of the same region in (d); (f) Intensity profiles of the microtubules (yellow line) and DAPI (pink line) in (e). Reprinted from Qiu et al. (2016), Copyright 2016, with permission from CC BY 4.0. References to color refer to the online version of this figure

11 Conclusions

Super-resolution microscopy is achieved through physical and chemical changes of fluorescent molecules and heavily relies on the computation methods. In this work, we reviewed several popular superresolution techniques in which advanced computational techniques are embedded. A tabular comparison of different super-resolution microscopy techniques is presented in Table 2. We hope that this review can serve as a bridge to connect the superresolution microscopy community and the computation community so that novel computational techniques can be applied toward better resolution, improved accuracy, and faster image processing. The currently developed super-resolution techniques may also benefit the computational image processing techniques in their respective applications, when the image data shares a similar digital nature through current or modified instrumentation.

Table 2 Comparison of different super-resolution microscopy techniques