Holographic deconvolution microscopy for highresolution particle tracking
Abstract.
RayleighSommerfeld backpropagation can be used to reconstruct the threedimensional light field responsible for the recorded intensity in an inline hologram. Deconvolving the volumetric reconstruction with an optimal kernel derived from the RayleighSommerfeld propagator itself emphasizes the objects responsible for the scattering pattern while suppressing both the propagating light and also such artifacts as the twin image. Bright features in the deconvolved volume may be identified with such objects as colloidal spheres and nanorods. Tracking their thermallydriven Brownian motion through multiple holographic video images provides estimates of the tracking resolution, which approaches 1 in all three dimensions.
Inline holographic video microscopy offers timeresolved information regarding the threedimensional distribution of matter that scatters a beam of laser light (1); (2). The scattered light interferes with the unscattered portion of the beam in the focal plane of an otherwise conventional microscope. A video camera then records the intensity of the magnified interference pattern with time resolution set by the exposure time and frame rate of the camera. Each holographic snapshot in the resulting video stream contains, in principle, comprehensive information regarding the threedimensional distribution of matter.
Variants of this technique differ principally in how the holographic snapshots are analyzed. Fitting recorded holograms to results based on the exact theory of light scattering (3); (4) has been used to track colloidal spheres with nanometer resolution (3); (4); (5), to characterize individual spheres' sizes, refractive indexes (3); (4); (6); (7), and porosities (8), and to detect molecularscale coatings on micrometerscale colloidal substrates (4); (6). This wealth of information comes at a price. The nonlinear leastsquares fits are computationally intensive, and must be targeted specifically for the kinds of samples being analyzed.
Inline holograms also may be analyzed more generally and more efficiently by numerically reconstructing the threedimensional light field (1); (2); (9) with scalar diffraction integrals (10). Such volumetric reconstructions do not commonly account for the physical properties of the scattering medium. Rather, the medium is treated as an isotropic and homogeneous dielectric. Bright features in the reconstruction then are identified with discrete objects in the sample (1); (2); (5), particularly those that are smaller than the wavelength of light (5). Dynamical measurements based on tracking such features through holographic video sequences have demonstrated 10 resolution for threedimensional location of colloidal spheres (5) and nanorods (11), albeit with large systematic axial offsets in some cases (5).
Features in holographic reconstructions can be emphasized by deconvolving the reconstructed light distribution with the volumetric pointspread function for the reconstruction process (12); (13). Here, we demonstrate and quantitatively assess the tracking resolution of holographic deconvolution microscopy using the RayleighSommerfeld diffraction integral both to reconstruct the volumetric intensity distribution and also to deconvolve it. For micrometerscale colloidal spheres, the results suggest typical inplane resolution approaching 1 and axial resolution of 10 . Threedimensional tracking of metaloxide nanorods yields comparably good results for locating the center of mass, and orientation resolution in three dimensions.
Our inline holographic microscope has been described previously (2). It consists of a commercial inverted microscope stand (Nikon TE2000U) outfitted with a highnumericalaperture oilimmersion objective (Nikon PlanApo, , NA 1.4). The conventional illuminator is replaced with the collimated beam from a fibercoupled diode laser (Coherent Cube) operating at a vacuum wavelength of 445 . The linearly polarized beam has a total power of 15 spread uniformly over . Holographic images are captured by a lownoise grayscale video camera (NEC TI 324AII) and are recorded as an uncompressed digital video stream at 30 frames per second with a total system magnification of 135 pixel. The camera's 0.5 exposure time is fast enough to avoid measurable effects of motion blurring (4); (14); (15); (16).
The incident plane wave,
(1) 
is assumed to propagate along with an realvalued amplitude that may depend on position in the transverse plane, and uniform polarization . The scattered wave,
(2) 
propagates in three dimensions with complex amplitude and spatially varying polarization . Their superposition in the focal plane yields the interference pattern
(3)  
(4) 
Deliberately placing the scatterer well above the focal plane ensures both that polarization rotations are small, , and also that the scattered wave is substantially less intense than the illumination. Normalizing by the illumination's intensity distribution then yields
(5) 
where the reduced scattered field is . Dropping from the definition of simplifies the analysis that follows at the cost of ignoring interference due to multiple scatterers. It therefore limits the complexity of the samples to which this formalism may be applied. In practice, the background image, , can be obtained either by moving the sample out of the field of view, or by taking a running median filter of a timeevolving sample.
If the complex scattered field were completely specified in the focal plane, it could be reconstructed at height as the convolution
(6) 
of the scattered amplitude in the focal plane with the RayleighSommerfeld propagator (10)
(7) 
where . The sign convention for accounts for the object's position upstream of the focal plane. Equation (6) may be rewritten with the Fourier convolution theorem as
(8) 
where
(9) 
is the inplane Fourier transform of and where
(10) 
is the Fourier transform of (10); (17); (18). To use this formalism to reconstruct the scattered field, we note that the Fourier transform of is
(11) 
From this,
(12) 
may be recognized as the superposition of the scattered field at height above the focal plane and a spurious field due to the object's mirror image in the focal plane, which is known as the twin image. The twin image's influence on the reconstructed field may be reduced by moving the sample away from the focal plane.
In the additional approximation that the illumination is uniform, the reconstructed field is
(13) 
The associated intensity, , is an estimate for the scattered light's intensity at height above the focal plane. This reconstruction differs from a numerically refocused image (1) because it comprises only the scattered field and not its superposition with the incident field.
The reconstruction also is not simply a representation of the object responsible for the scattered field, or even of the scattered field itself. The projection operation in Eq. (6) is formulated for light propagating through a homogeneous medium, and so does not account for the optical properties of any objects in the medium. The reconstructed field thus includes artifacts upstream of each scattering center in the sample. These artifacts and the twin image of the reconstructed field are both clearly visible in the volumetric reconstructions of colloidal spheres and a copperoxide nanorod (11); (19) in Figs. 1(a) and (b), respectively. These artifacts, together with outoffocus images tend to obscure features of interest in the reconstructed volume. Neither the nature nor the number of the scattering objects is immediately evident from the reconstructions in Fig. 1(a) or Fig. 1(b). These sources of blurring therefore hinder efforts to identify and track colloidal particles.
Most of these distractions can be removed by deconvolving the reconstructed data with the point spread function used for the reconstruction. Previous reports (12); (13) have used pointspread functions for particular objects in an effort to optimize detection resolution. Our deconvolution scheme instead is motivated by the observation that the RayleighSommerfeld formalism transforms an ideal pointlike focal caustic (20) at height above the focal plane into the threedimensional intensity distribution . The caustic's twin image, similarly, is projected into . Deconvolving with therefore should reduce the full threedimensional scattering pattern reconstructed from the hologram to the set of focal caustics created by the sample. Twin images also will be reduced to focal caustics, but on the other side of the effective focal plane, which will not be reconstructed, and so will not be seen. Using the RayleighSommerfeld pointspread function in this way also eliminates the need for calibration measurements (12); (13).
Deconvolving with is most easily implemented with the Fourier convolution theorem:
(14) 
where is the threedimensional Fourier transform of , is the Fourier transform of , and is a small factor chosen for numerical stability. The deconvolved intensity, is obtained as the inverse Fourier transform of . Figures 1(c) and (d) (Media 1) were obtained in this way from the data in Figs. 1(a) and (b), respectively, and are rendered with the same color and transparency tables. Blurring from artifacts and outoffocus images is strongly suppressed in the deconvolved volumes, leaving distinct and wellresolved bright features that we associate with the focal caustics created by the samples. For example, the sample in Fig. 1(c) now can be recognized to consist of 9 colloidal spheres arranged in a bodycentered cubic lattice, while that in Fig. 1(d) is quite clearly an inclined rod. These results were obtained with , and comparably good contrast and resolution were obtained for .
The threedimensional positions of the focal caustics can be recovered by analyzing moments of the threedimensional intensity distribution (21). For objects with dimensions comparable to the wavelength of light, furthermore, focal caustics should lie substantially on the objects themselves. This suggests that bright features in the deconvolved reconstruction also track the shape, size, position and orientation of such objects (5); (11). A similar analysis is used to track objects in a volume deconvolved with a targeted pointspread function (12). This approach requires the pointspread function to measured for the target object, or else to be computed based on a priori knowledge.
Tracking Brownian objects as they diffuse enables us to measure the accuracy and resolution attainable with holographic deconvolution microscopy (5); (11); (14); (15); (21). The apparent meansquare displacement of a colloidal sphere freely diffusing along ,
(15) 
increases linearly with time according to the familiar EinsteinSmoluchowski result, but is offset by the measurement error for centroid location along that axis (21). In this, we assume that the camera's shutter is fast enough that motion blurring may be ignored (14); (15); (16). The data in Fig. 2(a) were obtained for a diameter colloidal polystyrene sphere (Bangs Laboratories, lot number 912) diffusing in water at room temperature, . The three singleaxis diffusion coefficients, , () are consistent with each other and also are consistent with the StokesEinstein result given and using the radius obtained from LorenzMie characterization (3). Consistency among the three diffusion coefficients confirms that the sphere was far enough from the walls of the sample container that hydrodynamic coupling may be ignored. The fit values for the centroid tracking errors, , and , are a factor of two smaller than the corresponding errors obtained with RayleighSommerfeld reconstruction without deconvolution (5), and are a factor of two larger than those obtained with LorenzMie fits to the same holograms (5).
Although both the RayleighSommerfeld reconstruction and the subsequent deconvolution are numerically intensive, they can be less costly than LorenzMie fitting, particularly when multiple particles are in the field of view. Our implementation in the IDL programming language requires 30 seconds to locate the 9 spheres in Fig. 1(c) through fits to the LorenzMie theory (5) on a multicore processor running at 3 , but only 20 seconds for the present approach. Unlike fitting to LorenzMie theory, holographic deconvolution does not yield highresolution measurements of sphere radius or refractive index. The estimated axial position, furthermore, includes a systematic offset that depends on the size and composition of the object (5). This complicates measurements of threedimensional separations. Any such concerns are offset, however, by the generality of holographic deconvolution microscopy, which requires no foreknowledge of an object's shape or composition.
For instance, the nanorod's deconvolved reconstruction in Fig. 1(d) (Media 1) features a roughly cylindrical volume of length , whose length remains unchanged through a sequence of 5,000 snapshots as the nanorod rotates in threedimensions. It is tempting to identify the center of this cylinder with the nanorod's position, and the orientation of its axis with the nanorod's orientation. Figure 2(b) shows the meansquared displacement
(16) 
of the orientational unit vector obtained in this way. The nanorod's apparent orientation and centerofmass position are extracted from the deconvolved reconstruction using a peaktracking skeletonization algorithm originally developed (11) to analyze volumetric reconstructions of the type presented in Fig. 1(b). The solid curve in Fig. 2(b) is a fit to
(17) 
for the rod's rotational diffusion coefficient and the measurement error in the nanorod's threedimensional orientation. The latter figure is consistent with an error of in the rod's orientation, which improves on previous results by more than a factor of two.
The nanorod's translational diffusion is difficult to analyze in the laboratory frame because its viscous drag coefficient depends on its orientation, which is always changing (22). In the corotating frame, however, its meansquared displacement should evolve linearly in time,
(18)  
(19) 
when projected along and transverse to the measured orientations, respectively. The results plotted in Fig. 2(c) are consistent with this interpretation, and yield and .
The three diffusion coefficients for a freely diffusing rod,
(20)  
(21)  
(22) 
depend on the rod's length and diameter , and also on a geometric factor (23). From these, we obtain , which is consistent with the optical measurement, , and . The associated tracking error for the rod's centroid is 10 in the plane and 50 along the optical axis, averaged over orientations.
The success of holographic deconvolution microscopy for tracking colloidal spheres and nanorods with nanometer resolution does not necessarily extend to more general objects, particularly those with dimensions larger than the wavelength of light. Equation (14) ignores interference due to light emanating from multiple sources. Provided the scattering centers are separated by multiple wavelengths of light, however, interference effects manifest themselves principally in the dimmer regions of the reconstructed intensity distribution, well away from interesting intensity maxima. When these conditions are not met, however, Eq. (14) can introduce artifacts of its own. The data in Fig. 1(d), for example, show the reconstruction of a lithographically defined colloidal particle in the form of the letter “X” (24). Crafted from polymethylmethacrylate, this dielectric particle has features roughly one micrometer across, and thus scatters the incident plane wave illumination into a pattern that depends strongly on the particle's orientation. The deconvolved volumetric reconstruction emphasizes caustics in that scattering pattern, rather than the shape itself, and so its true shape is not readily apparent. The same will be true of more highly structured samples such as biological cells. The neglect of the scattered intensity in the derivation of Eq. (5) also may have contributed to a loss of definition.
When applied with appropriate care, deconvolving the RayleighSommerfeld reconstruction of an inline hologram with the RayleighSommerfeld pointspread function yields precise and accurate information for threedimensional particle tracking. This technique excels for highly symmetric colloids such as spheres whose caustics precisely track the particle's position, albeit with an axial offset (5). Objects such as nanorods that have dimensions smaller that the wavelength of light similarly can be tracked with nanometerscale resolution. Because the projected caustic coincides with the physical position of the object in such cases, the measurement's accuracy can approach its precision. For all such systems, deconvolution with the RayleighSommerfeld pointspread function greatly simplifies threedimensional tracking by deemphasizing extraneous features of the threedimensional scattering pattern in favor of its singularities. The measurements on model samples that we have presented confirm that deconvolution can substantially enhance tracking resolution compared with reconstruction alone (5).
We are grateful to Tom Mason for providing a sample of LithoParticles. This work was supported in part by the QORS Program of DARPA and in part by the National Science Foundation through grant DMR0922680. The software used in this study is available for download under the terms of the GNU Public License at http://physics.nyu.edu/grierlab/software.html
References

(1)
J. Sheng, E. Malkiel, and J. Katz, “Digital holographic microscope for measuring threedimensional particle distributions and motions,” Appl. Opt. 45, 3893–3901 (2006).

(2)
S.H. Lee and D. G. Grier, “Holographic microscopy of holographically trapped threedimensional structures,” Opt. Express 15, 1505–1512 (2007).

(3)
S.H. Lee, Y. Roichman, G.R. Yi, S.H. Kim, S.M. Yang, A. van Blaaderen, P. van Oostrum, and D. G. Grier, “Characterizing and tracking single colloidal particles with video holographic microscopy,” Opt. Express 15, 18,275–18,282 (2007).

(4)
F. C. Cheong, B. Sun, R. Dreyfus, J. AmatoGrill, K. Xiao, L. Dixon, and D. G. Grier, “Flow visualization and flow cytometry with holographic video microscopy,” Opt. Express 17, 13,071–13,079 (2009).

(5)
F. C. Cheong, B. J. Krishnatreya, and D. G. Grier, “Strategies for threedimensional particle tracking with holographic video microscopy,” Opt. Express 18, 13,563–13,573 (2010).

(6)
F. C. Cheong, K. Xiao, and D. G. Grier, “Characterization of individual milk fat globules with holographic video microscopy,” J. Dairy Sci. 92, 95–99 (2009).

(7)
K. Xiao and D. G. Grier, “Multidimensional optical fractionation with holographic verification,” Phys. Rev. Lett. 104, 028302 (2010).

(8)
F. C. Cheong, K. Xiao, D. J. Pine, and D. G. Grier, “Holographic characterization of individual colloidal spheres' porosities,” Soft Matter in press (2011). DOI: 10.1039/c1sm05577a

(9)
B. Rappaz, P. Marquet, E. Cuche, Y. Emery, C. Depeursinge, and P. J. Magistretti, “Measurement of the integral refractive index and dynamic cell morphometry of living cells with digital holographic microscopy,” Opt. Express 13, 9361–9373 (2005).

(10)
J. W. Goodman, Introduction to Fourier Optics, 3rd ed. (McGrawHill, New York, 2005).

(11)
F. C. Cheong and D. G. Grier, “Rotational and translational diffusion of copper oxide nanorods measured with holographic video microscopy,” Opt. Express 18, 6555–6562 (2010).

(12)
T. Latychevskaia, F. Gehri, and H.W. Fink, “Depthresolved holographic reconstructions by threedimensional deconvolution,” Opt. Express 21, 22,527–22,544 (2010).

(13)
Y. Cotte, M. F. Toy, N. Pavillon, and C. Depeursinge, “Microscopy image resolution improvement by deconvolution of complex fields,” Opt. Express 18, 19,462–19,478 (2010).

(14)
T. Savin and P. S. Doyle, “Role of finite exposure time on measuring an elastic modulus using microrheology,” Phys. Rev. E 71, 041106 (2005).

(15)
T. Savin and P. S. Doyle, “Static and dynamic errors in particle tracking microrheology,” Biophys. J. 88, 623–638 (2005).

(16)
L. Dixon, F. C. Cheong, and D. G. Grier, “Holographic particlestreak velocimetry,” Opt. Express 19, 4393–4398 (2011).

(17)
G. C. Sherman, “Application of the convolution theorem to Rayleigh's integral formulas,” J. Opt. Soc. Am. 57, 546–547 (1967).

(18)
U. Schnars and W. P. O. Jüptner, “Digital recording and reconstruction of holograms,” Meas. Sci. Tech. 13, R85–R101 (2002).

(19)
T. Yu, C. H. Sow, A. Gantimahapatruni, F. C. Cheong, Y. W. Zhu, K. C. Chin, X. J. Xu, C. T. Lim, Z. X. Shen, J. T. L. Thong, and A. T. S. Wee, “Patterning and fusion of CuO nanorods with a focused laser beam,” Nanotechnology 16, 1238–1244 (2005).

(20)
J. F. Nye, Natural Focusing and Fine Structure of Light (Institute of Physics Publishing, Bristol, UK, 1999).

(21)
J. C. Crocker and D. G. Grier, “Methods of digital video microscopy for colloidal studies,” J. Colloid Interface Sci. 179, 298–310 (1996).

(22)
Y. Han, A. M. Alsayed, M. Nobili, J. Zhang, T. C. Lubensky, and A. G. Yodh, “Brownian motion of an ellipsoid,” Science 314, 626–630 (2009).

(23)
M. Doi and S. F. Edwards, The Theory of Polymer Dynamics (Clarendon Press, Oxford, 1986).

(24)
C. J. Hernandez and T. G. Mason, “Colloidal alphabet soup: Monodisperse dispersions of shapedesigned LithoParticles,” J. Phys. Chem. 111, 4477–4480 (2007).