Machine-learning approach to holographic particle characterization

Aaron Yevick, Mark Hannel and David G. Grier
Department of Physics and Center for Soft Matter Research, New York University, New York, NY 10003, USA
Abstract

Holograms of colloidal dispersions encode comprehensive information about individual particles’ three-dimensional positions, sizes and optical properties. Extracting that information typically is computationally intensive, and thus slow. Here, we demonstrate that machine-learning techniques based on support vector machines (SVMs) can analyze holographic video microscopy data in real time on low-power computers. The resulting stream of precise particle-resolved tracking and characterization data provides unparalleled insights into the composition and dynamics of colloidal dispersions and enables applications ranging from basic research to process control and quality assurance.

I Introduction

Holograms of colloidal spheres obtained with holographic video microscopy ((1); (2)) can be interpreted with predictions of the Lorenz-Mie theory of light scattering ((3)) to track each particle in three dimensions, and to measure its size and refractive index ((4)). State-of-the-art implementations ((4); (5); (6); (7)) can locate a sphere and resolve its radius both to within a few nanometers, and can determine its refractive index to within a part per thousand ((8); (9); (10)). The cost of this powerful technique is the computational burden of fitting each hologram pixel-by-pixel to theoretical predictions ((4); (11)). Here, we demonstrate that techniques of machine learning can reduce the processing time by a factor of a thousand, yielding real-time performance.

Our approach to fast holographic characterization, depicted schematically in Fig. 1, employs the support vector machine (SVM) algorithm ((12)) to compare experimental measurements with pre-computed predictions of the Lorenz-Mie theory ((3); (4); (13)). Whereas nonlinear fitting typically requires more than a second on a 1 Gflop computer, a trained SVM can estimate the size, refractive index or axial position of a micrometer-scale sphere in under a millisecond on the same hardware.

Figure 1: Colloidal characterization by holographic microscopy and machine learning. Colloidal spheres flowing down a microfluidic sample scatter light from a collimated laser beam to form an in-line hologram. Features in the beam are identified, and their radial profiles presented to support vector machines (SVMs) that compare them with a library of training data to estimate each spheres’ radius apa_{p} and refractive index npn_{p}. The scatter plot shows results for 2,500 spheres drawn at random from a mixture of four different types of spheres. Each point is colored by the local density of data points, ρ(ap,np)\rho(a_{p},n_{p}).

II Fast Holographic Characterization with Machine Learning

The in-line holographic microscope used for these studies ((2); (4); (10)) illuminates the sample with a linearly polarized collimated laser beam (Coherent Cube, 20 mW) at a vacuum wavelength of λ=447nm\lambda=447~{}\text{nm}. The fluence of the 3 mm-diameter beam is comparable to that of a conventional microscope illuminator. Optical forces and light-induced heating therefore are negligible. Light scattered by a sphere propagates to the focal plane of a custom-built video microscope ((10)) where it interferes with the undiffracted portion of the original beam. The microscope magnifies this interference pattern onto the detector of a greyscale video camera (NEC TI 324AII), which records its intensity with a system magnification of 135 nm/pixel. Each snapshot in the video stream constitutes a hologram of the particles in the channel.

The electric field of the incident beam at position 𝐫\mathbf{r} in the focal plane may be modeled as a plane wave with spatial dependence

𝐄0(𝐫)=u0(𝐫)eiφ0(𝐫)eikzx^,\mathbf{E}_{0}(\mathbf{r})=u_{0}(\mathbf{r})\,e^{i\varphi_{0}(\mathbf{r})}\,e^% {ikz}\,\hat{x}, (1)

where k=2πnm/λk=2\pi n_{m}/\lambda is the wavenumber in a medium of refractive index nmn_{m}, and where u0(𝐫)u_{0}(\mathbf{r}) and φ0(𝐫)\varphi_{0}(\mathbf{r}) account for small variations in the beam’s amplitude and phase profiles, respectively.

A particle located at 𝐫p\mathbf{r}_{p} relative to the center of the focal plane scatters the incident illumination, 𝐄0(𝐫p)\mathbf{E}_{0}(\mathbf{r}_{p}), to the focal plane as

𝐄s(𝐫)=E0(𝐫p)𝐟s(k(𝐫-𝐫p)|ap,np),\mathbf{E}_{s}(\mathbf{r})=E_{0}(\mathbf{r}_{p})\,\mathbf{f}_{s}\!\left(k(% \mathbf{r}-\mathbf{r}_{p})|a_{p},n_{p}\right), (2)

where 𝐟s(k𝐫|ap,np)\mathbf{f}_{s}(k\mathbf{r}|a_{p},n_{p}) is the Lorenz-Mie scattering function ((3)) that describes how a sphere of radius apa_{p} and refractive index npn_{p} scatters an x^\hat{x}-polarized plane wave. The measured intensity then may be modeled as

I(𝐫)=|𝐄0(𝐫)+𝐄s(𝐫)|2.I(\mathbf{r})=\left|\mathbf{E}_{0}(\mathbf{r})+\mathbf{E}_{s}(\mathbf{r})% \right|^{2}. (3)

Normalizing the recorded hologram by I0(𝐫)=|𝐄0(𝐫)|2=u02(𝐫)I_{0}(\mathbf{r})=\left|\mathbf{E}_{0}(\mathbf{r})\right|^{2}=u_{0}^{2}(% \mathbf{r}) suppresses spurious structure in the illumination and yields a functional form for the normalized hologram ((4); (10)),

b(𝐫)=I(𝐫)I0(𝐫)|x^+e-ikzp𝐟s(k(𝐫-𝐫p)|ap,np)|2,b(\mathbf{r})=\frac{I(\mathbf{r})}{I_{0}(\mathbf{r})}\approx\left|\hat{x}+e^{-% ikz_{p}}\mathbf{f}_{s}(k(\mathbf{r}-\mathbf{r}_{p})|a_{p},n_{p})\right|^{2}, (4)

that can be calculated with standard software packages ((14)).

Previous implementations of Lorenz-Mie microscopy ((4)) fit Eq. (4) to measured holograms using apa_{p}, npn_{p} and 𝐫p\mathbf{r}_{p} as adjustable parameters. These fits are exquisitely sensitive to errors in the particle’s in-plane position, and so must be performed over the entire two-dimensional intensity distribution ((4)). Here, we instead use Eq. (4) to train support vector machines, which then are able to estimate apa_{p}, npn_{p} and zpz_{p} from a hologram’s one-dimensional radial profile. We obtain these profiles from measured holograms by averaging around centers of rotational symmetry ((13)) with single-pixel resolution, yielding 100-point data vectors. The microscope’s focal plane is adjusted so that the interference fringes in a typical sphere’s hologram extend to roughly this scale. Averaging over angles to obtain a radial profile reduces the dimensionality of the analysis problem and accounts in part for our method’s computational efficiency.

Our SVMs are implemented with scikit-learn, an open-source machine learning software package ((15)) that builds upon the libsvm library of Support Vector Machine algorithms ((16); (17)). Each SVM computes one output parameter from an input vector consisting of a radial profile, b(r)b(r), that has been digitized into 100 single-pixel bins. Characterizing and tracking a colloidal particle therefore requires three SVMs, one for each of apa_{p}, npn_{p} and zpz_{p}. Figure 1 schematically represents this process for estimating apa_{p} and npn_{p}.

An SVM computes its output by comparing b(r)b(r) with sets of training data, bn(r)b_{n}(r), that are obtained from Eq. (4) over a range of values of apa_{p}, npn_{p} and zpz_{p}. Each training set constitutes one support vector in the space spanned by these parameters. To facilitate these comparisons, we construct SVMs with radial basis functions ((12)),

kn(b)=exp(-γ|bn(r)-b(r)|2dr),k_{n}(b)=\exp\left(-\gamma\int\left|b_{n}(r)-b(r)\right|^{2}\,dr\right), (5)

that quantify the similarity of the experimental input with the nn-th support vector. The sensitivity of this comparison is set by γ\gamma, with larger values favoring more precise results at the cost of requiring more support vectors to span the parameter space. Given a value of γ\gamma, the training process determines a set of weights ωn\omega_{n} and an offset s0s_{0} such that the weighted sum,

s~(b)=nωnkn(b)+s0,\tilde{s}(b)=\sum_{n}\omega_{n}k_{n}(b)+s_{0}, (6)

constitutes an estimate for the parameter, s(b)s(b), that characterizes the input data vector, b(r)b(r). In general, errors in s~(b)\tilde{s}(b) depend smoothly on γ\gamma ((12)).

To prevent overfitting, the weights ωn\omega_{n} are constrained to have magnitudes less than a maximum value, which conventionally is denoted by CC ((12)). Optimizing an SVM with larger values of CC improves its ability to recognize its training data, but renders it less able to interpolate smoothly between its support vectors when presented with novel or noisy inputs. Some candidate support vectors may be assigned small weighting factors in optimizing s~(b)\tilde{s}(b) over a corpus of training data; these are automatically eliminated from the SVM ((12)). The choice of γ\gamma and CC thus determines which support vectors are included in the SVM, and their relative importance for computing the output. Because this process is nonlinear, optimal values are obtained by exhaustive search over the range 0.1γ100.1\leq\gamma\leq 10 and 0.1C1,0000.1\leq C\leq 1,000. Statistically indistinguishable results are obtained in the present application for values of γ\gamma and CC that vary from their optimal values by up to fifty percent.

We trained SVMs with a 5,000-member training set of radial profiles, bn(r)b_{n}(r), computed with Eq. (4) using the Lorenz-Mie theory. Parameters for these computed profiles were evenly distributed over a volume in the three-dimensional space spanned by 13.5μmzp75μm13.5~{}\mu\text{m}\leq z_{p}\leq 75~{}\mu\text{m}, 0.4μmap1.75μm0.4~{}\mu\text{m}\leq a_{p}\leq 1.75\mu\text{m}, and 1.4np1.81.4\leq n_{p}\leq 1.8 at a resolution of 1.35μm1.35~{}\mu\text{m} in zpz_{p}, 0.1μm0.1~{}\mu\text{m} in apa_{p} and 0.1 in npn_{p}. Training time increases dramatically with the number of training sets, and also with larger values of CC and γ\gamma. Once trained, however, an SVM translates input data vectors into output parameter estimates extremely rapidly.

The quality of a trained SVM can be assessed by presenting it with independently computed cross-validation data. Optimal values for CC and γ\gamma minimize differences between estimated parameters and the inputs. Using a 500-member cross-validation set, we obtained best performance for estimating zpz_{p} with C=100C=100 and γ=1\gamma=1, best performance for npn_{p} with C=10C=10 and γ=0.5\gamma=0.5, and best performance for apa_{p} with C=10C=10 and γ=0.6\gamma=0.6.

Sampling the entire parameter space accessible to holographic characterization with resolution comparable to the precision realized with nonlinear fits ((4)) would require more than 101010^{10} training sets. If, however, the system of interest is characterized by a more modest range of parameters, then results from an initial SVM analysis can be used to generate a comparatively small set of training data spanning the relevant range. This specialized training proceeds rapidly and yields substantial improvements in precision.

III Characterization of Colloidal Mixtures

The data plotted in Fig. 1 are SVM estimates for the radii and refractive indexes of 2,500 colloidal spheres flowing down a 20μm20~{}\mu\text{m}-deep microfluidic channel formed by bonding the edges of a glass microscope cover slip to the surface of a glass microscope slide. The peak flow speed of 1 mm/s transports a sphere across the field of view in no fewer than two video frames, ensuring that every particle in the flow has a chance to be analyzed. Anisotropic blurring due to a sphere’s 100 nm motion during the camera’s 0.1 ms exposure time suppresses contrast along the direction of motion, but does not appreciably influence the azimuthal average, b(r)b(r) ((18)). Spurious results arising when multiple spheres’ interference patterns overlap contribute outliers to the observed distribution of particle sizes and refractive indexes. Such artifacts are minimized by diluting the sample to a volume fraction less than 10-310^{-3} so that no more than three particles are present in any frame.

The sample was prepared by dispersing roughly equal proportions of four types of colloidal spheres in water: 1μm1~{}\mu\text{m}-diameter and 2μm2~{}\mu\text{m}-diameter spheres made of polystyrene (Thermoscientific Catalog No. 5100A and Bangs Laboratories Catalog No. SS04N, respectively) and silica (Duke Standards Catalog Nos. 8100 and 4202A, respectively). Each of these four populations is monodisperse, with a standard deviation in the radius of less than 5%. This four-component mixture was flowed through the observation volume during a 12 min interval, and analyzed particle-by-particle. Each data point in Fig. 1 corresponds to an individual sphere, and is colored by the local density of measurements.

SVM-mediated holographic characterization clearly identifies the four populations of particles and provides estimates for their relative abundances. The mode values of the refractive indexes, np1.4n_{p}\approx 1.4 and 1.6 are consistent with values for silica and polystyrene, respectively. Values for the radii, ap0.5μma_{p}\approx 0.5~{}\mu\text{m} and 1μm1~{}\mu\text{m} are consistent with manufacturers’ specifications. Characterizing multicomponent dispersions is a unique capability of holographic particle analysis, and can be performed with SVMs as fast as particles’ holograms can be acquired. This performance is all the more noteworthy because the distribution of particle properties is built up one particle at a time and so does not rely on a priori assumptions.

Neither the instrument nor the analytical technique requires extensive calibration. The wavelength of the laser and the effective magnification can be calibrated once and used for all samples. The refractive index of the medium is the only free parameter, and often can be obtained separately. These parameters are used to train the SVMs in advance, after which they can be used to analyze arbitrary samples dispersed in the medium.

IV Tracking and Assessment of Precision

Figure 2: Tracking and characterizing a single colloidal sphere. (a) The estimated axial position zp(t)z_{p}(t) relative to the focal plane of the microscope of a single polystyrene sphere sedimenting through water as it diffuses. The line is a least-squares fit. Insets show the sphere’s hologram at the beginning and end of the trajectory. (b) The radius apa_{p} and refractive index npn_{p} estimated from each hologram in the same sequence, colored by time. Each dot corresponds to values obtained from a single hologram. (c) The mean-squared displacement, Δzp2(τ)\Delta z_{p}^{2}(\tau) as a function of lag time τ\tau computed from the data in (a), including statistical error bars. The superimposed line is a fit to Eq. (7).

Tracking a single colloidal sphere as it sediments and diffuses provides insights into the precision and accuracy of SVM-mediated holographic characterization. The data in Fig. 2 were obtained with a 1.59μm1.59~{}\mu\text{m}-diameter polystyrene sphere (Duke Scientific, catalog 4016A) diffusing as it sedimented through deionized water near the midplane of a 120μm120~{}\mu\text{m}-deep channel. Figure 2(a) shows the time-resolved trajectory, zp(t)z_{p}(t), obtained from a sequence of 4,500 video frames recorded at 29.97 frames/s using iterative SVM training.

Because polystyrene is roughly 5 percent more dense than water, the sphere sediments more that 10μm10~{}\mu\text{m} over the course of the experiment. The insets to Fig. 2(a) show how markedly the hologram’s appearance changes from the beginning of the trajectory to the end. Despite these changes, the SVMs’ estimates for the radius and refractive index plotted in Fig. 2(b) remain clustered around the mean values ap=0.79±0.02μma_{p}=0.79\pm 0.02~{}\mu\text{m} and np=1.600±0.006n_{p}=1.600\pm 0.006.

Uncertainties in estimated parameters are computed as standard deviations of the distribution of results plotted in Fig. 2(b). These should be interpreted with care because errors in SVM estimates need not be independent or normally distributed. Data points in Fig. 2(b) cluster around different values as the particle descends, which suggests that different support vectors dominate the estimates for apa_{p} and npn_{p} when the sphere is at different axial positions. Systematic errors in the individual parameters therefore may vary with changes in any of the parameters’ values. Even so, the averages of the SVM estimates are consistent with the manufacturer’s specifications, and differ only slightly from those obtained with a full Lorenz-Mie analysis of the same data set ((10)), which yields ap=0.805±0.001μma_{p}=0.805\pm 0.001~{}\mu\text{m} and np=1.5730±0.0006n_{p}=1.5730\pm 0.0006. Nonlinear fitting offers ten times better precision and accuracy ((10)). SVM analysis is a thousand times faster.

The mean sedimentation speed, vp=66±1nm/sv_{p}=66\pm 1~{}\text{nm/s}, estimated from the slope of zp(t)z_{p}(t) is somewhat smaller than the value measured with fits to the Lorenz-Mie theory ((10)) of 75±1nm/s75\pm 1~{}\text{nm/s}. This discrepancy further suggests that the SVM estimate for a parameter’s value may depend on the value itself. If we nevertheless assume that errors in zpz_{p} are normally distributed with a root-mean-square value ϵz\epsilon_{z}, then the diffusing particle’s mean-squared displacement should evolve over time interval τ\tau as

Δzp2(τ)[zp(t+τ)-zp(t)]2t=2Dτ+vp2τ2+2ϵz2,\Delta z_{p}^{2}(\tau)\equiv\left<\left[z_{p}(t+\tau)-z_{p}(t)\right]^{2}% \right>_{t}=2D\tau+v_{p}^{2}\tau^{2}+2\epsilon_{z}^{2}, (7)

where D=kBT/(6πηap)D=k_{B}T/(6\pi\eta a_{p}) is the Stokes-Einstein value for the particle’s diffusion coefficient. The data in Fig. 2(c) yield D=0.319±0.004μm2/sD=0.319\pm 0.004~{}\mu\text{m}^{2}/\text{s}, which is slightly larger than the value of 0.292±0.004μm2/s0.292\pm 0.004~{}\mu\text{m}^{2}/\text{s} obtained with the full Lorenz-Mie analysis ((10)). The best-fit tracking error, ϵz=107±2nm\epsilon_{z}=107\pm 2~{}\text{nm}, exceeds the Lorenz-Mie bound by an order of magnitude ((10)).

V Conclusions

The results presented here are typical of the performance of SVMs for characterizing and tracking colloidal spheres of a wide variety of compositions and sizes over a large axial range. The speed and precision of SVM characterization is ideal for monitoring, feedback control and quality assurance in any industrial process involving colloidal spheres. In applications where the greatest precision is required, parameters estimated with SVMs can be used to initialize nonlinear least-squares fits. Starting from such reasonable estimates speeds convergence and reduces the likelihood of failed fits. Being able to resolve multimodal distributions by quickly amassing single-particle measurements avoids ambiguities inherent in population-averaging methods such as dynamic light scattering. Extracting the refractive index as well as the size offers insights into sample composition that otherwise would not be available. SVM-accelerated tracking can be used for real-time three-dimensional particle-tracking velocimetry ((8)). For applications such as microrefractometry ((9)), the medium’s refractive index, nmn_{m}, can be estimated instead of the particle’s.

This combination of capabilities enables new applications. For example, the distribution of properties in colloidal mixtures could serve as fingerprints for complex fluid systems, with the sizes, refractive indexes and relative abundances encoding information that can be accessed with SVM-mediated holographic characterization.

Such applications can be realized with comparatively simple instruments ((10)) conveying image data to low-power computers. Although training SVMs can be computationally intensive, the data comprising a set of trained SVMs occupies less than 100 Mb. Pre-computed SVMs therefore can be archived and rapidly retrieved when needed. This approach lends itself to implementation on embedded computers for integration into low-cost analytical instruments.

Other machine-learning techniques also might be effective for analyzing holograms of colloidal particles. Artificial neural networks, for instance, can be trained in the same manner as the present SVM implementation to interpret radial profiles of experimental holograms. SVMs have the advantage that their training process proceeds deterministically, and therefore tends to be faster. Once successfully trained, however, artificial neural networks are generally more computationally efficient. Regardless of implementation, the present results demonstrate that machine-learning methods facilitate fast and precise measurements of colloidal properties.

Acknowledgments

This work was supported by a research grant from Procter & Gamble.

References

  • (1) J. Sheng, E. Malkiel and J. Katz. “Digital holographic microscope for measuring three-dimensional particle distributions and motions.” Appl. Opt.45, 3893–3901 (2006).
  • (2) S.-H. Lee and D. G. Grier. “Holographic microscopy of holographically trapped three-dimensional structures.” Opt. Express15, 1505–1512 (2007).
  • (3) C. F. Bohren and D. R. Huffman. Absorption and Scattering of Light by Small Particles (Wiley Interscience, New York, 1983).
  • (4) S.-H. Lee, Y. Roichman, G.-R. Yi, S.-H. Kim, S.-M. Yang, A. van Blaaderen, P. van Oostrum and D. G. Grier. “Characterizing and tracking single colloidal particles with video holographic microscopy.” Opt. Express15, 18275–18282 (2007).
  • (5) A. Bourquard, N. Pavillon, E. Bostan, C. Depeursinge and M. Unser. “A practical inverse-problem approach to digital holographic reconstruction.” Opt. Express21, 3417–3433 (2013).
  • (6) M. Seifi, L. Denis and C. Fournier. “Fast and accurate 3D object recognition directly from digital holograms.” J. Opt. Soc. Am. A30, 2216–2224 (2013).
  • (7) J. Fung and V. N. Manoharan. “Holographic measurements of anisotropic three-dimensional diffusion of colloidal clusters.” Phys. Rev. E88, 020302 (2013).
  • (8) F. C. Cheong, B. Sun, R. Dreyfus, J. Amato-Grill, K. Xiao, L. Dixon and D. G. Grier. “Flow visualization and flow cytometry with holographic video microscopy.” Opt. Express17, 13071–13079 (2009).
  • (9) H. Shpaisman, B. J. Krishnatreya and D. G. Grier. “Holographic microrefractometer.” Appl. Phys. Lett.101, 091102 (2012).
  • (10) B. J. Krishnatreya, A. Colen-Landy, P. Hasebe, B. A. Bell, J. R. Jones, A. Sunda-Meya and D. G. Grier. “Measuring Boltzmann’s constant through holographic video microscopy of a single sphere.” Am. J. Phys.82, 23–31 (2014).
  • (11) F. C. Cheong, B. J. Krishnatreya and D. G. Grier. “Strategies for three-dimensional particle tracking with holographic video microscopy.” Opt. Express18, 13563–13573 (2010).
  • (12) A. J. Smola and B. Schölkopf. “A tutorial on support vector regression.” Statistics and Computing14, 199–222 (2004).
  • (13) B. J. Krishnatreya and D. G. Grier. “Fast feature identification for holographic tracking: The orientation alignment transform.” Opt. Express22, 12773–12778 (2014).
  • (14) See, for example, http://physics.nyu.edu/grierlab/software.html for holographic microscopy software written in the IDL programming language that was used in the present study. A comparable implementation in the python programming language is available at http://manoharan.seas.harvard.edu/holopy/.
  • (15) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot and E. Duchesnay. “Scikit-learn: Machine Learning in Python.” Journal of Machine Learning Research12, 2825–2830 (2011).
  • (16) C.-C. Chang and C.-J. Lin. “LIBSVM: A library for support vector machines.” ACM Transactions on Intelligent Systems and Technology2, 27:1–27:27 (2011).
  • (17) C.-C. Chang and C.-J. Lin. “Training ν\nu-Support Vector Regression: Theory and Algorithms.” Neural Computation14, 1959–1977 (2002).
  • (18) L. Dixon, F. C. Cheong and D. G. Grier. “Holographic particle-streak velocimetry.” Opt. Express19, 4393–4398 (2011).