Holograms of colloidal dispersions encode comprehensive information about individual particles’ three-dimensional positions, sizes and optical properties. Extracting that information typically is computationally intensive, and thus slow. Here, we demonstrate that machine-learning techniques based on support vector machines (SVMs) can analyze holographic video microscopy data in real time on low-power computers. The resulting stream of precise particle-resolved tracking and characterization data provides unparalleled insights into the composition and dynamics of colloidal dispersions and enables applications ranging from basic research to process control and quality assurance.
Holograms of colloidal spheres obtained with holographic video microscopy ((1); (2)) can be interpreted with predictions of the Lorenz-Mie theory of light scattering ((3)) to track each particle in three dimensions, and to measure its size and refractive index ((4)). State-of-the-art implementations ((4); (5); (6); (7)) can locate a sphere and resolve its radius both to within a few nanometers, and can determine its refractive index to within a part per thousand ((8); (9); (10)). The cost of this powerful technique is the computational burden of fitting each hologram pixel-by-pixel to theoretical predictions ((4); (11)). Here, we demonstrate that techniques of machine learning can reduce the processing time by a factor of a thousand, yielding real-time performance.
Our approach to fast holographic characterization, depicted schematically in Fig. 1, employs the support vector machine (SVM) algorithm ((12)) to compare experimental measurements with pre-computed predictions of the Lorenz-Mie theory ((3); (4); (13)). Whereas nonlinear fitting typically requires more than a second on a 1 Gflop computer, a trained SVM can estimate the size, refractive index or axial position of a micrometer-scale sphere in under a millisecond on the same hardware.
The in-line holographic microscope used for these studies ((2); (4); (10)) illuminates the sample with a linearly polarized collimated laser beam (Coherent Cube, 20 mW) at a vacuum wavelength of $\lambda=447~{}\text{nm}$. The fluence of the 3 mm-diameter beam is comparable to that of a conventional microscope illuminator. Optical forces and light-induced heating therefore are negligible. Light scattered by a sphere propagates to the focal plane of a custom-built video microscope ((10)) where it interferes with the undiffracted portion of the original beam. The microscope magnifies this interference pattern onto the detector of a greyscale video camera (NEC TI 324AII), which records its intensity with a system magnification of 135 nm/pixel. Each snapshot in the video stream constitutes a hologram of the particles in the channel.
The electric field of the incident beam at position $\mathbf{r}$ in the focal plane may be modeled as a plane wave with spatial dependence
$\mathbf{E}_{0}(\mathbf{r})=u_{0}(\mathbf{r})\,e^{i\varphi_{0}(\mathbf{r})}\,e^% {ikz}\,\hat{x},$ | (1) |
where $k=2\pi n_{m}/\lambda$ is the wavenumber in a medium of refractive index $n_{m}$, and where $u_{0}(\mathbf{r})$ and $\varphi_{0}(\mathbf{r})$ account for small variations in the beam’s amplitude and phase profiles, respectively.
A particle located at $\mathbf{r}_{p}$ relative to the center of the focal plane scatters the incident illumination, $\mathbf{E}_{0}(\mathbf{r}_{p})$, to the focal plane as
$\mathbf{E}_{s}(\mathbf{r})=E_{0}(\mathbf{r}_{p})\,\mathbf{f}_{s}\!\left(k(% \mathbf{r}-\mathbf{r}_{p})|a_{p},n_{p}\right),$ | (2) |
where $\mathbf{f}_{s}(k\mathbf{r}|a_{p},n_{p})$ is the Lorenz-Mie scattering function ((3)) that describes how a sphere of radius $a_{p}$ and refractive index $n_{p}$ scatters an $\hat{x}$-polarized plane wave. The measured intensity then may be modeled as
$I(\mathbf{r})=\left|\mathbf{E}_{0}(\mathbf{r})+\mathbf{E}_{s}(\mathbf{r})% \right|^{2}.$ | (3) |
Normalizing the recorded hologram by $I_{0}(\mathbf{r})=\left|\mathbf{E}_{0}(\mathbf{r})\right|^{2}=u_{0}^{2}(% \mathbf{r})$ suppresses spurious structure in the illumination and yields a functional form for the normalized hologram ((4); (10)),
$b(\mathbf{r})=\frac{I(\mathbf{r})}{I_{0}(\mathbf{r})}\approx\left|\hat{x}+e^{-% ikz_{p}}\mathbf{f}_{s}(k(\mathbf{r}-\mathbf{r}_{p})|a_{p},n_{p})\right|^{2},$ | (4) |
that can be calculated with standard software packages ((14)).
Previous implementations of Lorenz-Mie microscopy ((4)) fit Eq. (4) to measured holograms using $a_{p}$, $n_{p}$ and $\mathbf{r}_{p}$ as adjustable parameters. These fits are exquisitely sensitive to errors in the particle’s in-plane position, and so must be performed over the entire two-dimensional intensity distribution ((4)). Here, we instead use Eq. (4) to train support vector machines, which then are able to estimate $a_{p}$, $n_{p}$ and $z_{p}$ from a hologram’s one-dimensional radial profile. We obtain these profiles from measured holograms by averaging around centers of rotational symmetry ((13)) with single-pixel resolution, yielding 100-point data vectors. The microscope’s focal plane is adjusted so that the interference fringes in a typical sphere’s hologram extend to roughly this scale. Averaging over angles to obtain a radial profile reduces the dimensionality of the analysis problem and accounts in part for our method’s computational efficiency.
Our SVMs are implemented with scikit-learn, an open-source machine learning software package ((15)) that builds upon the libsvm library of Support Vector Machine algorithms ((16); (17)). Each SVM computes one output parameter from an input vector consisting of a radial profile, $b(r)$, that has been digitized into 100 single-pixel bins. Characterizing and tracking a colloidal particle therefore requires three SVMs, one for each of $a_{p}$, $n_{p}$ and $z_{p}$. Figure 1 schematically represents this process for estimating $a_{p}$ and $n_{p}$.
An SVM computes its output by comparing $b(r)$ with sets of training data, $b_{n}(r)$, that are obtained from Eq. (4) over a range of values of $a_{p}$, $n_{p}$ and $z_{p}$. Each training set constitutes one support vector in the space spanned by these parameters. To facilitate these comparisons, we construct SVMs with radial basis functions ((12)),
$k_{n}(b)=\exp\left(-\gamma\int\left|b_{n}(r)-b(r)\right|^{2}\,dr\right),$ | (5) |
that quantify the similarity of the experimental input with the $n$-th support vector. The sensitivity of this comparison is set by $\gamma$, with larger values favoring more precise results at the cost of requiring more support vectors to span the parameter space. Given a value of $\gamma$, the training process determines a set of weights $\omega_{n}$ and an offset $s_{0}$ such that the weighted sum,
$\tilde{s}(b)=\sum_{n}\omega_{n}k_{n}(b)+s_{0},$ | (6) |
constitutes an estimate for the parameter, $s(b)$, that characterizes the input data vector, $b(r)$. In general, errors in $\tilde{s}(b)$ depend smoothly on $\gamma$ ((12)).
To prevent overfitting, the weights $\omega_{n}$ are constrained to have magnitudes less than a maximum value, which conventionally is denoted by $C$ ((12)). Optimizing an SVM with larger values of $C$ improves its ability to recognize its training data, but renders it less able to interpolate smoothly between its support vectors when presented with novel or noisy inputs. Some candidate support vectors may be assigned small weighting factors in optimizing $\tilde{s}(b)$ over a corpus of training data; these are automatically eliminated from the SVM ((12)). The choice of $\gamma$ and $C$ thus determines which support vectors are included in the SVM, and their relative importance for computing the output. Because this process is nonlinear, optimal values are obtained by exhaustive search over the range $0.1\leq\gamma\leq 10$ and $0.1\leq C\leq 1,000$. Statistically indistinguishable results are obtained in the present application for values of $\gamma$ and $C$ that vary from their optimal values by up to fifty percent.
We trained SVMs with a 5,000-member training set of radial profiles, $b_{n}(r)$, computed with Eq. (4) using the Lorenz-Mie theory. Parameters for these computed profiles were evenly distributed over a volume in the three-dimensional space spanned by $13.5~{}\mu\text{m}\leq z_{p}\leq 75~{}\mu\text{m}$, $0.4~{}\mu\text{m}\leq a_{p}\leq 1.75\mu\text{m}$, and $1.4\leq n_{p}\leq 1.8$ at a resolution of $1.35~{}\mu\text{m}$ in $z_{p}$, $0.1~{}\mu\text{m}$ in $a_{p}$ and 0.1 in $n_{p}$. Training time increases dramatically with the number of training sets, and also with larger values of $C$ and $\gamma$. Once trained, however, an SVM translates input data vectors into output parameter estimates extremely rapidly.
The quality of a trained SVM can be assessed by presenting it with independently computed cross-validation data. Optimal values for $C$ and $\gamma$ minimize differences between estimated parameters and the inputs. Using a 500-member cross-validation set, we obtained best performance for estimating $z_{p}$ with $C=100$ and $\gamma=1$, best performance for $n_{p}$ with $C=10$ and $\gamma=0.5$, and best performance for $a_{p}$ with $C=10$ and $\gamma=0.6$.
Sampling the entire parameter space accessible to holographic characterization with resolution comparable to the precision realized with nonlinear fits ((4)) would require more than $10^{10}$ training sets. If, however, the system of interest is characterized by a more modest range of parameters, then results from an initial SVM analysis can be used to generate a comparatively small set of training data spanning the relevant range. This specialized training proceeds rapidly and yields substantial improvements in precision.
The data plotted in Fig. 1 are SVM estimates for the radii and refractive indexes of 2,500 colloidal spheres flowing down a $20~{}\mu\text{m}$-deep microfluidic channel formed by bonding the edges of a glass microscope cover slip to the surface of a glass microscope slide. The peak flow speed of 1 mm/s transports a sphere across the field of view in no fewer than two video frames, ensuring that every particle in the flow has a chance to be analyzed. Anisotropic blurring due to a sphere’s 100 nm motion during the camera’s 0.1 ms exposure time suppresses contrast along the direction of motion, but does not appreciably influence the azimuthal average, $b(r)$ ((18)). Spurious results arising when multiple spheres’ interference patterns overlap contribute outliers to the observed distribution of particle sizes and refractive indexes. Such artifacts are minimized by diluting the sample to a volume fraction less than $10^{-3}$ so that no more than three particles are present in any frame.
The sample was prepared by dispersing roughly equal proportions of four types of colloidal spheres in water: $1~{}\mu\text{m}$-diameter and $2~{}\mu\text{m}$-diameter spheres made of polystyrene (Thermoscientific Catalog No. 5100A and Bangs Laboratories Catalog No. SS04N, respectively) and silica (Duke Standards Catalog Nos. 8100 and 4202A, respectively). Each of these four populations is monodisperse, with a standard deviation in the radius of less than 5%. This four-component mixture was flowed through the observation volume during a 12 min interval, and analyzed particle-by-particle. Each data point in Fig. 1 corresponds to an individual sphere, and is colored by the local density of measurements.
SVM-mediated holographic characterization clearly identifies the four populations of particles and provides estimates for their relative abundances. The mode values of the refractive indexes, $n_{p}\approx 1.4$ and 1.6 are consistent with values for silica and polystyrene, respectively. Values for the radii, $a_{p}\approx 0.5~{}\mu\text{m}$ and $1~{}\mu\text{m}$ are consistent with manufacturers’ specifications. Characterizing multicomponent dispersions is a unique capability of holographic particle analysis, and can be performed with SVMs as fast as particles’ holograms can be acquired. This performance is all the more noteworthy because the distribution of particle properties is built up one particle at a time and so does not rely on a priori assumptions.
Neither the instrument nor the analytical technique requires extensive calibration. The wavelength of the laser and the effective magnification can be calibrated once and used for all samples. The refractive index of the medium is the only free parameter, and often can be obtained separately. These parameters are used to train the SVMs in advance, after which they can be used to analyze arbitrary samples dispersed in the medium.
Tracking a single colloidal sphere as it sediments and diffuses provides insights into the precision and accuracy of SVM-mediated holographic characterization. The data in Fig. 2 were obtained with a $1.59~{}\mu\text{m}$-diameter polystyrene sphere (Duke Scientific, catalog 4016A) diffusing as it sedimented through deionized water near the midplane of a $120~{}\mu\text{m}$-deep channel. Figure 2(a) shows the time-resolved trajectory, $z_{p}(t)$, obtained from a sequence of 4,500 video frames recorded at 29.97 frames/s using iterative SVM training.
Because polystyrene is roughly 5 percent more dense than water, the sphere sediments more that $10~{}\mu\text{m}$ over the course of the experiment. The insets to Fig. 2(a) show how markedly the hologram’s appearance changes from the beginning of the trajectory to the end. Despite these changes, the SVMs’ estimates for the radius and refractive index plotted in Fig. 2(b) remain clustered around the mean values $a_{p}=0.79\pm 0.02~{}\mu\text{m}$ and $n_{p}=1.600\pm 0.006$.
Uncertainties in estimated parameters are computed as standard deviations of the distribution of results plotted in Fig. 2(b). These should be interpreted with care because errors in SVM estimates need not be independent or normally distributed. Data points in Fig. 2(b) cluster around different values as the particle descends, which suggests that different support vectors dominate the estimates for $a_{p}$ and $n_{p}$ when the sphere is at different axial positions. Systematic errors in the individual parameters therefore may vary with changes in any of the parameters’ values. Even so, the averages of the SVM estimates are consistent with the manufacturer’s specifications, and differ only slightly from those obtained with a full Lorenz-Mie analysis of the same data set ((10)), which yields $a_{p}=0.805\pm 0.001~{}\mu\text{m}$ and $n_{p}=1.5730\pm 0.0006$. Nonlinear fitting offers ten times better precision and accuracy ((10)). SVM analysis is a thousand times faster.
The mean sedimentation speed, $v_{p}=66\pm 1~{}\text{nm/s}$, estimated from the slope of $z_{p}(t)$ is somewhat smaller than the value measured with fits to the Lorenz-Mie theory ((10)) of $75\pm 1~{}\text{nm/s}$. This discrepancy further suggests that the SVM estimate for a parameter’s value may depend on the value itself. If we nevertheless assume that errors in $z_{p}$ are normally distributed with a root-mean-square value $\epsilon_{z}$, then the diffusing particle’s mean-squared displacement should evolve over time interval $\tau$ as
$\Delta z_{p}^{2}(\tau)\equiv\left<\left[z_{p}(t+\tau)-z_{p}(t)\right]^{2}% \right>_{t}=2D\tau+v_{p}^{2}\tau^{2}+2\epsilon_{z}^{2},$ | (7) |
where $D=k_{B}T/(6\pi\eta a_{p})$ is the Stokes-Einstein value for the particle’s diffusion coefficient. The data in Fig. 2(c) yield $D=0.319\pm 0.004~{}\mu\text{m}^{2}/\text{s}$, which is slightly larger than the value of $0.292\pm 0.004~{}\mu\text{m}^{2}/\text{s}$ obtained with the full Lorenz-Mie analysis ((10)). The best-fit tracking error, $\epsilon_{z}=107\pm 2~{}\text{nm}$, exceeds the Lorenz-Mie bound by an order of magnitude ((10)).
The results presented here are typical of the performance of SVMs for characterizing and tracking colloidal spheres of a wide variety of compositions and sizes over a large axial range. The speed and precision of SVM characterization is ideal for monitoring, feedback control and quality assurance in any industrial process involving colloidal spheres. In applications where the greatest precision is required, parameters estimated with SVMs can be used to initialize nonlinear least-squares fits. Starting from such reasonable estimates speeds convergence and reduces the likelihood of failed fits. Being able to resolve multimodal distributions by quickly amassing single-particle measurements avoids ambiguities inherent in population-averaging methods such as dynamic light scattering. Extracting the refractive index as well as the size offers insights into sample composition that otherwise would not be available. SVM-accelerated tracking can be used for real-time three-dimensional particle-tracking velocimetry ((8)). For applications such as microrefractometry ((9)), the medium’s refractive index, $n_{m}$, can be estimated instead of the particle’s.
This combination of capabilities enables new applications. For example, the distribution of properties in colloidal mixtures could serve as fingerprints for complex fluid systems, with the sizes, refractive indexes and relative abundances encoding information that can be accessed with SVM-mediated holographic characterization.
Such applications can be realized with comparatively simple instruments ((10)) conveying image data to low-power computers. Although training SVMs can be computationally intensive, the data comprising a set of trained SVMs occupies less than 100 Mb. Pre-computed SVMs therefore can be archived and rapidly retrieved when needed. This approach lends itself to implementation on embedded computers for integration into low-cost analytical instruments.
Other machine-learning techniques also might be effective for analyzing holograms of colloidal particles. Artificial neural networks, for instance, can be trained in the same manner as the present SVM implementation to interpret radial profiles of experimental holograms. SVMs have the advantage that their training process proceeds deterministically, and therefore tends to be faster. Once successfully trained, however, artificial neural networks are generally more computationally efficient. Regardless of implementation, the present results demonstrate that machine-learning methods facilitate fast and precise measurements of colloidal properties.
This work was supported by a research grant from Procter & Gamble.