top of page
Compressed Ultrafast Photography

Video recording of ultrafast phenomena such as dynamic events in molecular biology would transform our understanding of a range of phenomena. However, using a detector array based on CCD or CMOS technologies is fundamentally limited by the sensor's on-chip storage and data transfer speed. To get around this problem, the most practical approach is to use a streak camera. In this ultrafast imaging device, the incident light first passes through a narrow entrance slit (usually 50μm wide) and is imaged onto the photocathode of a streak tube. Here, the incident light is converted to photoelectrons, which are accelerated by an accelerating mesh. A pair of electrodes then applies a sweeping (i.e., time-varying) voltage along the axis perpendicular to the device's entrance slit. Because of this sweeping voltage, electrons arriving at different times are deflected to different spatial positions, and these electrons are then multiplied by a microchannel plate. They subsequently bombard a phosphor screen and are converted back into light. The phosphor screen is imaged to a CCD, which records the image. However, the resultant image is normally 1D: only a single line of the scene can be seen at a time. Acquiring a 2D image requires mechanical scanning across the entire field of view, which poses severe restrictions on the recordable scenes because the event itself must be repetitive.

Previous approaches to enable 2D ultrafast imaging of nonrepetitive events include sequentially timed all-optical mapping photography and parallel streak imaging using a tilted lenslet array. However, most of them either rely on active illumination or suffer from significant throughput loss. To overcome these limitations, we have developed a new computational ultrafast imaging method, referred to as compressed ultrafast photography (CUP), which can capture 2D dynamic scenes at up to 100 billion frames per second. Akin to a conventional photographic camera, CUP is receive-only, thereby allowing high-speed video recording of a variety of luminescent—such as fluorescent or bioluminescent—objects.

Based on the concept of compressed sensing, CUP works by encoding the input scene with a random binary pattern in the spatial domain, followed by shearing the resultant image in a streak camera with a fully opened entrance slit. The random binary pattern, which encodes the scene as is standard in compressed sensing, also serves as the key to unlock and retrieve time information from the input scene in the subsequent image reconstruction. CUP forms images by successively applying three operators—a spatial encoding operator, temporal shearing operator, and spatiotemporal integration operator—on the input time-lapse event I(x, y, t), where x and y are spatial coordinates, and t is time. The image is reconstructed by solving the inverse problem of these three processes. A typical input scene is regarded to be sparse in the spatiotemporal domain, meaning that it can be represented as a matrix in which a large number of elements are zeros. Under this condition, the original event datacube can be reasonably estimated by employing a two-step iterative shrinkage/thresholding algorithm.

Figure 1 shows CUP's system schematic. The scene is first imaged onto an intermediate plane by a camera lens. Then this image is relayed to a spatial encoding unit, a digital micromirror device (DMD), which consists of tens of thousands of small MEMS (microelectromechanical systems) mirrors. Each micromirror can be individually turned ‘on’ and ‘off.’ The light reflected from the ‘on’ micromirrors is collected by a microscope objective and passed to the streak camera, while the light reflected from the ‘off’ micromirrors is out of the objective's collecting angle. In the streak camera, the entrance slit is opened to its maximum width, allowing the resultant 2D image to be temporally sheared along the vertical axis. The amount of shearing is dependent on the arrival time of incident photons. The final spatiotemporally multiplexed image is captured by a CCD within a single exposure. Using CUP, we can reconstruct an event datacube with 150×150×350 (x, y, t) voxels (volume elements).

005816_10_fig1.jpg

Figure 1. Optical setup of compressed ultrafast photography (CUP). DMD: Digital micromirror device.

To demonstrate CUP, we imaged light reflection, refraction, and racing in two different media (air and resin) (see Figure 2). Our technique, for the first time, enables video recording of photon propagation at a temporal resolution down to tens of picoseconds. Additionally, we imaged an apparent faster-than-light (FTL) phenomenon. By obliquely shining a laser pulse toward a stripe pattern and monitoring the movement of the intersected wavefront, we observed movement at twice light speed. However, this FTL propagation does not violate Einstein's relativity because no actual information transmission is associated with the motion. Moreover, to further expand CUP's functionality, we added a color separation unit to the system, allowing simultaneous acquisition of a 4D datacube (x, y, t, λ), where λ is wavelength, within a single camera snapshot.

005816_10_fig2.jpg

Figure 2. CUP of light propagation. (a) Laser pulse reflected from a mirror. (b) Laser pulse refracted from an air-resin interface. (c) Laser pulses racing in air and resin. Scale bar (yellow, top right image), 10mm.

CUP is a universal imaging platform that can be coupled to a variety of optical instruments, such as microscopes and telescopes, adding an unprecedented imaging speed to these modalities and facilitating new scientific discoveries at scales from cellular organelles to distant galaxies.

 

Recently, we have shown that CUP can be coupled to a high-resolution microscope and provide ultrafast dynamic movies inside a living cell (Fig. 3). The resultant method, compressed-sensing fluorescence lifetime imaging microscopy (compressed FLIM), can produce high-resolution two-dimensional (2D) lifetime images at an unprecedented frame rate. Compared to other state-of-the-art FLIM imagers, compressed FLIM has a striking advantage in acquiring a widefield lifetime image within a single camera snapshot, thereby eliminating motion artifacts and enabling fast recording of biological events.

CUP_edited.jpg
Fluorescence_lifetime_image.png

Figure 3.  Compressed-sensing FLIM

bottom of page