Figure 1
AFA takes a set of images with faces (a) and produces aligned versions of these images so that facial features are overlapping (b). AFA can also place a window over the aligned face images so that only inner facial features are shown (c). All photos licensed under CCL. Specific license info for each face can be found in the demo 1 folder of https://github.com/SourCherries/auto-face-align.
Figure 2
Morphs images produced by AFA that gradually transform Choi Min-sik (on the far left) into Sarah Silverman (on the far right) in 9 equal steps. Image of Choi Min-sik is by Petr Novák [Licensed under CC BY-SA 3.0] and image of Sarah Silverman is by Joan Garvin [Licensed under CC BY 2.0].
Figure 3
Comparison of the average of Angela Merkel faces (b) and the enhanced average of Angela Merkel faces (d). The normal average (b) is simply the mean of all the AFA-aligned faces in (a). The enhanced average is the mean of all the shape-corrected faces in (c). Landmarks in the shape-corrected faces share the exact same positions because each has been warped to the average of landmarks across faces in (a). Facial features are more well-defined in the enhanced average, with especially high local contrast around the lips and nostrils. All photos of Angela Merkel are CC license; license info for each photo in LICENSE file of demo_3_averaging folder of repo.
Figure 4
Overview of the alignment procedure. (a) Using GPA (described in Figure 5), a suitable reference shape (green dots) is derived from a set of many input shapes. An example input shape (red dot for each landmark) is shown in the next panel (b in red dots), overlaid on its corresponding face. In panel (c), we use Classical Procrustes Analysis (PA) to obtain the optimal rotation and scaling to align the input landmarks (red dots) with the reference (green dots). In panel (d) the image has been scaled and rotated appropriately (by 1.2 and –12 degrees). An additional rotation of +5 degrees is done because of the tilt in the reference shown in panel (a). Additional translation as well as cropping is performed such that either eye distance, image width or image height is fixed across all images. This image of Sarah Silverman is by Joan Garvin [Licensed under CC BY 2.0].
Figure 5
Flow chart of General Procrustes Analysis (GPA). The goal is to obtain a suitable reference shape that minimizes the least-squares distances between it and a set of input shapes, when each is superimposed onto the reference. By superimpose, we mean Classical Procrustes Analysis (PA) between pairs of shapes.
Table 1
Previous alignment types.
| AUTHORS | SCALING | ROTATION | TRANSLATION |
|---|---|---|---|
| [28] | Normalize distance between center of mouth and midpoint between eyes. | Vertical line connecting center of mouth and midpoint between eyes. | Center of image is midpoint between eyes. |
| [29] | Average semi-major and -minor axes of fitted ellipse. | Not specified. | Ellipses position on each face determined by eye centers, with common aspect ratio. Ellipses translated to overlap after scaling. |
| [30] | Distance between pupils. | Pupils aligned. | Pupils same position. |
| [31] | Minimize mean-square difference between 20 landmarks of each face with average of those landmarks across all faces. | Same. | Same. |
| [32] | Distance between eyes. | Not specified. | Nose tip. |
Figure 6
Construction of a landmark-based facial aperture shaped after a Moss’s egg. The Moss’s egg is a combination of 4 circles whose centers are colored discs with corresponding colors in this image. This image of Sarah Silverman is by Joan Garvin [Licensed under CC BY 2.0].
Figure 7
Schematic of the architecture for AFA, also displaying module dependencies for each function.
Figure 8
Distributions of landmarks (red ellipses) overlaid on mean of GPA-aligned images. Ellipse orientation based on angle between first and second eigenvectors of PCA applied to landmarks across all faces in a single category. Ellipse length and width are 2 SD of variability along those eigenvectors. Analysis is done separately for 4 categories of faces: Chinese females from the CAS database (a); Chinese males from the CAS database (b); predominantly Caucasian female faces from the GUFD database (c); predominantly Caucasian male faces from the GUFD database (d); in-lab photos of mostly Caucasian faces (mixed gender) from the KUFD database; and low-resolution photo-ID of mostly Caucasian faces (mixed gender and expression) from the KUFD database. Despite SD being a non-robust measure of variability that is strongly inflated by outliers, each ellipse is quite small relative to the size of facial features. Ellipse sizes are also roughly uniform across the different landmarks. The means of aligned faces have high contrast and normal-looking facial features. That provides additional, qualitative validation of the entire GPA-alignment process performed by AFA (including landmark detection).
Figure 9
Distributions of landmarks (red ellipses) overlaid on mean of GPA-aligned images of emotionally expressive faces. Unlike in the previous figure, alignments for these faces were based only on eye landmarks. [a to h] corresponds to anger, calm, disgust, fear, happy, neutral, sad, and surprised for closed mouth expressions. There was only 1 face available for closed-mouth surprise so that category has been excluded. [i to p] corresponds to anger, calm, disgust, fear, happy, neutral, sad, and surprised for open mouth expressions. As can be seen by the size of red ellipses around the eyes and the strong local contrast of eyes in the mean faces, AFA does an excellent job of aligning by eye landmarks alone. As expected from common experience, landmark variability is largest around the mouth.