A Python Package for Automatic Face Alignment (AFA) in Experimental Psychology

Introduction

Faces are a popular stimulus in psychology and alignment across a set of face stimuli is an important step in many studies. Alignment means that the overall size and orientation of faces is normalized and that facial features (eyes, brows, nose and mouth) are in similar positions. Common alignment procedures are varied, laborious and often not well documented. To address these shortcomings, we introduce Automatic Face Alignment (AFA), an open-source Python package that streamlines the full alignment process along with related functionality (aperture generation, morphing).

The one other open-source solution is WebMorphR, written in R [1]. Unlike WebMorphR, the full (end-to-end) alignment process of AFA is validated on face databases typical of psychological study, and across psychologically meaningful variations in faces (see section Performance evaluation). And unlike WebMorphR, the critical algorithms underlying both alignment (GPA) and morphing are exhaustively unit-tested using novel tests (see section Performance evaluation).

Everyone working with face stimuli in experimental psychology must align them in some way. The simplest reason is that faces must be seen in the same place on a computer display. Spatial alignment of faces can reduce spatial uncertainty; minimize eye movements; simplify the interpretation of neuroimaging results; ensure that the same areas of the face are being viewed trial after trial; and many other specific reasons. Typically, the exact reasons for alignment are not provided but some combination of the above reasons applies to almost all studies involving human responses to face stimuli. Therefore, face alignment can be considered a requirement by convention.

The alignment of faces by orientation and size can be just as important as alignment by position, especially given how the human visual system is finely tuned to specific bands of both spatial frequency and orientation [2, 3, 4, 5]. Just as with position, face size and orientation are also standardized in almost all studies. And while the exact reasons for standardization of size and position are often not stated, the reasons for such practice can include many of the same reasons stated for position alignment, in addition to the visual system’s precise preference for specific spatial frequency and orientation. Therefore, the standardization of face size and orientation can also be considered requirements by convention.

Face alignment can be critical in a wide range of studies that use face stimuli. Face alignment is important in fMRI studies with a focus on retinotopy [6], brain areas that are retinotopic [7], face orientation [8] and face size [9]. Face alignment is also important in behavioral studies and neuroimaging studies that require precise specification of image-information across different dimensions of the face. Examples include: measurement of an ideal observer [10, 11, 12]; measurement of observer spatial-frequency [2] or orientation channels [3]; reverse correlation [13, 14]; and models of task-specific face information [15, 16, 17].

Despite the importance of face alignment in many studies, face alignment procedures used by many psychology studies are often time-consuming and lack transparency. Typically, face alignment for a set of face photographs is primarily done using either one of 2 approaches: (1) manual landmark selection and affine manipulation using an image-editing application like Adobe Photoshop; or (2) manual landmark selection and affine manipulation using software like InterFace or custom software. The first approach is quite common, while the second approach is typically taken in studies that use morphed faces as morphing is one of the primary functions of InterFace. Regardless of approach, the manual selection of landmarks is required in both cases. That can be highly time consuming for large stimulus sets or it can present a disincentive from using larger sets that would allow for greater generalizability or novel types of hypotheses to be tested. Manual landmark estimation may also be subject to individual differences in landmark placement that simply would not exist with an algorithmic process.

Additionally, software used for alignment should be made available as open-source in a public repository with full documentation of its inner-workings. A fully open-source solution would provide greater transparency, reproducibility and extensibility. Ideally anyone in the scientific community should be able to fully understand and replicate the alignment process that is used for a given study. Finally, an open-source package would allow for the scientific community to make use of existing code to extend its functionality to test novel hypotheses.

There is only one open-source solution for face alignment and morphing that is used by psychologists – WebMorphR. However, the accuracy of the full alignment process in WebMorphR has not been empirically validated. In other words, it is not known to what degree landmarks agree with each other across faces after the alignment process, and whether that varies across landmark type. Here we introduce Automatic Face Alignment (AFA), a Python package that fully automates all steps in the face alignment process and is available as a public repository on GitHub with documentation, demos and unit tests (https://github.com/SourCherries/auto-face-align). And unlike WebMorphR, the full alignment process of AFA has been empirically validated on 3 popular face databases, across variations in emotional expressiveness, ethnicity and image quality. We document these tests here and demonstrate robust alignment even in cases of emotional expression.

One additional benefit of AFA over WebMorphR is its implementation in Python. Having a solution for face-alignment and stimulus preparation in Python facilitates coupling with the most popular open-source software tools for machine learning and computer vision (PyTorch, TensorFlow, scikit-learn, scikit-image, NumPy and SciPy). This is especially valuable given the increased interest in links between natural and artificial intelligence, at least in face perception.

Usage example: Alignment and windowing

Examples of all AFA use-cases (alignment/windowing, morphing, enhanced averages) are written as fully documented Python scripts available for anyone to use as templates for their own research code (https://github.com/SourCherries/auto-face-align/tree/main/demos and https://github.com/SourCherries/auto-face-align/tree/main/results). In addition, documentation is provided for interruption handling and verification.

The main purpose of this Python package is to align and window a set of face images. And for most users, the following Python script will generate the appropriate set of aligned and windowed face images:

import alignfaces as afa
faces_path = “/Users/Me/faces_for_my_study/”
pfx = “” # relevant input files may start with a string
sfx = “jpg” # input image format
afa.get_landmarks(
        faces_path,
        file_prefix=pfx,
        file_postfix=sfx,
)
aligned_path = afa.align_procrustes(
        faces_path,
        file_prefix=pfx,
        file_postfix=sfx,
)
afa.get_landmarks(
        aligned_path,
        file_prefix=pfx,
        file_postfix=sfx,
)
the_aperture, aperture_path = afa.place_aperture(
        aligned_path,
        file_prefix=pfx,
        file_postfix=sfx,
)

The variables file_prefix and file_postfix indicate which files correspond to the input faces. file_prefix can be an empty string as it is here, suggesting filenames can start with any string; whereas file_prefix gives us the file extension and must be for a valid image format such as png or jpeg. The variable faces_path specifies where the original face images reside on your computer. This folder can have a complicated structure with images residing in various subdirectories of this folder. As an example, the original face images might look like those in Figure 1a.

Figure 1

The call to afa.get_landmarks() measures landmarks for every face and writes those values to a JSON-formatted file called landmarks.txt, placed under faces_path. This is necessary before alignment can be done.

The call to afa.align_procrustes() uses GPA to align all of the face images and writes those new images to a new folder whose full path is in the variable aligned_path. These aligned face images will also be cropped to the same image dimensions so that, on average, the full head is just contained within the image. As an example, the aligned face images might look like those in Figure 1b.

If one then wants to place a window over each face to show only the inner facial features, then landmarks must be estimated again on the aligned faces. Then calling afa.place_aperture() will create a new set of face images that are both aligned and windowed appropriately. Those new images will be put into a new folder whose full path name is in the variable aperture_path. As an example, the windowed face images might look like those in Figure 1c.

Usage example: Morphing between identities

The Python script showing an example of how to do morphing is https://github.com/SourCherries/auto-face-align/tree/main/demos/demo_2_morphing. Example output is shown in Figure 2.

Figure 2

The easy and recommended way to make morphs is to create a new folder that contains only the 2 faces we wish to morph into each other. We specify the full pathname of that folder like this, for example:

source_dir = “users/bob/face_experiments/an-unusual-pair/”

Next we align the faces and then measure the landmarks for those aligned faces:

pfx = “” # relevant input files may start with a string
sfx = “jpg” # input image format
af.get_landmarks(source_dir, file_prefix=pfx, file_postfix=sfx)
aligned_path = af.align_procrustes(source_dir,
                        file_prefix = pfx,
                        file_postfix = sfx,
                        color_of_result=”rgb”)
af.get_landmarks(aligned_path,
                        file_prefix=pfx,
                        file_postfix=sfx)

Now we are ready to morph the 2 aligned faces within the folder aligned_path:

face_array, p, morph_path = afa.morph_between_two_faces(aligned_path,
                        do_these=[0, 1],
                        num_morphs=num_morphs,
                        file_prefix=pfx,
                        file_postfix=sfx,
                        new_dir = “morphed”,
                        weight_texture=True)

Whenever there are just 2 faces within aligned_path we always set do_these=[0, 1] . num_morphs is the number of morphing steps. new_dir is set to “morphed” so all the morphed images will be written to a new directory whose name is the concatenation of the aligned_path and “morphed”. When weight_texture is true then intermediate morphs are weighted combinations of the images, which is preferable.

Our call to morph_between_two_faces produces all of the morph images and writes them to the folder specified by morph_path. The filenames of the morphs range from N0.png for face A to N10.png for face B. If we wanted 20 morphs then the filenames would range from N0.png for face A to N20.png for face B. Finally, if we want to window our morph faces so that only inner facial features are shown, then we can do this:

the_aperture, aperture_path = af.place_aperture(morph_path,
                        file_prefix = “N”,
                        file_postfix = “png”,
                        aperture_type=”MossEgg”,
                        contrast_norm=”max”,
                        color_of_result=”rgb”)

You may not wish to create a new folder with only 2 faces to morph. Instead, you may prefer to specify a pair of faces from a folder containing many faces. In that case, you need to determine the indices of the 2 faces you would like to morph into each other. First, examine a list of all the images that you’ve aligned:

image_files = afa.get_source_files(aligned_path,
                        file_prefix = pfx,
                        file_postfix = sfx)

The variable image_files is a list of all the aligned images. Examine this variable and find the indices of the 2 images you would like to combine. For demo 2, Choi-min Sik and Sarah Silverman are images 9 and 29. These are the two faces we wish to combine so:

do_these = [9, 29]

Now we are ready to morph these 2 faces using a call to afa.morph_between_two_faces.

Usage example: Enhanced average

The Python script showing an example of how to make an enhanced average is https://github.com/SourCherries/auto-face-align/tree/main/demos/demo_3_averaging . Example output is shown in Figure 3.

Figure 3

Before an enhanced average can be made, the faces must be aligned and landmarks for those aligned faces should be calculated:

pfx = “merkel” # relevant input files may start with a string
sfx = “jpg” # input image format
 
# Estimate landmarks.
af.get_landmarks(my_faces_path, file_prefix=pfx, file_postfix=sfx,
start_fresh=True)
 
# Now we’re ready to align the faces - via generalized Procrustes analysis.
aligned_path = af.align_procrustes(my_faces_path,
 
                        file_prefix = pfx,
                        file_postfix = sfx)
 
# Estimate landmarks of aligned faces.
af.get_landmarks(aligned_path, file_prefix=pfx, file_postfix=sfx)

Once this is done, you can now warp each of the aligned faces to the mean of landmarks across faces:

# Warp each face to mean of landmarks
original_images, warped_to_mean =
af.warp_to_mean_landmarks(aligned_path,
                        file_prefix=pfx,
                        file_postfix=sfx)

This returns all of the warped faces in a single Numpy array warped_to_mean that can be averaged like this:

# Mean of warped faces
enhanced_average = warped_to_mean.mean(axis=0)

Remember to first align the faces and then measure the landmarks for those aligned faces.

Implementation and architecture

AFA is based on facial landmark detection that is powered by the reliable and open-source DLIB library [18]. DLIB’s landmark detection has been used in numerous machine-learning and deep neural-net applications, a small fraction of which we cite here [19, 20, 21, 22, 23, 24]. DLIB is used to preprocess training faces for an open-source deep neural network [20]. Empirical research demonstrates excellent agreement between DLIB-estimated landmarks and those given by human observers [25]. Finally, we evaluate DLIB on psychologically relevant databases and this is documented the open-source repository for AFA (https://github.com/SourCherries/auto-face-align/tree/main/results).

The alignment procedure. Using GPA (shown in Figure 5 and described in the next section), a suitable reference shape (green dots in Figure 4a) is derived from a set of many input shapes (the landmarks for all faces that need to be aligned together). In Figure 4a, the black cross is the average of landmarks (green dots) for x and y axes; which is centered at 0-origin. This reference shape is scaled so that distance between the eyes (solid black line) is specified by the user (94 pixels here). Notice the solid black line is tilted away from horizontal (dashed black line), by 5 degrees. This is dealt with afterward; between the steps described by Figure 4c and d. Figure 4 b shows an example input shape (red dot for each landmark) overlaid on its corresponding face. The black cross is the average of landmarks for x and y axes in all panels. Note that the facial image has been translated so that the average of landmarks is centered in the image (0, 0). In Figure 4c, we use Classical Procrustes Analysis (PA) to obtain the optimal rotation and scaling to align the input landmarks (red dots) with the reference (green dots). In this case, required scaling-factor and rotation is 1.2 and –12 degrees, respectively. In Figure 4 d the image has been scaled and rotated appropriately (by 1.2 and –12 degrees). An additional rotation of +5 degrees is done because of the tilt in the reference shown in Figure 4a. An additional translation shifts the image down by 5/12 times eye-distance, which ensures the entire head is within the image. Therefore, the mean of landmarks (black cross) is now shifted below image center (red cross). Finally, the image is cropped to the area shown in green. This is done by fixed proportions between eye distance, image width and image height. The user can decide which of these 3 dimensions to specify and the rest are calculated using the appropriate proportions (as shown by the area of Figure 4d shaded in green).

Figure 4

Figure 5

Generalized Procrustes Analysis (GPA). Due perhaps to the time-consuming nature of manual landmark estimation, most studies align faces using only 2 or 3 landmarks: the centers of the pupils and sometimes the center of the mouth. Alignment procedures are often not fully reported in publications; even when they are reported, they are quite variable. Examples of clearly detailed alignment procedures from existing studies are summarized in Table 1. Dupuis-Roy et al. (2014) is a notable exception to the norm, in that they devise a sophisticated method to ensure face alignment that does not bias toward perfect alignment of any single landmark, while allowing greater variability in other landmarks. This type of unbiased alignment can be quite advantageous for a few reasons. For example, any kind of pixel-based analysis of image information like an ideal observer model [10, 26] or Gabor-jet model [17] will naturally depend on how well different types of facial features overlap. Greater variance in the position of one facial feature relative to another will then lead to a higher concentration of information in that feature [26], which can be seen as an artifact if it wasn’t explicitly controlled for or modelled. And even in studies where this type of modelling is not an explicit part of the underlying hypothesis, biased alignment might still be cause for concern considering how rapidly and efficiently human participants can isolate such concentrations of image-based information [27]. between corresponding landmarks (described in Figure 5). Dupuis-Roy et al. (2014) use the average of landmarks across faces as the reference, which has the following problem: if there is great variability in the global position, orientation or size across faces, then the average of landmarks may be highly distorted. GPA’s iterative process for estimating the best reference shape for alignment does not have this problem. As described in the Performance Evaluation section of this paper, AFA does quite well. Please see Figure 8 which shows distributions of landmarks across various databases of faces that we already aligned using AFA. As one can see, positional distributions are equally small across the different landmarks. For emotionally expressive faces with large variability in mouth and eyebrow shape, one can set AFA to align using only the eye landmarks. The results of eyes-only alignment of emotionally expressive faces is shown in Figure 9. We recommend that GPA-based alignment should be the default method for face studies that have no specific theoretical stake in how face alignment should be done. As far as we know, AFA is the only open-source implementation of GPA that includes full unit-testing, including a test of the entire process using a novel method described in Performance evaluation.

Moss’s egg aperture. Figure 6 shows the construction of a landmark-based facial aperture shaped after a Moss’s egg. The Moss’s egg is a combination of 4 circles whose centers are colored discs with corresponding colors in this image. The top part of this aperture lies above a horizontal line connecting the blue and green discs and is made up of the top half of the orange circle. The middle part lies between the horizontal line and another that connects the intersections (not shown) between the red circle and each of the green and blue circles. This middle section of the aperture is the intersection of areas within the green and blue circles. Finally, the bottom part of this aperture is the red circle. The orange disc lies ¼ of the way down a vector that goes toward the mean of landmarks (shown in Figure 4) from the average of all eye landmarks. The radius of the orange circle is 47/100 of the width of an ellipse that is centered on the mean of all landmarks and just fits all the landmarks across all faces. The radii of the green and blue circles are each 2 times the orange radius. The green and blue discs each lie a distance from the orange disc equal to the orange radius and are aligned with image horizontal. The vertical position of the red disc is the average y position of the nose-tip landmark across all faces, and its horizontal position is the same as the orange disc. The radius of the red circle is calculated so that it intersects the green and blue circles exactly once.

Figure 6

Warping. Warping is a non-affine image transformation; it changes the shape of image content. In the context of AFA, the goal is to warp the face so that facial landmarks are in new positions. Those new positions can be defined however one wants. When morphing from one facial identity to another, the new positions will be a weighted sum of landmarks between the 2 identities. When constructing an enhanced average, the new positions will be the average of landmarks across instances of the same individual. AFA follows the standard warping algorithm used in popular Active Appearance Models [33]. AFA warping will use Delaunay triangulation to decompose an image into triangles whose vertices are the facial landmarks detected by DLIB. Corresponding triangles, defined by the same vertices, are calculated for the new positions (the desired landmark locations). Image content in the source triangles will then be mapped on to corresponding triangles defined by new positions, using bilinear interpolation. This same warping procedure forms the backbone of many influential generative models of facial appearance [34]. As far as we know, AFA provides the first open-source implementation of the warping algorithm in Active Appearance Models that also has full unit-testing of critical algorithms (see Performance evaluation).

Organization of code. Common usage of AFA involves direct use of functions defined in the core module make_aligned_faces. This core module depends on a set of independent submodules that are organized by functionality (See Figure 7). Normally, users will not need to reference submodules. However, functionality of AFA can be extended by adding to submodules or replacing calls to them within make_aligned _faces.

Figure 7

Performance evaluation

All quality-control results are fully documented in the software (https://github.com/SourCherries/auto-face-align/tree/main/results). One must simply request the face databases (described in the next section) from their corresponding authors and put those faces in the appropriate folders; e.g., male faces from the GUFD database should go into results/GUFD-male/.

In this manuscript, we highlight the (1) unit-testing of GPA implementation; and (2) empirical evaluation of the entire alignment process on multiple face databases. For brevity, other tests (e.g., evaluation of DLIB on multiple databases) are left out of this manuscript but can be found in the software documentation.

AFA also comes with unit-tests which help to confirm the validity of primary algorithms within AFA. All of these reside in the folder https://github.com/SourCherries/auto-face-align/tree/main/alignfaces/src/alignfaces/tests. Many of the functions required for GPA were extensively unit tested, including the main function that computes an ideal reference shape generalized_procrustes_analysis(). Because of its critical role in AFA, that function alone was subject to 7 different unit tests. This included two different comparisons of AFA’s GPA output against that of a popular R package which does the same analysis [35]; and also a unique test of validity which we call the “donut test”. The donut-test. Input shapes were pseudo-randomly generated triangles. Half were random affine perturbations of a “base-shape”. For each of these shapes, we also include a corresponding shape that is equidistant and oppositely directed from the base shape in multidimensional space. The base shape itself is not included as an input shape. And critically, the base shape cannot be recovered from simply averaging all the shapes. However, GPA should be able to recover the base shape if our AFA implementation is correct. AFA recovers the base shape with high precision.

We evaluated the final product of all alignment steps on multiple face databases. The databases we used: the CAS-PEAL database of Chinese faces [36]; the Glasgow Unfamiliar Face Database GUFD [37]; faces from the Kent Face Matching Test KFMT [38]; and emotionally expressive faces from the NimStim database [39]. The first 3 databased comprise mostly neutrally expressive faces, which is the most typical kind used in face perception research. The NimStim database is has faces with 8 different categories of emotional expression, along with open- and closed-mouth versions of each. We report results separately for the 3 neutral and the NimStim databases. These 3 emotion-neutral databases were chosen to reflect the diversity of frontal-pose face photos that are normally used psychology and neuroscience research. In particular, GUFD and KFMT are tailored to the study of face perception in realistic scenarios, so the faces from these databases were photographed with variable quality cameras and resolutions and allowed for natural poses instead of a strictly frontal-parallel orientation of the face. Photographic subjects in both the GUFD and KFMT databases were allowed to have their hair worn in any fashion. And unlike the other two databases, photographic subjects in the KFMT database were allowed to wear glasses. Image resolutions in width-by-height (pixels) are 360 × 480 and 350 × 518 for the CAS-PEAL and GUFD databases, respectively. KFMT was divided between KFMT-DC and KFMT-ID which were photos taken in-lab (283 × 332 pixels) and pre-existing photo-identification (142 × 192) obtained from the same people.

We can evaluate the final outcome of AFA alignment for each face database by (1) visually confirming that the mean of all aligned faces retains high local contrast around the 51 landmark areas that were used for alignment; and (2) visually comparing the distributions of aligned points for each landmark against the mean aligned face, in order to confirm the small size of these distributions relative to facial features. Additionally, for the neutral databases we can visually confirm the uniformity of spatial distribution size across landmarks, as GPA is meant to align each landmark position across faces as best as possible. Results for the neutral face databases are shown in Figure 8, and they provide positive confirmation of AFA alignment validity. For the NimStim faces, we opted to align only by eye landmarks. Therefore, uniformity of spatial distribution size across landmarks is not expected; instead we expect extremely small spatial distributions for each of the eye landmarks and a high degree of local contrast around the eyes. Results for the NimStim are shown in Figure 9, and they provide positive confirmation of AFA alignment validity.

Figure 8

Figure 9

Quality control

Both unit and functional testing has been performed on AFA (https://github.com/SourCherries/auto-face-align/tree/main/alignfaces/src/alignfaces/tests and https://github.com/SourCherries/auto-face-align/tree/main/results).

Operating system

AFA should function on any software capable of running Python 3.12 or greater. Current operating systems that AFA has been tested on include macOS Sequoia, Windows 11, and Ubuntu 20.04.

Programming language

Python (version 3.12 or greater.)

Additional system requirements

The minimum system requirements needed to run Python 3.12 or greater.

Dependencies

dlib >= 19.24.2

scikit-image >= 0.24.0

matplotlib >= 3.9.2

List of contributors

Carl Gaspar is the primary developer, and current maintainer of the AFA package. Oliver Garrod helped to develop the warping module, advised on package organization and is a co-author of this article.

Software location

Language

English