An Open-Source Adaptive Comparative Judgement App for Technology Education Research and Practice: Alpha Version

Preparation of Portfolios

In typical ACJ software solutions, portfolios are digitized pieces of work, such as PDF files, slideshows, or webpages, which are uploaded to the system for computer-based and mobile-based assessment. This is one of two options available in the app. For this option a user needs to have the relevant files hosted on Google Drive and shared such that anyone with a link can view the file. The files do not need to be in the same folder or shared by the same Google account, but it may be logistically easier to compile the relevant files into a single Google Drive folder and make that folder visible to anyone with the link. That will ensure the files within that folder are shared correctly. Next, the links need to be uploaded to the app. For this, a user can upload an Excel worksheet (a downloadable template is available) or input the information manually. Whether put into an Excel worksheet or entered manually, each portfolio needs to be given a name and have a link provided. Users can name portfolios any way they choose, and if no link is provided the file viewers in the next steps will display a note that no file has been provided. Figure 1 shows an example of this step where an Excel worksheet was used to upload links for six portfolios.

Figure 1

The other option is a manually defined list. In technology education, student work is often a manufactured physical artefact. This could be a prototype or finished product. Newhouse (2014), in a study using ACJ with digitized outputs from visual arts students, observed that assessors noted the quality of digital representations could be quite poor. For example, photographs of work could be blurry, or not accurately represent texture, details, scale, or dimensions of the actual work. Similar comments were made regarding video materials with assessors commenting on shaky videos. There was collective critique for both photographed and videoed works of distracting backgrounds, poor image resolution, and poor lighting, and one assessor noted the digitizing of physical works can make it more difficult to see issues with the work. This is not an argument against digitizing work in technology education. It is an acknowledgment that the use of ACJ should be considered beyond the existing computer-based paradigm and that when works are digitized, it is important that such representations are fair. For this approach, a user again has the option of uploading a list of portfolio names via an Excel worksheet or entering the names manually. Figure 2 shows an example of this step where an Excel worksheet was used to upload links for six portfolios.

Figure 2

Random Pairs

Once a user has defined a list of portfolios to include in the ACJ process they can progress to either making random judgements or to making manual judgements. It is recommended that a selection of random judgements is completed first. There are three aspects of the random judgements phase that are important to note which relate to (1) the number of random judgements which should be completed, (2) portfolio chaining, and (3) how portfolios are selected for comparison.

In terms of how many random comparisons need to be made, this decision is made by selecting a number of rounds to complete. A round is defined as the creation of pairwise comparisons such that each portfolio in the sample is paired with one other portfolio. For example, if the ACJ process is being conducted with a sample of 10 portfolios, a round will include 5 comparisons with each portfolio included in one comparison. It is typical that six rounds of random judgements are used in this initial rough sorting round, possibly because this was the amount used in the e-scape project (Kimbell et al., 2009). That said, in this app, the user can select any number of random rounds to begin with, they can complete all specified random comparisons and then generate more, and they can progress to adaptive and manual pairings and then return to add more random comparisons.

A critical feature of the random pairings is portfolio chaining (Pollitt, 2012). Chaining is important for two reasons. As the primary output of the ACJ process is a rank order of included portfolios, it is important that each portfolio is connected to all other portfolios either directly or indirectly so that the rank can provide relative differences. Additionally, while the next section will explain in detail how adaptive pairings are determined, portfolios need to be chained for this step. To take the need away from users to plan for this, when any number of random pair rounds are determined the system checks at the end of those rounds whether from the created rounds the full sample of portfolios are chained or not. In other words, if six random rounds are specified, the full set of pairs for the six rounds are immediately determined and then the system checks to see whether, from these six rounds, all portfolios are chained or not. If they are, nothing occurs beyond presenting the user with the random pairs for these rounds. If they are not chained, pairs are automatically created purposefully to ensure full chaining is accomplished and these pairs are added to the end of the initially determined random rounds.

Figure 3 provides an illustration of this for five portfolios where each arrow illustrates a pair for comparison. In the left image, portfolio one is compared with portfolio four, portfolio four is compared with portfolio two, and portfolio five is compared with portfolio three. These three random pairings essentially create two chains, portfolios one, four, and two, and portfolios five and three. The goal instead is one singular chain. The image on the right shows the addition of one additional pair between portfolios one and five which results in the connection of all portfolios in the sample into a single chain.

Figure 3

A final feature of the app relates to how portfolios are selected for comparison in these random pairs. In this app random pairs are completely random. In each round the first portfolio in the list is randomly paired with another portfolio from the list, and then these are excluded from future pairings in that round such that the next portfolio in the list is randomly paired with one of the remaining available portfolios. Once the pairs for the round are all decided, the process occurs the same way for subsequent rounds and there is no relationship between rounds.

In contrast to this process, Pollitt (in Kimbell et al., 2009) argues for a Swiss tournament approach due to increased efficiency and discriminability between portfolios. This approach to rough sorting has the same random approach to the initial round with the outcome being each portfolio having a score of 1 (if they were deemed the better of their pair) or 0 (if they were deemed the worse) at the end. In subsequent rounds, portfolios are randomly paired but, where possible, only with portfolios of the same score. In the second round, for example, portfolios with a score of 1 are paired randomly with others with a score or 1, and those with a score of 0 are paired with others with a score of 0. The current app may be less efficient than one which uses Pollitt’s recommended Swiss tournament approach at this stage, however this decision was based on bridging the arguments for and against the adaptive component in ACJ. Bramley (2015), on this issue, notes that:

While the proposed solution from Bramley (2015) was to explore a reference set against which new works are adaptively compared, and this has been investigated (Verhavert et al., 2022), the presented work relates to the development of an app to enable ACJ research as opposed to a methodology for how ACJ should be used for assessment. The use of complete randomness for this rough sorting instead aims at being a balance in these views on comparative judgement.

Beyond the decision making as to how portfolios will be selected for comparison, from a use perspective the user is initially presented with the decision to choose a number of random rounds. A note suggesting that six rounds are recommended is provided but the user can choose any amount they wish. Once they generate the list of comparisons, the portfolios for comparison are then presented on the screen (Figure 4). A feature not typical of existing ACJ tools is that the user can choose to see any number of upcoming comparisons in the list and cycle through these to make the comparisons in any order they wish. This was introduced for pedagogical reasons. It may be useful to generate several random rounds for example and assign a selection to students in a way that the students can simultaneously make comparisons and return their decisions to an educator for entry into the app in the order the students return them in. As the app is currently only a single user/device system, this feature at least allows for the several upcoming comparisons to be viewed such as they can be distributed physically to multiple judges and the decisions can be inputted on the single device for a close to multiple user experience.

Figure 4

Once the random comparisons are made, a user can choose to generate more completely random rounds or progress to adaptively generated comparisons or to manually defined comparisons. There has yet to be any research on whether there is value in iterating between randomly and adaptively generated comparisons, so at this stage all that is being commented on is that the functionality is included to permit moving between the different types. Users should at least complete all initially specified random comparisons to ensure complete chaining, but after that they would be free to iterate. There may be a benefit to producing several random rounds to get a longer list of comparisons in the current version of the app as it does only run from a single device. In so doing, these random comparisons could be assigned to a group of people. In contrast, the adaptive comparisons are generated round by round so less pairs are generated at a time. Beyond this, there will need to be further research to clarify optimal types of comparison for different purposes.

Adaptive Pairs

After completing a set number of random pairwise comparisons to derive initial ability scores for each portfolio and to ensure chaining, users can progress to adaptive pairings if they wish. In terms of the judging experience this is equivalent to the random pairs in that a decision is to be made on which portfolio is the better/worse. As such, no figure is presented to illustrate this stage as it appears identical. The main difference at this stage lies solely in the process by which portfolios are paired for comparison. There are several algorithms which can be used to adaptively pair portfolios (e.g., Verhavert, 2018). The algorithm used in the app has limitations which are described later, but it is being used as a starting point for more systematic investigation of appropriate ways to manage this part of the process. Instead of random pairs, the adaptive rounds are generated for portfolios by pairing them based on the Fisher Information statistic. The Fisher Information statistic (Equation 3), in this case is the product of the probability of one portfolio in a comparison winning and the probability of the other portfolio winning. The most information is gained when the probability of either portfolio winning is 0.5, which would occur when the portfolios being compared are equal in perceived quality. In other words, more information is gained from a comparison between two portfolios that are close together than from a comparison between portfolios very far apart in the rank (Pollitt, 2012). For each round, initial ability scores are determined by fitting the Bradley-Terry-Luce (BTL) model, these are used to compute the Fisher Information statistic for each possible pair of portfolios, and then pairs are determined by selection of those which will provide most information. The process occurs as follows:

1. A number of rounds of random pairs are completed, such that each portfolio in the sample is connected directly or indirectly, and this results in a dataset in which the number of wins each portfolio gains is recorded, and the number of wins and losses each portfolio has gained against all other portfolios are recorded.

2. Based on this data, the BTL model is fit to derive ability scores by

   Equation 1

   ${\alpha }_i=\frac{{\displaystyle {\sum }_j\frac{w_{\mathit{\text{ij}}}{\alpha }_j}{{\alpha }_i+{\alpha }_j}}}{{\displaystyle {\sum }_j\frac{w_{\mathit{\text{ji}}}}{{\alpha }_i+{\alpha }_j}}}$

   where $w_{\mathit{\text{ij}}}$ is the number of wins portfolio i has against portfolio j, $w_{\mathit{\text{ji}}}$ is the number of wins portfolio j has against portfolio i, ${\alpha }_i$ is the ability score estimate of portfolio i, and ${\alpha }_j$ is the ability score estimate of portfolio j (Hunter, 2004). Initially, all ability scores are estimated as 1 and then normalized to maximum likelihood estimates. See Buckley (2024) Supplementary Material for a more detailed explanation of this formula in the context of technology education research.

3. These ability scores are converted to logits by

   Equation 2

   $\mathit{\text{logit}}({\alpha }_i)=\mathit{\text{ln}}\left(\frac{\exp ({\alpha }_i-\max (\alpha ))}{{\displaystyle {\sum }_{j=1}^n\exp }({\alpha }_i-\mathit{\text{max}}(\alpha ))}\right)$

   where ${\alpha }_i$ is each ability score for set $\alpha =\{{\alpha }_1,{\alpha }_2,\dots ,{\alpha }_n\}$.

4. A matrix is then created which pairs each portfolio with all other portfolios, and the Fisher Information statistic (I) is computed for each pair of portfolios (e.g., portfolios i and j) by

   Equation 3

   $I_{\mathit{\text{ij}}}=P_j({\alpha }_i\left(1-P_j({\alpha }_i)\right)$

   where $P_j({\alpha }_i)$ is the probability that portfolio i wins in a comparison with portfolio j, computed by

   Equation 4

   $P_j({\alpha }_i)=\frac{\exp ({\alpha }_i-{\alpha }_j)}{1+\exp ({\alpha }_i-{\alpha }_j)}$

   where ${\alpha }_i$ is the ability score of portfolio i and ${\alpha }_j$ is the ability score of portfolio j.

5. The matrix is examined, and the portfolio pair which has the highest Fisher Information statistic is set as the first pair in the adaptive pair round, and these portfolios are removed from the matrix. The matrix is then re-examined for the pair that provides the next highest Fisher Information statistic, which becomes the second pair for the adaptive pair round. This process repeats until all portfolios are put into pairs, and these pairs define the adaptive round.

Manual Judgements

Beyond random and adaptive judgements, the app permits the addition of manually defined judgements. This was included for both research and educational purposes. If being used in an educational setting, it may be valuable for an educator to ask a student(s) to compare two specific portfolios (Figure 5). This could be formative in that the educator may wish to see what students think about certain portfolios which have certain characteristics.

Figure 5

From a research perspective there can be value in the capacity for making comparisons between specific portfolios, such as if two previously created ACJ ranks were being compared (cf. Buckley et al., 2023; Buckley & Canty, 2022; Seery et al., 2022; Verhavert et al., 2022). It may also be useful for judge calibration-type activities where assessors may be being trained or informed about standards and whomever was leading this activity may want to orchestrate which portfolios are presented for comparison. On this page of the app a user sees the full list of portfolios in the sample and can select two to compare based on dropdown menus. If this process is being done with digital portfolios the file viewers will update based on the dropdown menu selection (Figure 5).

Viewing the Results of Pairwise Comparisons

There is a page on the app where users can view the results of each pairwise comparison made and entered into the judging session. Each comparison made is recorded in a table which shows both portfolios (portfolio.a and portfolio.b), the result (1 indicates that portfolio.a won and 0 indicates that portfolio.b won), and the person or judge who made the comparison. This is illustrated in Figure 6, however it should be noted that from this stage onwards the Figures come from a trial session with nine portfolios instead of the six which have been shown thus far. And that viewing this information may not be very useful in and of itself.

Figure 6

The reason for this page is for a different feature of the app. The table which can be viewed is simply a table from an Excel worksheet, and the information in this table is used for the analysis and ranking of portfolios in the next step.

A user can upload their own table to this view for analysis. This creates several possibilities:

1. A user who finished or paused a session could download this table, and at any later date resume making comparisons by uploading this table and picking up where they left off.

2. A user can download the results of a judging session and add/remove/edit any comparisons they want to. They could therefore, for example, remove all comparisons made by a particular judge simply by deleting rows in the Excel sheet and reuploading the table. Alternatively, they could create several tables following an ACJ session based on sub-groups of judges, and compare ranks produced by these different groups.

3. If two (or more) ACJ sessions were conducted with sets of portfolios {a₁, a₂, a₃, a₄, …, a_n} and {b₁, b₂, b₃, b₄, …, b_n}, a user could merge the two results table into one and upload it on this section of the app and make a series of judgements to merge the ranks (Buckley et al., 2023). Alternatively, these different ACJ sessions may have the same portfolios but have been conducted with different groups of judges, opening possibilities for group comparisons for educational and research purposes.

Analyzing the Results

On the analysis page of the app the information from the results table from the previous step can be analyzed through fitting the BTL model as described in Equation 1. An important and novel aspect to the app is that any set of pairwise comparisons can be recorded in a table of the same format to the results table, be uploaded to the app and be analyzed to compute a rank of the included portfolios. The judgement capacities of the app do not need to be used. They are useful in the selection of portfolios for comparison and the viewing of portfolios, but this page enables BTL model fitting and analysis of data coming from a wider variety of sources beyond an ACJ process. This may be of particular interest to researchers or educators who can capture comparison-type decisions in a variety of ways and for a variety of purposes as this will enable data analysis and presentation.

Further, it also provides the specifications of the analysis in full. For example, the approach to fitting the BTL model in this app is through the sirt package in the statistical analysis software RStudio (Robitzsch, 2021). This package contains several functions for item response theory modelling. It is the btm() function which is of specific interest for ACJ as it can be used to compute a BTL model with a dataset of outcomes from pairwise comparisons (e.g. Buckley et al., 2023). The version of this package being used is noted on this page as well as the R version and date of analysis. Several pieces of statistical information are provided in raw format which may be of interest, although these are presented more clearly based on typical usage in technology education on at the next stage. For reference, scale separation reliability (SSR: Verhavert et al., 2018) is provided and denoted by “MLE Rel”, portfolio ability scores/parameter values are denoted by “theta”, and infit statistics are shown for portfolios and judges. It should be noted that the SSR reliability coefficient has been given different names throughout the literature (cf. Bramley & Wheadon, 2015), such as by Pollitt (2015) who referred to it as the “judge consistency coefficient”

Figure 7

Generating Outputs

On the final page of the app a user can select the option to generate outputs. This takes the analysis information from the previous page and converts it into a more accessible presentation. For example, Figure 8 illustrates a table providing the rank and ability score (denoted as “parameter value”) for each portfolio such that performance can be compared. Standard error and infit statistics for portfolios are also provided. This table is also downloadable such that users can analyze further if they wish.

Figure 8

For ease of use and communication purposes, the information from this table also generates as a plot which can be downloaded as a PDF file (Figure 9). This plot also provides the SSR reliability estimate which in this case is SSR = 0.823.

Figure 9

A table similar to the version shown in Figure 8 for judges is also produced. It includes judge agreement levels and misfit statistics. A plot for this table is also produced which provides details on the mean misfit value, the misfit threshold (mean misfit + 2 standard deviations), and the critical misfit value of 2 (Figure 10). Judge misfit statistics can be compared to these values to see whether any judges were particularly misaligned with the rest of the cohort and this can be investigated further to understand why this may have occurred.

Figure 10