Identifying Inconsistencies in Data Quality Between FAOSTAT, WOAH, UN Agriculture Census, and National Data

4.1.1 Method 1: Normal distribution of growth rates

The first investigation was to identify yearly rates of change that are highly improbable (or need ‘ground truth’ verification from expert or in-country sources). To achieve this, we took all of the yearly growth rates for the population of animals in a country and then plotted them to produce a histogram overlaid with the normal curve. Growth rates should have minimal variation between each other, so most yearly growth rates should have similar magnitude and more unusual growth rates will appear less often. Performing analysis with a normal curve is done because animal populations should be relatively similar each year since animal populations typically have a sustainable growth rate unless perturbed by special circumstances (natural disasters, climate change, changing agricultural/economic policies, conflict). Each yearly growth rate will fit onto this normal curve and should be similar to past growth rates (Pardoe et al., n.d.). Normal distributions can also display standard deviations which is desirable because data points outside of the third standard deviation are generally considered to be outliers.

Using the normal curve, points outside three standard deviations can be labelled as outliers and it can be concluded that there is a high chance that they are internally inconsistent (Oracle, n.d.). This means that any year that the animal populations grew or shrank too quickly was most likely a suspicious data point that needed investigation and annotation for users of the data. This method is to compare yearly growths for a single data source at a time. For example, Figure 3 shows three standard deviations in either direction of the rates of change for the data shown in Figure 1. The blue vertical line is the mean growth rate, the others being standard deviations. It is evident from the example that there are no data points outside the third standard deviation indicating there are no outliers in this data.

Figure 3

Inconsistencies were found in the FAOSTAT data as elucidated by Figure 2. When looking at the growth rates for sheep in Ethiopia (Figure 2) between 2008 and 2011, the graph has more higher positive and lower negative slopes. When looking at the rates of change for this in Figure 4 it can be seen that there is an outlier during this period. Over the same range, the rates of change for the WOAH data do not have an outlier at this point, indicating both FAOSTAT and WOAH have possible issues with data for this year. WOAH has consistency issues for data on Sheep in Ethiopia as well. All of WOAH’s data falls between the third standard deviation lines, but the third standard deviation exists at more than 40 percent growth. This shows the data changing rapidly each year. This is evidence that WOAH data is problematic for Sheep in Ethiopia. Once again, this does not immediately indicate that the WOAH numbers should be ignored by users, but they should be annotated and investigated (if possible).

Figure 4

4.1.2 Method 2: Interquartile range analysis

The IQR is used to calculate the variance in magnitude for each quartile of the data. It also indicates outliers outside each data source’s upper and lower fences. These fences visualise the first and fourth quartiles of data. These outliers can be easily seen and then investigated further. The IQR shows the median rate of change for the animal population, as well as shows the second and third quartile as boxes around the median. Figure 5 shows the spread of the yearly population rates of change for cattle in the United States of America. Assuming the population is large, good data should not have any outliers and the IQRs for each source should look similar to each other.

Figure 5

It is evident from Figure 5 that FAOSTAT has a much larger data spread for the rate of changes than the other sources. This could be because it has more data, so the uncertainty of the WOAH and national agricultural data is higher. Some examples of this are cattle in Greece where FAOSTAT data exists between 1960 and 2019 whereas WOAH data only exists between 2005 and 2015, or goats in Ireland where the FAOSTAT data also exists between 1960 and 2020, whereas WOAH data only exists between 205 and 2019. More evidence of this can be found using the general data viewer tool provided. Despite the differences in how much data is collected between all the sources, the median yearly rate of change is similar between all sources. The biggest difference between all three sources (excluding UN Agriculture Census Data due to its small sample size) is that FAOSTAT has much larger upper and lower fences. This is most likely due to FAOSTAT containing more data than either national agricultural data sources or WOAH. So, this indicates that there is a high probability that all three data sources are consistent and support each other.

When comparing the rates of change for sheep in Ethiopia the IQR looks very different (see Figure 6). Figure 6 shows a large data spread for WOAH despite WOAH having less data than FAOSTAT. The second and third quartile of WOAH is almost twice the size of that for FAOSTAT. It also has long error bars and an outlier far beyond the top error bar. Excluding the outliers (assuming that they indicate an inconsistency), the data from WOAH for Ethiopian sheep is much less internally consistent than that of FAOSTAT. There are other examples of WOAH having data with extreme outliers. One example is goats in South Africa. In Figure 7, the extreme outliers are visibly showing a –90% population rate for a single year with no other points coming close to it. Excluding the outliers (Figure 8), it is clear that WOAH and FAOSTAT have similar median rates of change, but WOAH has only two years below the median. This is very surprising as it means it had almost half of the years surveyed at 0 percent population growth (this would not seem likely as animal populations usually change every year for a country of this size). FAOSTAT contradicts this as its median rate of change is 0.121 percent growth per year with no years of zero percent change. This illustrates a difference in the nature of the data recorded by WOAH and FAOSTAT.

Figure 6

Figure 7

Figure 8

Figure 8 shows the interquartile ranges for goats in South Africa and there are some clear outliers in the data for WOAH. Figure 9 shows the same data excluding the outliers and shows a fairly similar IQR box plot showing the data is mostly identical. The only other possible inconsistency is that WOAH has 4 data points with zero rate of change. Investigating the WOAH data points further exposes some suspicious data points. 2005–2007 has the exact same population between all years and 2008–2010 also has the exact same population between the years. However, 2018 is most likely the largest data inconsistency because that year WOAH reported 54.2 million goats in South Africa, which is suspicious since it reported 5.4 million goats. Since FAOSTAT reported 5.4 million goats that year it is almost certain the WOAH put the decimal place in the wrong spot and actually meant to report 5.42 million goats.

Figure 9

4.1.3 Method 3: Trailing five-year analysis

The final analysis performed on the data sources was by finding the trailing five-year population averages across all four data sources. Then the rate of change between a trailing five-year average and the previous trailing five-year average is found. This process is performed for each data source on all trailing five-year groups of data and then plotted to identify the differences. This method of analysis was chosen for our study because it will show the average growth over five years. This will show general trends more than sudden yearly changes. If for one year there was a large growth of an animal population and the next year that animal population declined by the same amount, over the trailing five-year span the average will not reflect these sudden changes. So, if this does show a large change over those five years then it is evidence that that time span should be investigated further as there would have been something influencing that period to have such change. An example of the usefulness of this analysis is seen in an examination of the chicken population in the United States of America. It is clear from Figure 9 that there was more than 200 percent growth between the trailing five-year averages ending in 2006 and 2011 for FAOSTAT. This is far beyond what is normal for this time range when comparing this time span’s rate of change to the historic rates of change. The half-decade following also had nearly 50 percent growth, showing this population growth was not a one-off incident. It can be determined from this that either the United States of America massively increased its chicken population post-2006, the method for counting chickens was changed during this time, or there is an inconsistency in the data.

Figure 10 shows the cattle population in Botswana. In the FAOSTAT data, it is clear that the population has periods of rapid growth then stagnation and decreasing population numbers. However, the data from WOAH shows a brief data period where the population has been growing at a much faster rate than what FAOSTAT has shown over the same period. WOAH shows the population growing very fast between the 2006 and 2011 averages, with almost 35 percent growth over the half-decade. This is in stark contrast to FAOSTAT which shows an 8 percent decrease in population over that same period. That’s a large difference! Consulting the actual animal populations in Figure 11, it is clear that the animal population measurements between both sources are rarely identical and WOAH’s numbers are stagnant between 2012 and 2018. This is the opposite of what FAOSTAT is reporting with a large decrease in cattle populations over that same period. Between these two data sources, it is very hard to tell which source is most accurate. WOAH may be more accurate as it shows a population more similar to the pre-2005 population average for FAOSTAT. But, FAOSTAT and the UN Agriculture Census data are identical in 2015 which indicates that these population numbers have support as well. This shows that it is difficult to determine what is the most ‘reasonable/representative’ population numbers for cattle in Botswana.

Figure 10

Figure 11

Figure 12 shows the cattle population in India (excluding buffaloes and mithuns) and it is clear that the three sources have very similar data. Each change for a trailing five-year average is less than 10 percent between its peers so this is a very reasonable rate of change. FAOSTAT data also follows a reasonable curve having few massive gains and drops between years. This is a good example of what consistent data should look like. This shows that FAOSTAT, WOAH, and UN Agriculture Census data can be similar and corroborate animal population numbers for specific years and specific countries.

Figure 12

4.2.1 Policy change

With the growing need for countries to reduce their environmental footprint and the high level of greenhouse gas emissions from livestock production, many countries are incentivising farmers to reduce their livestock population (Lyman and Nalley, 2013); for instance, when Czechoslovakia split into Czechia and Slovakia in 1993 and each country joined the European Union (EU) in 2004. In the 2000s, the livestock populations of Czechia and Slovakia declined. At face-value, the decline may appear to be a hangover from the split between the two countries (e.g., hurt livestock sectors, issues reporting livestock in new government structures, changes in data infrastructure and reporting). However, upon joining the EU, countries are required to be compliant with the EU’s environmental regulations (e.g., reducing livestock populations, re-wilding land, switching to growing crops with lower environmental impact), leading to changes in livestock structure and populations. By identifying the inconsistencies in the data, the decline in livestock population in Czechia and Slovakia post-split could be investigated, showing that countries attempting to be compliant with the EU’s sustainability goals can impact the trend of livestock population over time.

4.2.2 Inaccurate livestock population surveys

Data collection methods, such as paper surveys, may impact the reliability and accuracy of data. For example, surveys performed by governments often require farmers to fill out a form to report data about their herds. However, farmers may underestimate the livestock population in fear that the government may tax them more, leading to inaccurate or underestimated livestock populations for a country. This has been reported in Australia, where there is evidence that in the national livestock census, their cattle population is underestimated (Geoffry Fordyce et al., n.d.).

Cultural differences in the significance of the value of livestock may also contribute to inaccuracies in livestock population surveys. For example, pastoralist communities in Ethiopia sometimes do not have access to bank accounts and instead, keep their wealth in the animal herds that provide their income and livelihood (Catley et al., 2014). So when asking these groups the population size of their livestock, it is like asking somebody what is the balance of their bank account, which is very private, so they could give false answers to keep it private or make themselves appear more wealthy than they actually are (Catley et al., 2014). Some cultures also believe talking about herd size can bring bad luck (International Wool Textile Organization, n.d.). All these factors can make herd owners hide the true sizes of their herds.

Some sources also do some minor editing on the data to make the data more reasonable. FAOSTAT will add missing dates and perform some corrections on incorrect data to make a complete dataset (FAOSTAT Statistical Standard Series – Imputation, 2019). This is to fill in data between missing dates from a country or for errors such as an incorrect decimal point. They then add flags on the data to indicate if a data point is official, unofficial, forecasted, or imputed. WOAH in contrast does not change the data they receive from countries since they view themselves as a data distributor, not collectors or editors.

4.2.3 Environmental factors

Drought can affect population numbers due to the mortality of animals and because farmers can submit false numbers to get greater government assistance. Some countries are especially prone to drought as they lie in arid areas where rain is the primary source of fresh surface water. One example of this is Ethiopia, where malnutrition or dehydration and other effects of drought caused between 61 and 100 percent of excess livestock deaths during drought periods (Catley et al., 2014). This can cause sudden drops in livestock population counts and this can be seen in livestock surveys. Governments may seek to replace livestock lost during natural disasters to support their farmers. As a result, some farmers report higher mortality than what actually occurred to receive more support and resources from their government (Catley et al., 2014). These can all cause sudden changes in livestock population numbers that might appear inaccurate when compared to past and future population numbers but are true.

4.2.4 Diseases

Livestock infected with disease can impact population trends due to large-scale culling events to eradicate or prevent disease spread. For example, the 1985 Bovine Spongiform Encephalopathy (BSE, also known as ‘mad cow disease’) outbreak in the United Kingdom led to high levels of cattle infected by BSE. The outbreak resulted in beef and cow import bans from the United States and the EU due to the hazards to other livestock and human safety. As a result, Britain culled and incinerated over one million cows in 1986 in attempts to eradicate the disease from the country (CNN, 1999). Therefore, disease outbreaks can lead to rapid declines in livestock population due to large-scale culling events over a short time period.

4.2.5 Nomadic livestock

There are many different methods of farming livestock in the world and most of them are stationary. Farms like ranches and factory farms exist in one place, and it is easier for governments and regulatory bodies to count the population since they know where they are. In more traditional countries there can be nomadic herders and farmers. These groups migrate with their herds to find better pastures and resources as humans did thousands of years ago. This can be an issue for surveyors because it can be hard to find these people to gather information on their herd sizes. An example of this is Camelids in Algeria, where farmers move their camels between Algeria and Niger for their herds to graze (Faye, 2020). There also exist cases of this between Saudi Arabia and Qatar (Faye, 2020). This can pose an issue for surveyors because there is no address to send surveys to nomadic farmers; or, if they are in another country when being surveyed, they might not be counted. This can make it appear that the animal populations are smaller in certain countries than they actually are.

4.2.6 Regionality

Inconsistencies are present in all regions. However, different regions may have different factors that could be impacting the populations. For example, Ethiopia has experienced droughts frequently in the past (International Federation of Red Cross and Red Crescent Societies, n.d.) and this has negatively affected its livestock populations during these periods. In contrast, Canadian livestock is less affected by drought due to the country’s large size and how spread out livestock farming is. Free trade agreements affect the Canadian livestock populations more than weather influences. The Canadian cattle population started increasing after Canada joined NAFTA and was able to more easily sell its beef to the USA (Government of Canada, n.d.). Ethiopia is less affected by free trade as it cannot as easily get its beef to the USA, which is the largest beef consumer in the world. So, inconsistencies are found in all countries’ population data; however, the reasons for these inconsistencies can be different.