The Problem of Institutional Fit: Uncovering Patterns with Boosted Decision Trees

Introduction

The concept of contingency is a defining feature of the Bloomington School of institutional analysis and related strands of research in environmental governance and social-ecological systems (SESs) (Ostrom and Ostrom 1999, Young 2002, Sproule-Jones 2005, Acheson 2006, Ostrom et al. 2007). Indeed, there is a growing consensus that institutions and the outcomes they generate are heavily influenced by the complex social-ecological contexts in which they are found (Galaz et al. 2008, Young 2008, Cox 2012, Epstein et al. 2015) and that there are no institutional panaceas for addressing environmental problems (Brock and Carpenter 2007, Ostrom et al. 2007, Morrison 2017). However, despite many years of research and growing interest in middle-range theories and archetypes (Magliocca et al. 2018, Meyfroidt et al. 2018, Oberlack et al. 2019), progress in understanding the contexts in which different institutions are likely to be found and contribute to successful outcomes has been slow (Cumming et al. 2020).

Institutions, which are the formal and informal rules, norms and conventions that structure the incentives, opportunities, and constraints that actors face when making decisions (Crawford and Ostrom 1995), play a critical role in influencing how people interact with their environment and each other. However, institutions are not random features of the environment, but rather vary systematically across a gradient of social, and ecological characteristics (Leslie et al. 2015, Cumming and Epstein 2020). Protected areas, for example, tend to be situated far from major population centres (Joppa and Pfaff 2009) where there is less intense competition with alternative uses such as agriculture and resource extraction and associated political conflicts (Devillers et al. 2015). Communities exploiting mobile resources such as fish, wildlife, and water, meanwhile, tend to avoid quotas in favour of timing, location or technology rules owing to the relatively high costs associated with collecting and processing information about those resources (Schlager et al. 1994, Acheson and Wilson 1996, Cifdaloz et al. 2010). These and other patterns emerge over time as groups experiment with different institutions and retain those that deliver upon the social and ecological objectives of decision-makers. Indeed, research on the topic of institutional fit asserts that outcomes depend greatly upon the match between institutions and the contexts in which they are used (Cumming et al. 2006, Galaz et al. 2008, Young 2008, Epstein et al. 2015).

The literature on institutional fit has developed over the last few decades in the face of mounting evidence of the failure of theoretically optimal solutions to address environmental problems. This literature, which examines the fit between institutions and the context in which they are embedded (Epstein et al. 2015), offers a cautionary tale about the dangers of undermining local and traditional governance systems based on theories or models (Johannes 1978, Nalau et al. 2018) and transplanting institutions that work well in one context to others (Ban et al. 2009). On the other hand, it offers practical guidance about how institutions can be designed or adapted to different contextual conditions. Ultimately, the goal is to develop theory about the socio-ecological context (i.e., conditions) under which different types of institutions are most effective in steering behavior and solving environmental problems. For example, several studies have suggested that participatory environmental governance is more likely to contribute to successful outcomes when communities have strong local leaders and high levels of social capital (Singleton and Taylor 1992, Gutierrez et al. 2011; Frey 2017). Resources that cross jurisdictional boundaries (i.e., transboundary rivers, migratory species) and those that are closely connected through biophysical linkages (i.e., wildlife/habitat, predator/prey), meanwhile, tend to benefit from institutional mechanisms that facilitate coordination (Bodin 2017, Garrick 2018). These mechanisms can range from informal governance networks and arrangements that have been developed and adapted by communities over generations, (Bodin and Tengö 2012, Bodin et al. 2014) to large international organizations, such as Regional Fisheries Management Organizations and River Basin Organizations, designed to facilitate coordination of policies and sharing of information and resources (Rayfuse 2015; Milman and Gerlak 2020).

Recent research on the topic of institutional fit has used qualitative comparative analysis to better understand the efficacy of Ostrom’s (1990) design principles across different social-ecological contexts and outcomes. Vallury et al. (2022), for instance, found that the design principles were more likely to contribute to successful irrigation outcomes in South India when communities were small and the reservoirs they used were small in relation to the size of the population they served. Whittaker et al. (2021), on the other hand, uncovered patterns between several design principles and outcomes from lakes in Wisconsin to generate several useful insights for theory and practice. These include highlighting the importance of graduated sanctioning in contexts that are more ecologically conducive to the establishment of an invasive species, and where the density of buildings on the surrounding shoreline is higher. Although these and other insights from the literature on institutional fit have proved useful for policy and practice, the literature has struggled to move beyond a few core principles and advance a more general understanding of how complex social-ecological contexts affect the performance of institutions.

A generalised understanding of the relationship between contexts, institutions and their performance has been hindered by several factors. First, there are relatively few large-n social-ecological datasets that consistently capture variation in the complex social-ecological contexts in which institutions are found and outcomes are realized. Second, there is a general tendency among institutional scholars to view large-n datasets as a source of information for hypothesis testing, rather than an opportunity for applying exploratory methods for hypothesis development. To be clear, we acknowledge the importance of developing large-n datasets for hypothesis testing, but also believe that there are certain types of datasets or moments in the life cycle of those datasets where they hold greater value with respect to hypothesis development. Third and finally, most conventional quantitative research methods are best suited to address questions about the average “effects” of institutions within a population of cases and tend to struggle with problems involving large numbers of potentially influential and oftentimes correlated variables, interactions and non-linear responses (Agrawal 2003, Rana and Miller 2018). Fortunately, recent advances in machine learning have contributed to the development of a growing portfolio of tools that can help to respond to some of these challenges (Frey 2020a). These allow researchers to sift through large numbers of contextual variables to identify those that are more or less important for predicting outcomes and developing insights about how institutions and their performance vary across a gradient of social and ecological conditions.

This paper applies one such machine learning tool – boosted decision trees – to move beyond high-level claims that “context matters” and to develop insights about the specific contexts in which institutions are likely to be found and contribute to successful outcomes. It does so by drawing upon data collected as part of the International Forest Resources and Institutions (IFRI) program (Huntington et al. 2016, Ostrom et al. 2016) to examine the social and ecological contexts in which forest user groups successfully self-organize to develop and maintain rules for the use of forest resources (local rulemaking), and the contexts in which local rulemaking is associated with better forest conditions.

In what follows, we provide a brief review of research on the social, ecological and institutional factors that influence prospects for collective action and sustainable community-based natural resource management (CBNRM). We then describe the data and methods used in this study and summarize the results of two boosted decision tree models examining the conditions in which local rulemaking is likely to be found and contribute to better forest conditions. We find noteworthy differences between the contexts that appear conducive to local rulemaking and those in which local rulemaking is likely to contribute to successful ecological outcomes. Finally, the paper concludes by discussing the implications of these results for future research on CBNRM, and opportunities to advance contextually explicit theories of environmental sustainability.

Data and Methods

This paper draws upon data from the International Forest Resources and Institutions (IFRI) database (Wertime et al. 2007). IFRI is a multidisciplinary program of collaborative research that has collected data from over 500 forests and 800 forest user groups across fifteen countries. The database includes information about forests, the people that use them, and the rules that govern their interactions with the forest and each other. The IFRI database is one of the few large-n datasets available to researchers on the commons to develop and test hypotheses concerning the role of communities and informal institutions for the governance of common-pool resources. Although the IFRI database suffers from a number of well-known limitations related to sampling and measurement of some concepts, it remains one of the largest available datasets that can be used to advance understanding of collective action in natural resource management and we believe that the use of legacy cases (i.e., prior to 2011) offers an important opportunity to apply exploratory methods to develop hypotheses, and avoid “burning” through newer cases and datasets that may undermine their utility for hypothesis testing.

The sample includes 176 unique observations of forest commons from six countries (Guatemala, Bolivia, India, Kenya, Madagascar, Nepal and Uganda) between 1993 and 2011. The sample was selected by first dropping observations of privately-owned forests and those with missing values for any of the variables included in the analysis. Next, in cases where forests had repeated observations over time, the most recent observation was retained for analysis. Finally, countries with fewer than five remaining observations were excluded. Further details on the variables used in this analysis can be found in Table 1.

This analysis includes two dichotomous dependent variables that record details about the source of rules for the use of forest products and forest conditions, respectively. Local rulemaking records whether all rules for the use of forest resources were developed by current or former members of local user groups (a value of 1), or whether rules for at least some resources were developed by external actors (a value of 0). Success, meanwhile, is based upon two subjective measures of the vegetation density and species diversity of the forest relative to other forests in the same ecological zone. A forest is defined as successful (a value of 1) if both vegetation density and species diversity are assessed as typical or better when compared to forests in the same ecological zone.

The explanatory variables were selected based upon theory and research on the drivers of collective action in natural resource management (Baland and Platteau 1999, Agrawal 2003, Poteete and Ostrom 2004, Epstein et al. 2021), and filtered by the availability of corresponding variables and data in the IFRI database. It is important to note that two different samples are used in this analysis to develop insights about the factors influencing local rulemaking and local rulemaking success, respectively. The local rulemaking model includes the full sample of 176 observations as summarized in Table 1. The successful local rulemaking model, meanwhile, analyzes a subsample of this dataset which includes only those observations in which local rulemaking is equal to one, resulting in a sample of 112 observations. Summary statistics for variables included in the successful local rulemaking subsample can be found in Table S1, while Table S2 compares means between the full sample and local rulemaking sample and Table S3 reports country-level means for outcomes (i.e., local rulemaking and successful local rulemaking).

Results

The results of this analysis are summarized in the following sections, providing details about the relative influence of variables and partial dependence plots for the top three most influential variables for local rulemaking (3.1) and successful local rulemaking (3.2), respectively.

Local rulemaking

The results of the local rulemaking model, which correctly classifies approximately 79% of the observations in the sample (Table S4), with a cross-validated accuracy of 71%, are summarized in Figure 1. As can be seen, three features play a more influential role in determining the likelihood that forest user groups have developed rules for the use of forest resources: group size, forest size and Kenya. In general, it appears that forest user groups are more likely to develop and maintain rules when those groups and the forests they use are smaller in size (i.e., <100 households and 250 ha); while user groups located in Kenya are less likely to have developed rules than groups in other countries. Further details about the relationships between the remaining features (with non-zero relative influence) are supplied in Figure S1.

Figure 1

Successful Local Rulemaking

The successful local rulemaking model, meanwhile, correctly classifies approximately 81% of observations in the sample (Table S5) with a cross-validated accuracy of 75%. The results of this model are summarized in Figure 2, showing that the likelihood of successful local rulemaking is heavily influenced by the size of the forests that are managed by local forest user groups and to a lesser extent, their level of subsistence dependence on the forest and the size of associated forest user groups. In general, it appears that groups that have developed and maintained rules for the use of forest resources are more likely to be successful in terms of maintaining a forest with higher levels of vegetation density and species diversity when forests are larger than 200 ha, and forest user groups have high levels of subsistence dependence and include more than 100 households. Further details about the relationships between the remaining features (with non-zero relative influence) are supplied in Figure S2.

Figure 2

Discussion

CBNRM continues to hold considerable promise as a strategy for protecting forest resources and supporting a transition toward a more sustainable, equitable and just relationship between people and the environment (Erbaugh et al. 2020). Indeed, there is a large and growing body of evidence linking community engagement and de facto management rights to better forest conditions (Chattre and Agrawal 2009; Persha et al. 2011; Blackman et al. 2017; Hajjar et al. 2021). At the same time, policies designed to support community forest management have often failed to adequately empower communities with rights to use and manage local forest resources to deliver upon their potential (Ribot and Agrawal 2006; Sapkota et al. 2020; Hajjar et al. 2021).

The results of this analysis, for instance, show that the contexts in which groups are able to successfully self-organize to develop and maintain rules for the use of forest products are systematically different from those in which local rulemaking is likely to contribute to successful ecological outcomes. More specifically, prospects for local rulemaking appear to be highest when user groups and the forests they use are smaller in size (<250 hectares; <100 households); while prospects for successful local rulemaking rise when they are larger in size (> 200 hectares; > 100 households). Whereas small group size and forests may ease the costs and complexity of developing rules (Olson 1965), larger groups with high levels of subsistence dependence may face lower per capita costs for governance activities such as enforcement and have greater incentives and lower discount rates to ensure the long-term sustainability of resource flows (Ostrom 2000; Epstein et al. 2021).

In addition to the core findings of this research, partial dependence plots (Figure S1 and S2) suggest several potentially useful directions for future research. These include suggestions that groups are more likely to successfully self-organize to develop rules for the use of forest resources when literacy and land ownership rates are higher, and settlements are located in close proximity to markets for forest products. They also provide some evidence to suggest that local rulemaking might be more likely to contribute to better forest conditions when settlements are located further from markets and user groups include at least some individuals with a college education. Collectively these and other insights from the two boosted decision tree models have potentially valuable insights concerning the fit between local rulemaking and complex social-ecological contexts and may inform further empirical research. However, before considering these insights, it is important to note the limitations of this study.

Boosted decision trees and related approaches in machine learning offer a powerful approach for sifting through large numbers of potentially influential variables to characterize their relative importance and develop insights about the nature of their relationships with outcomes. However, boosted decisions and the IFRI dataset we used may have some influence on the results. First, the IFRI database is not a random sample of forest commons and thus important questions remain concerning the population of cases to which the results of this analysis might apply. In general, similar research has suggested that findings are more likely to apply to cases that fall within the range of independent variables in the sample (Chhatre and Agrawal 2009, Coleman 2009). Although this assumption is generally accepted by many audiences (King et al. 2001), future research on the relationship between contexts, institutions and their performance would surely benefit from a more structured approach to sampling. Second, despite including over twenty predictors in each model they do not exhaust all factors that have been proposed to influence collective action and sustainable environmental governance (Ostrom 2009). Cultural heterogeneity, for example, which has been posited to affect collective action by influencing transaction costs (Fearon and Laitin 1996), was not included due to differences in how the cultural and religious backgrounds of group members were recorded across cases.

Third, decision trees are known to be biased towards the selection of continuous variables (Strobl et al. 2009), potentially resulting in higher estimates of the relative influence of variables such as forest size and group size. Correlation among variables (Figure S3 and S4), meanwhile, can have the opposite effect on the relative influence of variables, potentially contributing to lower scores for variables such as economic heterogeneity and poverty. Fourth and finally, although the models provide insights about the contexts in which local rulemaking is likely to be found and contribute to successful ecological outcomes, it does not provide any evidence to suggest that local rulemaking might be preferred to alternative institutional arrangements in these contexts. In fact, it seems plausible that most institutional arrangements would achieve better ecological performance in larger forests given well known ecological relationships between species diversity and area, edge effects and practical constraints on harvesting rates (Bowker et al. 2017). It is worth noting, however, that one recent study found that protected area deforestation rates in protected areas (many of which are managed by state agencies) tended to be higher when protected areas were larger in size (Wolf et al. 2021). Nevertheless, the findings of this research suggest several potentially fruitful directions for future research.

First, given the limitations discussed above there is a clear need for further empirical testing of the findings. More specifically, although cross-validation provides an estimate of model performance against withheld observations, it does not provide an independent test to validate model predictions. Instead, new data developed from case studies or other large-n datasets are needed to validate the hypotheses that local rulemaking is more likely to be found in contexts where user groups and forests are smaller in size and more likely to contribute to successful ecological outcomes when forests and user groups are larger in size and include a larger fraction of users deriving subsistence benefits from forests. If these and other hypotheses derived from this study are confirmed, they may provide insights to inform the development of policies and programs to support community-based forest management tailored to different communities and ecological contexts. For example, external actors might consider investing additional time and resources in building capacity and supporting local self-organization when communities and the forests they use are larger in size.

Second, the unfortunate mismatch between the contexts in which local rulemaking is likely to be found and those in which it is likely to be ecologically successful has several implications for theory and future research. On the one hand, this finding connects with a longstanding theoretical debate in the literature on community-based natural resource management questioning whether the conditions that explain institutional development are the same as those that explain the robustness of those institutions and the environmental outcomes they produce (Singleton and Taylor 1992; Morrison 2017). While the literature has tended to assume that the factors contributing to the establishment of CBNRM and its robustness are broadly similar, others have questioned the validity of this assumption (Agrawal 2003, Basurto 2013). Our findings present additional empirical evidence in favour of the latter hypothesis. On the other hand, differences in the conditions that support local rulemaking and those in which it is likely to be successful highlight a clear need for research examining the barriers that groups face with respect to self-organization and how they vary with changes in the size of communities and the forests they use. Indeed, while many countries have taken steps to create space for communities in natural resource management through decentralization, co-management and other participatory models of environmental governance (Njaya 2007, Agrawal et al. 2008), the results have been decidedly mixed (Blaikie 2006, Ribot et al. 2006; Sapkota et al. 2020; Hajjar et al. 2021).

In some cases, barriers to collective action are embedded within the government policies that enable community participation themselves. These can include placing limits on where, when and how communities can manage resources and/or establishing onerous conditions for the transfer of powers (Ribot et al. 2006, Larson and Soto 2008). Central governments in many countries including Nepal and Uganda have, for instance, retained significant control over forest resources by establishing spatial limitations on where local communities and governments are permitted to manage their forests. This allows governments to transfer powers and avoid costs of managing smaller or less valuable forests, while retaining control over the use and management of larger, or more valuable forests. Central governments can also exercise de facto control by directly restricting the scope of local rulemaking, or alternatively by requiring them to seek approval for their management plans (Ribot et al. 2006, Epstein et al. 2020). Although these and other external factors may create barriers for local rulemaking, internal factors may also influence prospects for collective action. Transaction costs may, for instance, be higher in larger groups (Olson 1965), exacerbating challenges associated with developing and securing agreements about rules.

Third and finally, this research highlights an important opportunity to advance understanding of the relationship between complex social-ecological contexts, institutions, and outcomes by leveraging a growing portfolio of tools to explore and uncover patterns in social-ecological datasets. Methodological limitations of conventional statistical techniques have played an important role in limiting progress. Boosted decision trees and many other machine learning techniques are not subject to the same limitations (Elith et al. 2008), and can therefore be used to gain traction on increasingly important questions about tailoring institutions to address diverse and mounting sustainability challenges. However, computational tools are not a panacea for advancing theory and improving environmental policies, but there are steps that can be taken to enhance their contributions. These include further efforts to develop large-n datasets that allow researchers to examine the distribution and performance of institutions across a gradient of social and ecological contexts, tools that enable consistent collection and coding of data (Lam 1998, Poteete et al. 2010, Gurney et al. 2019, Cox et al. 2021) and a more structured approach to sampling such as a census of well-defined populations (Rana and Miller 2018; Brewer et al. 2022).

As environmental problems continue to mount in a wide range of different social, ecological and institutional contexts, the need for a diverse portfolio of potential strategies for addressing them and knowledge concerning the contexts in which these strategies are more (or less) likely to give rise to sustainable outcomes has never been greater. Collectively the insights presented here are a useful starting point for future research on the relationship between contexts, institutions, and their performance, particularly with respect to the contexts that enable or constrain local rulemaking and contribute to its success.

Additional File

The additional file for this article can be found as follows:

Funding Information

This work was supported by the National Socio-Environmental Synthesis Center (SESYNC) through funding received from the National Science Foundation (DBI-1639145). GE would to acknowledge support from the Canada First Research Excellence Fund under the Global Water Futures Programme.

Competing Interests

The authors have no competing interests to declare.