Living labs: a systematic review of success parameters and outcomes

Jan Moritz Müller

doi:10.5334/bc.624

Figures & Tables

Table 1

Inclusion and exclusion criteria of living labs (LLs).

INCLUSION CRITERIA	EXCLUSION CRITERIA
Discussion of LLs Engagement with LL evaluation, impact assessment, performance measurement, value creation or success factors Peer-reviewed journal articles, conference papers, book chapters and academic reports Conceptual, empirical or review-based contributions analysing LL activities, outputs or outcomes	Lack of evaluation or impact dimension Technical reports focusing on infrastructure or organisation Non-academic documents Inaccessible full texts Studies unrelated to LL outcomes or LL governance models

Living labs’ (LLs) success parameters and outcomes: selection of publications.

Table 2

Definitions of the key analytical concepts of living labs (LLs).

PERFORMANCE	EFFICACY	IMPACT	SUCCESS
Performance denotes the degree to which LL processes, methods or practices function effectively to achieve goals and deliver expected outputs (Neely et al.2005)	Efficacy refers to the capacity of an LL to generate its intended effects under real-world conditions, distinct from mere outputs or activity counts (Rossi et al.2004)	Impact refers to the extent to which observed changes can be causally attributed to LL activities, including medium- and long-term outcomes (White 2009)	Success parameters are measurable criteria used to assess whether a given initiative has achieved its intended outcomes, e.g. metrics or benchmarks against which success is judged (Shenhar et al.2001)

Table 3

Coding process.

STAGE 1: OPEN CODING	STAGE 2: CATEGORY DEVELOPMENT	STAGE 3: CONSOLIDATION
Relevant segments were identified based on their connection to LL evaluation, success parameters, performance indicators or impact mechanisms	Recurring patterns were grouped into broader analytical categories. Initial codes were refined, merged or differentiated through constant comparison	The final coding framework consisted of three major categories: LL activities and methodologies Output assessment dimensions Value creation and assessment Those categories were supported by a set of subcodes

[i] Note: LL = living lab.

Table 4

Overview of living lab (LL) types.

LL TYPE	CHARACTERISTICS	OUTCOMES	REFERENCES
Provider-driven LLs	Typically initiated by academia or research Theoretical, knowledge-focused approach	New knowledge supporting operations development	Berniak-Woźny & Szelągowski (2023); Nyborg et al. (2024); Rogers et al. (2023); Satria et al. (2023)
Utiliser-driven LLs	Established by the private sector Testing, iterating and business-focused approach	Insights into product and business development	Cigir (2018); Lin et al. (2013)
User-driven LLs	Grassroots-led initiatives Community-led approach to everyday issues in a certain environment	Solutions for users’ everyday problems	Chen et al. (2010); Mulder & Stappers (2009); Pascu & van Lieshout (2009)
Enabler-driven LLs	Public-sector or non-governmental organisation initiated Long-term societal, environmental or policy focus	Guided strategy change in a preferred direction	Galbraith et al. (2008); Grüneis et al. (2020)
Network-driven LLs	Stronger integration of several interest groups of equal standing Distribution of leadership across stakeholders, fostering shared decision-making and accountability	Radical innovation rather than incremental	Del Vecchio et al. (2017); Nguyen et al. (2021); Merino-Barbancho et al. (2023)

Table 5

Evaluation framework type overview.

EVALUATION FRAMEWORK TYPE	EVALUATION FRAMEWORK VALUE	REFERENCES
Outcome-oriented frameworks	Focus on assessing immediate, tangible results, such as innovations, prototypes, user satisfaction and learning enhancements	Ballon et al. (2018); Emblen-Perry (2019); Leminen et al. (2016); Santally et al. (2014); Ståhlbröst & Holst (2017); Van Geenhuizen (2018, 2019); Veeckman et al. (2013)
Impact-oriented frameworks	Assessing broader, longer term effects and societal contributions beyond just the immediate outputs	Bouwma et al. (2022); Bronson et al. (2021); Ceseracciu et al. (2023); Ciliberti et al. (2022); Grüneis et al. (2020); Paranunzio et al. (2023)
Effectiveness-oriented frameworks	Providing insights into operational mechanics of living labs (LLs) and evaluate whether the intended goals are achieved and LLs deliver on their claims of promoting innovation	Banerjee (2022); Berniak-Woźny & Szelągowski (2023); Guzmán et al. (2013); Huang et al. (2024); Kalinauskaite et al. (2021); Logghe & Schuurman (2017)
Hybrid frameworks	Attempt to bridge these gaps but highlight persistent challenges related to methodological integration and standardisation	Bronson et al. (2021); Paskaleva & Cooper (2021); Rosa et al. (2024); Schafer et al. (2024); Toffolini et al. (2021)

Table 6

Overview of the identified living lab (LL) success parameters.

ANALYTICAL DIMENSION	SUCCESS PARAMETERS	REFERENCES
Economic and business	Increased investments and follow-up private funding New job creation and employment impact Increased revenue and sales Successful market introduction and launch of new products/services	Ballon et al. (2018); Banerjee (2022); Fuglsang et al. (2021); Guzmán et al. (2013); Jernsand (2019)
User centricity	Improved alignment of innovations with user needs and preferences Increased user satisfaction and perceived value of specific innovation Adoption and usage of the co-created solutions by end-users	Bronson et al. (2021); Dell’Era & Landoni (2014); Eriksson et al. (2005); Guzmán et al. (2013); Konstantinidis et al. (2021); Mastelic et al. (2015); Ståhlbröst (2012); Ståhlbröst & Holst (2017); Svensson et al. (2010)
Innovation	Degree of novelty and radicalness of innovation Number of iterations and improvements made to the innovation based on user feedback Reduced risks and increased success rate of the innovation process	Ballon et al. (2018); Greve et al. (2021); Kemeç (2023); Leminen et al. (2016, 2023); Leminen & Westerlund (2012); Schuurman et al. (2016); Ståhlbröst (2012); Veeckman et al. (2013); Yilmaz & Ertekin (2023)
Knowledge and learning	Increased knowledge and competence of participating organisations Extent of cross-fertilisation and exchange of knowledge between stakeholders	Archibald et al. (2021); Berniak-Woźny & Szelągowski (2023); Ceseracciu et al. (2023); Eriksson et al. (2005); Mastelic et al. (2015); Nyström et al. (2014); Purcell et al. (2019); Smith et al. (2022); Ståhlbröst (2012)

Table 7

Initial overview of living lab (LL) types, associated evaluation frameworks and success parameters serving as a foundation for future research and practical validation.

TYPE OF LIVING LAB (LL)	MOST RELEVANT EVALUATION FRAMEWORK	MOST RELEVANT SUCCESS PARAMETER	CONCEPTUAL JUSTIFICATION
Provider-driven LL	Effectiveness-oriented Outcome-oriented	Knowledge and learning Innovation	Initiated by academia or research institutes, provider-driven LLs prioritise structured experimentation, method testing and pedagogical outcomes. Thus, evaluating operational effectiveness (e.g. knowledge flows, process quality) and short-term outcomes (e.g. prototypes, learning) is most suitable.
Utiliser-driven LL	Outcome-oriented	Economic and business value Innovation	Driven by private firms aiming for product or service optimisation, utiliser-driven LLs benefit from outcome frameworks that track commercial results and iterative innovation cycles. The primary concern is value realisation, making business-focused and innovation-related outputs the main evaluation interest
User-driven LL	Impact-oriented Hybrid	User-centricity Society and community	Rooted in grassroots or civic initiatives, user-driven LLs focus on needs-based innovation, empowerment and community impact. Long-term social change and democratic innovation demand impact frameworks, but hybrid models can help trace the bottom-up engagement process and its translation into outcomes
Enabler-driven LL	Impact-oriented Hybrid	Society and community Impact (long-term) User-centricity	Enabler-driven LLs pursue societal transformation, regional strategies or inclusive policymaking. Impact-oriented frameworks capture these ambitions, and hybrid models help combine short-term responsiveness with long-term systemic change, including user integration
Network-driven LL	Hybrid Effectiveness-oriented	Innovation Knowledge and learning User-centricity Society and community	Network-driven LLs involve multi-stakeholder co-ownership, promoting ecosystemic value creation, radical innovation and continuous learning. Hybrid frameworks are best suited to capture multidimensional impacts, while effectiveness-oriented elements help evaluate governance and coordination