Ontology Usability Scale: Context-aware Metrics for the Effectiveness, Efficiency and Satisfaction of Ontology Uses

Xiaogang Ma; Linyun Fu; Patrick West; Peter Fox

doi:10.5334/dsj-2018-010

Abstract

Both ontology builders and users need a way to evaluate ontologies in terms of usability, but existing ontology evaluation approaches do not fit this purpose. We propose the Ontology Usability Scale (OUS), a ten-item Likert scale derived from statements prepared according to a semiotic framework and an online poll in the Semantic Web community to provide a practical way of ontology usability evaluation. Case studies were conducted to bookkeep current usability evaluation results for ontologies expecting revisions in the future, and discussions of the poll results are presented to help proper use and customization of the OUS.

References

1Bikakis, N, Tsinaraki, C, Gioldasis, N, Stavrakantonakis, I and Christodoulakis, S. 2013. The XML and semantic web worlds: technologies, interoperability and integration: a survey of the state of the art. In: Semantic Hyper/Multimedia Adaptation. Springer, pp. 319–360. DOI: 10.1007/978-3-642-28977-4_12
Back to article
2Brank, J, Grobelnik, M and Mladenic, D. 2005. A survey of ontology evaluation techniques.
Back to article
3Brickley, D and Guha, R. 2014. RDF Schema 1.1. W3C Recommendation. Available at: https://www.w3.org/TR/rdf-schema/ (Accessed on April 17, 2018).
Back to article
4Brooke, J. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189: 4–7.
Back to article
5Burton-Jones, A, Storey, V, Sugumaran, V and Ahluwalia, P. 2005. A semiotic metrics suite for assessing the quality of ontologies. Data & Knowledge Engineering, 55(1): 84–102. DOI: 10.1016/j.datak.2004.11.010
Back to article
6Casellas, N. 2009. Ontology Evaluation through Usability Measures. In: Meersman, R, Herrero, P and Dillon, T. (Eds.) On the Move to Meaningful Internet Systems: OTM 2009 Workshops, LNCS, Vol. 5872. Springer, Berlin, pp. 594–603. DOI: 10.1007/978-3-642-05290-3_73
Back to article
7Fox, M, Barbuceanu, M and Gruninger, M. 1995. An organisation ontology for enterprise modelling: preliminary concepts for linking structure and behaviour. In: Enabling Technologies: Proceedings of the Fourth Workshop on Infrastructure for Collaborative Enterprises. IEEE, pp. 71–81. DOI: 10.1109/ENABL.1995.484550
Back to article
8Fox, P and Lynnes, C. 2015. Additional items for ontology evaluation. Available at: http://tw.rpi.edu/web/project/SeSF/workinggroups/OntologyEvaluation (Accessed on Sept. 9, 2015).
Back to article
9Gangemi, A, Catenacci, C, Ciaramita, M and Lehmann, J. 2006. Qood grid: A metaontology-based framework for ontology evaluation and selection. In: Proceedings of the EON2006 Workshop.
Back to article
10Gomez-Perez, A. 2001. Evaluation of ontologies. International Journal of intelligent systems 16(3): 391–409. DOI: 10.1002/1098-111X(200103)16:3<;391::AID-INT1014>3.0.CO;2-2
Back to article
11Gruber, T. 1995. Toward principles for the design of ontologies used for knowledge sharing. International Journal of Human-Computer Studies 43(5–6): 907–928. DOI: 10.1006/ijhc.1995.1081
Back to article
12Gruninger, M and Fox, M. 1995. Methodology for the design and evaluation of ontologies. Workshop on Basic Ontological Issues in Knowledge Sharing, IJCAI-95, Montreal, Canada. 10pp.
Back to article
13Hartmann, J, Spyns, P, Giboin, A, Maynard, D, Cuel, R, Suarez-Figueroa, M and Sure, Y. 2005. D1. 2.3 methods for ontology evaluation. EU-IST Network of Excellence (NoE) IST-2004-507482 KWEB Deliverable D, 1.
Back to article
14ISO. 1998. ISO 9241-11 Ergonomic requirements for office work with visual display terminals (VDT)s – Part 11 Guidance on usability. International Organization for Standardization.
Back to article
15Jokela, T, Iivari, N, Matero, J and Karukka, M. 2003. The standard of user-centered design and the standard definition of usability. In: CLIHC ’03 Proceedings of the Latin American conference on Human-computer interaction, pp. 53–60. DOI: 10.1145/944519.944525
Back to article
16Lozano-Tello, A and Gomez-Perez, A. 2004. Ontometric: A method to choose the appropriate ontology. Journal of Database Management 2(15): 1–18. DOI: 10.4018/jdm.2004040101
Back to article
17Lozano-Tello, A, Gomez-Perez, A and Sosa, E. 2003. Selection of ontologies for the semantic web. In: Web Engineering. Springer, pp. 413–416. DOI: 10.1007/3-540-45068-8_77
Back to article
18Ma, X and Fox, P. 2013. Recent progress on geologic time ontologies and considerations for future works. Earth Science Informatics 6(1): 31–46. DOI: 10.1007/s12145-013-0110-x
Back to article
19McGuinness, D and Van Harmelen, F. 2004. Owl web ontology language overview. W3C recommendation Available at: https://www.w3.org/TR/owl-features/ (Accessed on April 17, 2018).
Back to article
20Miller, E. 1998. An introduction to the resource description framework. Bulletin of the American Society for Information Science and Technology 25(1): 15–19. DOI: 10.1002/bult.105
Back to article
21Morris, C. 1938. Foundations of the Theory of Signs. University of Chicago Press. Chicago, IL.
Back to article
22Pratchett, T. 1990. Moving Pictures. Gollancz, London.
Back to article

Ontology Usability Scale: Context-aware Metrics for the Effectiveness, Efficiency and Satisfaction of Ontology Uses

Abstract

Paradigm

My account