Table 1
The large pool of statements for ontology usability evaluation.
| 1. I found the various concepts in this ontology well integrated |
| 2. The ontology misses some important concepts – Changed to “The ontology has all the important concepts included” in the survey |
| 3. The ontology has unnecessary concepts – Changed to “The ontology does not have unnecessary concepts” |
| 4. I found the ontology unnecessarily complex – Changed to “I found the ontology brief but comprehensive” |
| 5. I thought there was too much inconsistency in this ontology – Changed to “I found the various parts of this ontology well integrated” |
| 6. I found the formal specification of concepts in this ontology coincides with their descriptions in natural language |
| 7. I found the formal specification of relations in this ontology coincides with their descriptions in natural language |
| 8. I think the attributes in this ontology describe the concepts well |
| 9. I think the relations in this ontology relate appropriate concepts |
| 10. I found the subclasses in this ontology are properly defined |
| 11. I found the disjoint classes in this ontology are properly asserted |
| 12. The purpose of this ontology is clear |
| 13. I am confident I understand the conceptualization of the ontology |
| 14. I need to ask a lot of questions before I could understand the conceptualization of the ontology – Changed to “I could understand the conceptualization of the ontology without asking a lot of questions” |
| 15. I find the ontology easy to understand |
| 16. The annotations are helpful |
| 17. I found the concepts in this ontology properly described in natural language |
| 18. I found the relations in this ontology properly described in natural language |
| 19. I need further theoretical support to understand this ontology – Changed to “I do not need further theoretical support to be able to understand this ontology” |
| 20. I would imagine that most domain experts would understand this ontology very quickly |
| 21. I would imagine that most people would learn to use this ontology very quickly |
| 22. I needed to learn a lot of things before I could get going with this ontology – Changed to “I do not need to learn any extra things before I could get going with this ontology” |
| 23. I thought the ontology was easy to use |
| 24. It is clear to me how to use this ontology |
| 25. I felt very confident using the ontology |
| 26. I found the ontology very cumbersome to use – Deleted in the survey since it is the negative form of statement 23 and we decided to present all the statements in their positive forms only. |
| 27. I think that I would need the support of a person experienced with this ontology to be able to use it – Changed to “I do not need the support of a person experienced with this ontology to be able to use it” |
| 28. I need some more examples than provided in the documentation to make sure how to use the ontology – Changed to “I think the documentation provides sufficient examples for me to make sure how to use the ontology” |
| 29. I think that I would like to use this ontology frequently |
| 30. I think that I could contribute to this ontology |
Table 2
The top 11 statements resulted from an online poll and the votes to each statement received.
| Number | Statement | Category | Votes |
|---|---|---|---|
| 1 | I think the documentation provides sufficient examples for me to make sure how to use the ontology. | Pragmatics | 14 |
| 2 | The purpose of this ontology is clear. | Semantics | 9 |
| 3 | I found the concepts in this ontology properly described in natural language. | Semantics | 9 |
| 4 | I think the relations in this ontology relate appropriate concepts. | Syntax | 8 |
| 5 | I am confident I understand the conceptualization of the ontology. | Semantics | 8 |
| 6 | I would imagine that most domain experts would understand this ontology very quickly. | Semantics | 7 |
| 7 | I think the attributes in this ontology describe the concepts well. | Syntax | 7 |
| 8 | I found the subclasses in this ontology are properly defined. | Syntax | 7 |
| 9 | I found the relations in this ontology properly described in natural language. | Semantics | 7 |
| 10 | I found the formal specification of concepts in this ontology coincides with their descriptions in natural language. | Syntax | 7 |
| 11 | I do not need the support of a person experienced with this ontology to be able to use it. | Pragmatics | 7 |
Table 3
Ontology usability evaluation questionnaire (draft).
| Number | Statement | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
| 1 | I think the documentation provides sufficient examples for me to make sure how to use the ontology. | |||||
| 2 | The purpose of this ontology is clear. | |||||
| 3 | I found the concepts and relations in this ontology properly described in natural language. | |||||
| 4 | I think the relations in this ontology relate appropriate concepts. | |||||
| 5 | I am confident I understand the conceptualization of the ontology. | |||||
| 6 | I would imagine that most domain experts would understand this ontology very quickly. | |||||
| 7 | I think the attributes in this ontology describe the concepts well. | |||||
| 8 | I found the subclasses in this ontology are properly defined. | |||||
| 9 | I found the formal specification of concepts and relations in this ontology coincides with their descriptions in natural language. | |||||
| 10 | I do not need the support of a person experienced with this ontology to be able to use it. |
Table 4
Recommended ontology usability scale.
| Number | Statement | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
| 1 | The purpose of this ontology is clear. | |||||
| 2 | I need more examples than provided in the documentation to make sure how to use the ontology. | |||||
| 3 | I found the concepts and relations in this ontology properly described in natural language. | |||||
| 4 | There is inconsistency between the formal specification of concepts and relations in this ontology and their descriptions in natural language. | |||||
| 5 | I would imagine that most domain experts would understand this ontology very quickly. | |||||
| 6 | I think that I would need the support of a person experienced with this ontology to be able to use it. | |||||
| 7 | I am confident I understand the conceptualization of the ontology. | |||||
| 8 | The attributes in this ontology fail to describe the concepts properly. | |||||
| 9 | I think the relations in this ontology relate appropriate concepts. | |||||
| 10 | I think the class hierarchy of this ontology needs better organization. |
Table 5
Results of ontology usability evaluation in a case study.
| S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | S10 | |||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Case 1 | DCO | 5 | 4 | 3 | 2 | 4 | 3 | 5 | 2 | 4 | 2 | 38 |
| Case 2 | DCO | 5 | 4 | 4 | 1 | 3 | 2 | 5 | 1 | 5 | 2 | 42 |
| Case 3 | DCO | 3 | 1 | 4 | 3 | 2 | 2 | 2 | 4 | 3 | 5 | 29 |
| Case 4 | GCIS | 4 | 2 | 4 | 2 | 4 | 3 | 4 | 2 | 4 | 3 | 38 |
| Case 5 | GCIS | 5 | 3 | 4 | 1 | 5 | 2 | 5 | 1 | 5 | 3 | 44 |
| Case 6 | SIO | 5 | 5 | 5 | 2 | 5 | 5 | 2 | 1 | 5 | 3 | 36 |
| Case 7 | VSTO | 1 | 5 | 1 | 3 | 2 | 4 | 4 | 2 | 4 | 2 | 26 |
Table 6
The least voted statements in the online poll.
| Number | Statement | Category | Votes |
|---|---|---|---|
| 1 | I do not need to learn any extra things before I could get going with this ontology. | Pragmatics | 0 |
| 2 | The ontology does not have unnecessary concepts. | Syntax | 2 |
| 3 | The annotations are helpful. | Semantics | 2 |
| 4 | I would imagine that most people would learn to use this ontology very quickly. | Pragmatics | 2 |
| 5 | I found the various parts of this ontology were well integrated. | Syntax | 2 |
Table 7
Ontologies that the online surveyees worked with.
| Ontology name | Number of people worked with it |
|---|---|
| SWEET | 11 |
| Dublin Core | 8 |
| FOAF | 7 |
| SKOS | 5 |
| PROV-O | 3 |
| GCIS | 3 |
Table 8
Top 12 statements selected by SWEET users.
| Number | Statement | Votes |
|---|---|---|
| 1 | The purpose of this ontology is clear. | 8 |
| 2 | I think the documentation provides sufficient examples for me to make sure how to use the ontology. | 8 |
| 3 | I think the relations in this ontology relate appropriate concepts. | 7 |
| 4 | I found the concepts in this ontology properly described in natural language. | 6 |
| 5 | I am confident I understand the conceptualization of the ontology. | 6 |
| 6 | I think the attributes in this ontology describe the concepts well. | 5 |
| 7 | I found the subclasses in this ontology are properly defined. | 5 |
| 8 | I do not need the support of a person experienced with this ontology to be able to use it. | 5 |
| 9 | I found the relations in this ontology properly described in natural language. | 4 |
| 10 | I found the ontology brief but comprehensive. | 4 |
| 11 | I found the formal specification of concepts in this ontology coincides with their descriptions in natural language. | 4 |
| 12 | I find the ontology easy to understand. | 4 |