Figure 1
Timeline of research activities.
Figure 2
Anonymized screenshot of later tagging in pressbooks and hypothes.is.
Table 1
Thematic analysis of teacher and learner comments.
| POSITIVE OR NEGATIVE PERCEPTION | OF WHICH | SUM | MID-Q SURVEY | FINAL SURVEY | FG1 | FG2 | FG3 | FG4 (TEACHERS) | ||
|---|---|---|---|---|---|---|---|---|---|---|
| Positive – all assignment designs | 112 | 0 | 39 | 0 | 6 | 8 | 18 | 5 | ||
| Positive general re: OER-enabled pedagogy | 8 | 1 | 1 | 1 | 3 | 2 | ||||
| Positive – tagging increased: | 88 | 38 | 1 | 2 | 9 | 2 | ||||
| careful reading | 28 | 21 | 1 | 2 | 4 | |||||
| critical thinking | 21 | 19 | 1 | 1 | ||||||
| focus | 18 | 16 | 1 | 1 | ||||||
| memory | 5 | 4 | 1 | |||||||
| reading | 7 | 5 | 1 | 1 | ||||||
| understanding of lecture | 6 | 6 | ||||||||
| (was simple) | 3 | 2 | 1 | |||||||
| Positive – peer review | 13 | 1 | 1 | 4 | 6 | 1 | ||||
| Positive – working group | 8 | 2 | 2 | 3 | 1 | |||||
| Positive – final exercise | 3 | 2 | 1 | |||||||
| Negative – all assignment designs | 22 | 0 | 13 | 0 | 1 | 4 | 4 | 0 | ||
| Negative – tagging | 18 | 13 | 1 | 4 | ||||||
| General | ||||||||||
| tagging resulted in disconnecting paragraphs from overall text | 4 | 2 | 2 | |||||||
| tagging distracted from reading for content | 12 | 10 | 1 | 1 | ||||||
| Negative – peer review | ||||||||||
| Negative – working group | 1 | 1 | ||||||||
| Negative – final exercise | 3 | 3 | ||||||||
| Neutral (or positive to teachers, negative to students) | 14 | |||||||||
| Tagging resulted in students re-reading | 11 | 5 | 2 | 2 | 2 | |||||
| Tagging was time consuming | 3 | 2 | 1 | |||||||
| Design Guidance – Structure within assignments | ||||||||||
| Integrate individual skills | 6 | 1 | 4 | 1 | ||||||
| Compare only after contributing | 7 | 1 | 4 | 2 | ||||||
| Design Guidance – Structure across assignments | ||||||||||
| Create opportunities to dive deeper | 19 | 5 | 3 | 6 | 3 | 2 | ||||
| by creating glossary | 2 | 1 | 1 | |||||||
| Design for depth and breadth | 8 | 2 | 2 | 1 | 1 | 2 | ||||
| Technology guidance | ||||||||||
| Allow tags+comments | 14 | 6 | 1 | 4 | 2 | 1 | ||||
| Ease of use | 12 | 0 | 8 | 4 |
Table 2
Descriptive statistics of Likert-style questions in mid-quarter survey.
| DESCRIPTIVE STATISTICS | |||||
|---|---|---|---|---|---|
| N | MINIMUM | MAXIMUM | MEAN | STD. DEVIATION | |
| Tagging is a useful exercise | 22 | 2 | 6 | 4.55 | 1.011 |
| I’d like to see other student’s tags while I’m tagging | 22 | 1 | 5 | 2.82 | 1.097 |
| …like to see other students’ tags after I’m done tagging | 22 | 1 | 6 | 4.18 | 1.708 |
| …like access to others’ tags for final project | 22 | 2 | 6 | 5.27 | 1.120 |
| Valid N (listwise) | 22 | ||||
Table 3
Counts on scale of Likert-style questions in the mid-quarter survey.
| COUNTS ON LIKERT SCALE | ||||||
|---|---|---|---|---|---|---|
| “STRONGLY AGREE” | “STRONGLY DISAGREE” | |||||
| 6 | 5 | 4 | 3 | 2 | 1 | |
| Tagging is a useful exercise | 4 | 7 | 9 | 1 | 1 | 0 |
| I’d like to see other student’s tags while I’m tagging | 0 | 1 | 5 | 8 | 5 | 3 |
| …like to see other students’ tags after I’m done tagging | 9 | 0 | 4 | 5 | 3 | 1 |
| …like access to others’ tags for final project | 14 | 2 | 5 | 0 | 1 | 0 |
Table 4
Student response counts to number of tags students would prefer to have when tagging a chapter and using others’ tags to revise a chapter.
| # OF TAGS STUDENTS WOULD PREFER TO HAVE IF THEY WERE… | |||
|---|---|---|---|
| 3–5 | 5–7 | 7–9 | |
| Tagging a chapter | 3 | 6 | 3 |
| Revising a chapter with someone else’s tags | 3 | 8 | 1 |
Table 5
Student perceptions of learning with tags, as compared to without.
| READING WITH TAGS, I LEARNED: | ||
|---|---|---|
| A LOT MORE THAN WITHOUT | A LITTLE MORE THAN WITHOUT | ABOUT THE SAME |
| 5 | 2 | 5 |
Table 6
Student responses to the usefulness of new tags in learning.
| OVERALL, THE ADDITION OF NEW TAGS: | ||
|---|---|---|
| HELPED ME LEARN | MADE NO DIFFERENCE | MADE IT HARDER TO LEARN |
| 9 | 2 | 1 |
Table 7
Counts for Likert-style question about how helpful other students’ peer review and tagging contributions were when revising the final project.
| LIKERT-STYLE COUNTS OF A QUESTION ON THE HELPFULNESS OF OTHER STUDENTS’ ANNOTATION AND PEER REVIEW RESPONSES | |||||
|---|---|---|---|---|---|
| “EXTREMELY HELPFUL” | “Not Helpful at All” | ||||
| Count | 6 | 4 | 1 | 0 | 1 |
Table 8
Student-Driven Guidance for Renewable Assignments Within an OER-Enabled Course.
| WITHIN-ASSIGNMENT GUIDANCE | |
|---|---|
| THEME | LEARNER QUOTE(S) |
| 1. Create Opportunities to Dive Deeper | |
| Create opportunities for an excellent contribution to a smaller part of the open resource, and to learn a part of the subject in depth. Have multiple assignments contribute to the same part of text or have several steps in an assignment iteratively improve an open resource. | “Overall, the most helpful thing for learning in the course was the working group assignments. (Another student nodding) One for each chapter, had to collaborate and discuss. With WG assignments, assigned a working group on each chapter. Know one thing in that chapter very well. (In-depth contributions) Could work well.” |
| 2. Design for Depth and Breadth | |
| Use a mix of assignments as opportunities to contribute to open knowledge about both specific topics and the scope and structure of the field – empowering students to learn about the field’s breadth. “Broad” co-constructions can include glossaries, chapter listings, summaries, and more. | “I did enjoy these assignments and feel that the concept of an open text like this is wonderful for many reasons, but I do think it’s worth noting that I definitely learned more about the sections my group was assigned than other topics presented in class.” “Make a glossary hyperlinking words. Could we make a glossary when doing edits? Would be really helpful for an online textbook, to have that.” |
| ACROSS-ASSIGNMENT GUIDANCE | |
| 3. Integrate Student Skills | |
| Students appreciate opportunities to use the skills they bring to the class to contribute to the open resources through role-based groups. | “(My) group had an english minor, had a lot of grammatical changes. Mostly biology students- writing isn’t the biggest skill. A lot of edits were grammatical. Could be used with role-based peer review.” |
| 4. Compare only after Contributing | |
| Learners and educational theory agree that opportunities for learners to compare their contributions with others come best after, not during or before, they contribute. For example, showing other students’ tags only after students completed their own tagging. | In a survey question, no students strongly agreed with wanting to see each other’s tags while tagging. 16 of 22 disagreed to strongly disagreed. 9 strongly agreed with seeing tags after they were done tagging, 14 strongly agreed that other’s tags are useful when they edit a section. |
| TECHNOLOGY GUIDANCE | |
| 5. Enable Layered Contributions | |
| Most learning technologies focus on one-time viewings, contributions, and submissions. Learning technologies in general, and renewable learning technologies in particular, will benefit from enabling students to return, compare, revise, and recontribute. | “It can be helpful to have already thought through the good areas and potential problem areas of the reading before group work so I already have some ideas of what could be improved upon (or can look to the well-done areas for inspiration of how to make things better).” |
| 6. Maximize Ease of Use | |
| Despite the priority given ease of use and design in technology construction, learners still feel systems fall short – and appreciate technologies that are very easy and straightforward to use. | “I found both platforms to be easy to use, and don’t have any major grievances with either…. google would occasionally lag behind, and we couldn’t see each other comments that were made on separate computers for a few minutes…. Pressbooks had a different issue where some people’s comments were hiding/reappearing…” |