UT Snapshot: The Business of Peer Review

MIS372 (Business Systems Development) 

1. I’ve wanted to ask the University to change the descriptions of all WB classes or give them some instruction on whether they are the best person to take the online format. This has been challenging to get this done so I have canned language I send to students at the beginning that explain the important of time management and that no exceptions are made.  This influences some to take the traditional class or to hopefully better manage their time in the online class.  I also send a message to students on the last day of Week 1 deadlines that have not bought materials or started the work to remind them that no exceptions are made.  This way if they do miss deadlines, they know we were watching and are not willing to hear excuses.  I use the Canvas Gradebook message feature to do this.
2. I use peer evaluations to grade student’s papers. We can have 700 papers each graded by 3-4 anonymous students using a standard rubric.  We then average the grades together to get a final grade and allow students to appeal if they think the grade has an error.
3. We have implemented a comprehensive final exam that is online and administered using ProctorU software. This forces the actual student to have to prove their identify and take the final exam.

1. Students that are not good at time management do poorly because they wait until the last 1-2 days before deadlines to do work. Many deadlines are missed which can be frustrating to manage if you choose to do so.
2. The more self-paced or auto-graded a course is, the less rich the feedback is for the student. In large online classes that have students writing more subjective papers and doing less auto-graded quizzes, require more work on the teaching team to manage. 
3. You can’t 100% guarantee that some students are doing all the work. 

1. I tried this first with one objective assignment and then experimented with subjective assignments. After I saw that only 5% of students appeal their grades and also that grades were accurate, we implemented this form of grading on all non-auto-graded assignments.
2. I also met with other professors who use this tech to learn about it.
3. Time to create the rubric. It took many hours to devise a way to download peer scores into Excel using the Canvas API but after many trail runs it’s now about a 20-30 minute effort to consolidate and calc grades for an assignment.
4. 1-2 hours to create the exam based on existing test banks. The effort to create the test bank was substantial and done many semesters ago.

1. It’s hard to gauge the student response, but it appears that being clear about time management early on helps reduce missed deadlines in the first week.
2. The students don’t like the idea when they hear about it but I get no complaints about the peer grading method at the end of a course.
3. I'm getting less emails asking for more time and exceptions due to procrastination
4. It has greatly reduced grading time for assignments. Used to ~6 hours and now only takes 30 minutes.

1. I really love to use the “Message Students Who…” option in Gradebook on Canvas. It’s a great way to intervene with students based on their progression through the course and get them moving.
2. If you’re good with Excel I can show you how to do the calculation of peer grades using the Canvas API but we’re also beta testing an automated too that will calculate the grades for you.