AI Course Guide

Contents

Flipped Classroom strategy
Readings and Videos
Tools
Moodle
Overview (from Moodle)
How We'll Use the Tool
Learning Catalytics
Overview (from Learning Catalytics)
How We'll Use the Tool
Quizzes
Project Deliverables
Peer- and Self-Assessment
Overview (from UNSW)
Calibration
Peer-Assessment
Self-Assessment
Deadlines
Concerns
Teamwork Assessment

Welcome to AI. This guide has been prepared to help you and your team get the most out of the course, and to introduce the tools we'll be using throughout the semester. It's important that you understand the pedagogical approaches we will be using, that of the flipped classroom model, and peer- and self-assessment of your work.

Flipped Classroom strategy

Our class takes a flipped classroom strategy, in which class time is often used to review the topics covered in the assigned readings, carry out exercises, and assess your learning, both individually and in groups. In short, this means that most of your learning will be self-directed, and little class time will be devoted to lecturing.

Readings and Videos

For this reason, it is important that you complete the assigned readings and optionally, view the related videos in advance of each class. If you find yourself struggling with any of the assigned materials, raise these points for discussion in class. Note that many of the classes have a set of optional readings listed. These are only intended as references for those students who wish to delve deeper into any specific topic, but are not required. You will not be assessed on this content.

Tools

In addition to the course website, we will be using two tools throughout the course: the Moodle learnware system and Learning Catalytics, an interactive class response system. As expected with any non-trivial software, neither of these systems is without its problems, but to a large extent, they work well for what they are intended to do. Unfortunately, this also means that you'll need to make regular visits to three web sites and keep track of three sets of login credentials.

Moodle

In advance of the first class, you'll be receiving an email with your Moodle registration details, with which you can log in to the tool.

Overview (from Moodle)

Moodle is an Open Source Learning Platform or course management system (CMS), similar in concept to McGill's myCourses.

How We'll Use the Tool

We'll be using Moodle for class discussions, exercise submissions, and all phases of peer- and self-assessment. Electronic communication with your instructor should be conducted through Moodle.

Learning Catalytics

In advance of the first class, make sure you've registered with Learning Catalytics with the access code you've been provided. You'll need to bring a tablet, or laptop, or in the worst case, a smartphone, to each class, so that you can participate in assessments with this tool.

Overview (from Learning Catalytics)

Learning Catalytics is an interactive student response tool that encourages team-based learning by using students' smartphones, tablets, or laptops to engage them in interactive tasks and thinking.

How We'll Use the Tool

We will be using Learning Catalytics for both informal review and formal assessment of your learning through the course. You will be notified in class of planned formal assessments--quizzes--at least one class in advance. These quizzes will be biased toward recently covered material, but will also include questions relating to content seen earlier in the course.

Quizzes

For in-class quizzes, you will first answer the questions individually. Your individual score accounts for 80% of your grade. You will then have another opportunity to answer the questions in teams for a further 20% of your grade. We hope you will take advantage of this opportunity to discuss the quiz items and learn from each other. For each question, if your team answers it correctly on the first attempt, all members of the team earn full marks. Otherwise, if your team answers the question correctly on the second attempt, your team earns half marks. Answering correctly on the third attempt earns your team quarter marks.

Assignments and Project

Most of your grade in the course comes from your individual assignments and the term project, as well as your peer- and self-assessments of these deliverables.

Peer- and Self-Assessment

All peer- and self-assessment will be conducted through our class Moodle. For many of you, the process of assessing someone else's (or your own) work is likely to be a new experience, so we offer the following pointers to make this a success.

Overview (from UNSW)

Peer assessment is the assessment of students' work by other students of equal status. Students often undertake peer assessment in conjunction with formal self-assessment. They reflect on their own efforts, and extend and enrich this reflection by exchanging feedback on their own and their peers' work.

Peer assessment is a powerful meta-cognitive tool. It engages students in the learning process and develops their capacity to reflect on and critically evaluate their own learning and skill development. It supports the development of critical thinking, interpersonal and other skills, as well as enhancing understanding within the field of knowledge of a discipline.

[Only relevant for group work:] Peer and group assessment are also often undertaken together. Typically, the members of a group assess the performance of their peers in terms of their contribution to the group's work.

Calibration

At each submission deadline, Moodle will transition to the assessment phase. You will first practice with a couple of submissions that were already assessed in previous years. After entering your own assessment of how well these submissions met the criteria of the various rubrics, you will be able to compare with the reference, which was vetted by the instructor. If you find that there were significant discrepancies between your assessment and the reference, make sure you understand where you might have gone wrong. This is very important, because 20% of your grade for each deliverable comes from your assessments. The assessment grade is split between the accuracy of your scoring, and the quality of your written feedback.

Peer-Assessment

Once you have completed the practice round, you can proceed to assess the submissions of other students that have been randomly assigned, as well as your own. Below the set of assessment rubrics, there is a textbox where you can provide some explanation and justification for your assessment, and offer critique that helps suggest improvements that the assessed student should keep in mind. This written feedback is important not only to the student being assessed, but also in the scoring of the quality of your assessment. This is an important opportunity for you to demonstrate that you've thought carefully about the work you are assessing. As such, you earn marks for thoughtful critique that is meaningful from an AI perspective. Don't forget to be positive in your critiques where appropriate! Feedback is evaluated according to the following examples.

ScoreExampleJustification
0 *no comment* or Great work Trivial feedback that does not give the assessed any explanation for the grades they received.
1 Could have done with more polish, and better following of the guidelines.
In particular, the code documentation could have been improved, and the methodology for testing could have been better explained.
Meaningful feedback is given, but not all of the criterion are justified.
3 Criterion 1:Great
Criterion 2:The explanation of an obvious appreciation of the performance benefits of alpha-beta pruning is not that thorough, but it achieves the requirements.
Criterion 3: There are no graphs but the report explains how to achieve optimal savings in theory.
Criterion 4:I can't understand how the heuristic evaluation function works exactly without something like a formula. I think he gives every position weights, but I would have liked to know what those weights are exactly.
Criterion 5: Nothing about memory.
Criterion 6: I think the program is well organized but the code should of had more comments.
Meaningful feedback is given and all of the criterion are justified.

Self-Assessment

After assessing the work of your peers, you will self-assess your own deliverable. Although there may be a temptation to inflate your grade, this will actually penalize you, because Moodle will detect the inflation and lower your assessment score. It pays to reflect honestly on your own work. Take this opportunity to identify what went well, and what that you can do better for the next deliverable.

Deadlines

The due date for each deliverable is noted in the course syllabus and the Moodle course calendar. Assessments are due one week after each project deliverable deadline. The Moodle course calendar can be exported in ics format and imported into your favourite calendar tool.

Concerns

Students may well be concerned about the peer-assessment process. Here are some concerns that were expressed in the past, and your instructor's response to each:

  1. My peers are not experts in the subject matter, so how can they assess my work correctly and offer me meaningful feedback, as would a TA or official grader?

    Perhaps surprisingly, peer-assessment has been found to work extremely well, and may offer advantages over conventional TA-grading. Although graders and TAs have typically taken the same course, or an equivalent, in the past, they are not necessarily experts in the subject either. In any case, since our department decimated the budget for TA and grader support, that option is no longer even feasible.

  2. Lazy assessments: What if the peer assessors don't bother to read our deliverables carefully? If they don't invest the effort in grading the work according to the assessment criteria, this might result in a lower average being assigned.

    It's quite likely that some assessors won't take the time to go through your work with anywhere near the level of attention that it merit. However, the same can be true for TAs, graders, academic conference, journal reviews, and job performance assessments. The hope is that with multiple assessors, the effects of outliers are minimized.

    Keep in mind that part of your grade for each assignment or project comes from your assessments, both the quantitative portion (scored by Moodle), and your written feedback (scored by human). Given multiple assessments, Moodle generally does a good job of identifying the outliers, because of the differences in scoring against the idealized assessment. Thus, while it is true that sloppy peer assessments have the potential to pull down your grade, this is more likely to penalize a lazy assessor disproportionately because of the metrics Moodle employs.

  3. Normalizing effect. Stronger students will provide higher quality feedback, which will help their peers improve their respective projects. The same is not true in reverse.

    If so, that's one of the primary benefits of the peer-assessment process, helping to maximize the learning opportunities for everyone in the class. Ideally, every student would benefit from the same high calibre feedback, but if your peers' critique of your own work does not provide helpful feedback, you are free to ignore them, so your subsequent deliverable isn't adversely affected. If you are seeking additional critique regarding your work, please let the instructor know of any specific questions on which you'd like such input.

  4. Those investing minimal effort give little to no feedback. The top projects might receive no helpful critique, and ultimately end up discouraged, not able to develop to their full potential.

    Not only are the numeric components of the peer- and self-assessments of each deliverable worth 15% of your grade, but we recently introduced the additional evaluation of your written feedback, worth 5% of your grade. Thus, those students who make a serious attempt to provide quality feedback are likely to find these efforts rewarded.

  5. It's hard work critiquing someone else's work (as well as my own)!

    Yes indeed, but developing the experience in carrying out such assessment, offering constructive criticism, and reflecting on your own submissions, are important soft skills from which you are likely to benefit significantly, regardless of whether you end up in industry or academia.

  6. For the project, my team members are not all pulling their weight. What can I do about it?

    If you find yourself in a situation where your peers don't appear to be contributing equally, and you're unable to resolve such issues internally within a few days, or at most a week, please speak with the instructor right away! In the instructor's experience, students in such situations tend to wait until the end of term to raise such issues, by which point it is far too late to address the problem.

Teamwork Assessment

When working on the term project, you should regularly assess yourself and each of your group members against these teamwork criteria:

  1. Cooperation: Worked cooperatively with other members of the group and was willing to help with any task.
  2. Respect: Listened to others' ideas, considered their points of view and offered constructive suggestions.
  3. Effort: Contributed as much as could to group discussions and to the work required.
  4. Responsibility: Worked responsibly and to the best of your/his/her ability on contributions to the task.
  5. Task commitment: Was able to focus on what the group needed to do throughout the process of the task and kept working even when something was challenging.
  6. Problem solving: Used good problem solving strategies throughout the process of completing the task.

Score the criteria above as follows:

Although such internal scoring is meant to be an informal process, hopefully, everyone in your group is doing well on teamwork. However, if you find that your assessment of any of your group members falls below a score of 10 (out of a possible 18), this is a good indication that it's time for a meeting to discuss your group dynamics. In this case, based on past experience, you are strongly advised to meet with your instructor if you can't resolve matters right away within your group. Please do so before it's too late.

Last updated on 14 October 2017