mcautograder Demo Notebook

This notebook is intended to demonstrate the multiple-choice autograder. We import the autograder below along with some other libraries.

In [ ]:
from utils import *
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use("ggplot")

Autograder Setup

This notebook contains a series of demo questions as an example of how to use the autograder. To set up the autograder, you need to import the file mcautograder.py and create an instance of the Notebook class. If you want the assignment to be scored, make sure that you set the scored argument to True. In this notebook, we will run 2 autograders, one with scoring and one without. It is also possible to set a maximum number of retakes, which we will do with the unscored autograder.

In [ ]:
import mcautograder
grader = mcautograder.Notebook("tests.py", max_attempts=2)
scored_grader = mcautograder.Notebook("tests.py", scored=True)

Demo Questions

Please see the demonstration questions below. While reading through them, it may be helpful to look at the structure of tests.txt as this is the file that encodes the answers.

Question 1

Which is the right answer?

  1. Not this one
  2. Keep going
  3. Yup! This is it!
  4. You've gone too far. Go back to (3).

Assign your answer to q1 below.

In [ ]:
q1 = 3
In [ ]:
grader.check("q1", q1)
In [ ]:
scored_grader.check("q1", q1)

Question 2

What is the approximate distribution of the variable in the plot below?

  1. Poisson
  2. Normal
  3. Uniform
In [ ]:
q2_plot()

Assign your answer to q2 below.

In [ ]:
q2 = 2
In [ ]:
grader.check("q2", q2)
In [ ]:
scored_grader.check("q2", q2)

Question 3

Which of the following will generate 500 normal random variables ($\mu=0$, $\sigma^2=4$)?

A. np.random.normal(500, 0, 4)

B. np.random.normal(0, 500, 4)

C. np.random.normal(4, 0, 500)

D. np.random.normal(0, 4, 500)

Assign your answer to q3 below.

In [ ]:
q3 = "D"
In [ ]:
grader.check("q3", q3)
In [ ]:
scored_grader.check("q3", q3)

Max Retakes

Because we set the max_retakes parameter to 2 in the grader autograder, we have 2 chances to answer each question, one of which we've already used. This means the cell below will run fine, but the one below that will error out because we've hit the maximum number of retakes.

In [ ]:
grader.check("q1", q1)
In [ ]:
# this will error
grader.check("q1", q1)

Persistence

One feature added in v0.0.6 prevents students from bypassing the maximum number of retakes by storing the state of the autograder every time Notebook.check() is called. This file is loaded whenever the Notebook is initialzied, so that killing a kernel does not mean that the Notebook forgets its previous state.

An important consequence of this feature is that if you are testing the autograder locally, you must be sure not to distribute your copy of the state to your students, lest that prevent that from using the autograder. The file it is stored in is .MCAUTOGRADER_STATUS, so this should be added to your .gitignore or some other such file to prevent it from being sent out.

Acceptable Inputs

The autograder accepts single character (of type str) or single-digit (of type int) answers. If the answer provided is neither or these, then the autograder will throw an assertion error.

In [ ]:
# this will error
grader.check("q3", "this will error")

The autograder will also error if a question that is not in the tests file is passed as the first argument.

In [ ]:
# this will error
grader.check("question1", 2)

Scoring Assignments

For the scored autograder, you can have students run Notebook.score() to see their score at the end of the notebook. It is best that students wait until the end because in the beginning their scores will be very low as they haven't answered any questions yet.

In [ ]:
scored_grader.score()

The Tests File

The tests file contains the answers to your multiple choice questions. You can name the file however you like, but the relative path to the file from the notebook needs to be passed to the Notebook instance you create when initializing the autograder. If you want your answer key to be hard to find, I recommend making it a hidden file (i.e. .tests.txt instead of tests.txt). This repo has an unhidden key so that you can see the structure of the key. When I deploy the autograder, I always use a hidden file.

You Try

In this section of the notebook, you will write your own question and use Python to append it to the answer key.

Step 1: Create your Question

In the Markdown cell below, type your question.

Type your question here, replacing this text.

Step 2: Create an Identifier and an Answer

Now, assign identifier to the identifier string for your question and answer to the correct answer.

In [ ]:
identifier = ...
answer = ...

Step 3: Add your Answer to the Tests File

Now you need to create your own tests file. Right click on the Jupyter icon in the top left of this notebook to open up your server in a new tab. You should see a folder call demo; open it. In this folder, create a new text file and then open it in the editor. Set the file name to my_tests.py and then copy the code below into the file, replacing _identifier_, _answer_, and _points_ with the values you defined above and the number of points you want your question to be worth.

answers = [
    {
        "identifier": _identifier_,
        "answer": _answer_,
        "points": _points_
    }
]

You can continue on in this notebook to load your autograder once that file is saved.

Step 4: Initialize the Autograder

Now we can initialize the autograder. Decided whether or not you want the autograder to score the assignment and how many retakes you want. If you don't care how many retakes are allowed, set it to its default "inf".

In [ ]:
my_grader = mcautograder.Notebook("my_tests.py", scored = ..., max_attempts = ...)

Step 5: Run the Check

Finally, verify that your question is correct in the cell below. You can play around with the values to see how the function behaves based on different types of inputs.

In [ ]:
my_grader.check(identifier, answer)

Step 6: Score the Assignment (if applicable)

Run Notebook.score() to get the score. It will be low because you haven't answered the other questions since reinitializing the autograder.

In [ ]:
my_grader.score()