mcautograder
Demo Notebook¶This notebook is intended to demonstrate the multiple-choice autograder. We import the autograder below along with some other libraries.
from utils import *
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use("ggplot")
This notebook contains a series of demo questions as an example of how to use the autograder. To set up the autograder, you need to import the file mcautograder.py
and create an instance of the Notebook
class. If you want the assignment to be scored, make sure that you set the scored
argument to True
. In this notebook, we will run 2 autograders, one with scoring and one without. It is also possible to set a maximum number of retakes, which we will do with the unscored autograder.
import mcautograder
grader = mcautograder.Notebook("tests.py", max_attempts=2)
scored_grader = mcautograder.Notebook("tests.py", scored=True)
Please see the demonstration questions below. While reading through them, it may be helpful to look at the structure of tests.txt
as this is the file that encodes the answers.
Which is the right answer?
Assign your answer to q1
below.
q1 = 3
grader.check("q1", q1)
scored_grader.check("q1", q1)
What is the approximate distribution of the variable in the plot below?
q2_plot()
Assign your answer to q2
below.
q2 = 2
grader.check("q2", q2)
scored_grader.check("q2", q2)
Which of the following will generate 500 normal random variables ($\mu=0$, $\sigma^2=4$)?
A. np.random.normal(500, 0, 4)
B. np.random.normal(0, 500, 4)
C. np.random.normal(4, 0, 500)
D. np.random.normal(0, 4, 500)
Assign your answer to q3
below.
q3 = "D"
grader.check("q3", q3)
scored_grader.check("q3", q3)
Because we set the max_retakes
parameter to 2 in the grader
autograder, we have 2 chances to answer each question, one of which we've already used. This means the cell below will run fine, but the one below that will error out because we've hit the maximum number of retakes.
grader.check("q1", q1)
# this will error
grader.check("q1", q1)
One feature added in v0.0.6 prevents students from bypassing the maximum number of retakes by storing the state of the autograder every time Notebook.check()
is called. This file is loaded whenever the Notebook
is initialzied, so that killing a kernel does not mean that the Notebook forgets its previous state.
An important consequence of this feature is that if you are testing the autograder locally, you must be sure not to distribute your copy of the state to your students, lest that prevent that from using the autograder. The file it is stored in is .MCAUTOGRADER_STATUS
, so this should be added to your .gitignore or some other such file to prevent it from being sent out.
The autograder accepts single character (of type str
) or single-digit (of type int
) answers. If the answer provided is neither or these, then the autograder will throw an assertion error.
# this will error
grader.check("q3", "this will error")
The autograder will also error if a question that is not in the tests file is passed as the first argument.
# this will error
grader.check("question1", 2)
For the scored autograder, you can have students run Notebook.score()
to see their score at the end of the notebook. It is best that students wait until the end because in the beginning their scores will be very low as they haven't answered any questions yet.
scored_grader.score()
The tests file contains the answers to your multiple choice questions. You can name the file however you like, but the relative path to the file from the notebook needs to be passed to the Notebook
instance you create when initializing the autograder. If you want your answer key to be hard to find, I recommend making it a hidden file (i.e. .tests.txt
instead of tests.txt
). This repo has an unhidden key so that you can see the structure of the key. When I deploy the autograder, I always use a hidden file.
Type your question here, replacing this text.
Now, assign identifier
to the identifier string for your question and answer
to the correct answer.
identifier = ...
answer = ...
Now you need to create your own tests file. Right click on the Jupyter icon in the top left of this notebook to open up your server in a new tab. You should see a folder call demo
; open it. In this folder, create a new text file and then open it in the editor. Set the file name to my_tests.py
and then copy the code below into the file, replacing _identifier_
, _answer_
, and _points_
with the values you defined above and the number of points you want your question to be worth.
answers = [
{
"identifier": _identifier_,
"answer": _answer_,
"points": _points_
}
]
You can continue on in this notebook to load your autograder once that file is saved.
Now we can initialize the autograder. Decided whether or not you want the autograder to score the assignment and how many retakes you want. If you don't care how many retakes are allowed, set it to its default "inf"
.
my_grader = mcautograder.Notebook("my_tests.py", scored = ..., max_attempts = ...)
Finally, verify that your question is correct in the cell below. You can play around with the values to see how the function behaves based on different types of inputs.
my_grader.check(identifier, answer)
Run Notebook.score()
to get the score. It will be low because you haven't answered the other questions since reinitializing the autograder.
my_grader.score()