How does the PLQ thing work?

The format of the PLQ questions is to have N quick questions in each lecture (hopefully around 3 or more). You will use the PLQ app (on your phone or laptop) to submit an answer. Questions vary from short to extremely short, and the subject is either from the current lecture material or from the last one.

The goal is to have enough data to take the place of a traditional exam as the main way of evaluating your performance. The average numbers tend to be similar to those of on exam, and having a lower average on them compared to your homework grades is as common as it is in the case of an exam.

Since this is replacing the role of an exam, you should be prepared for it: make sure you pay attention to your overall numbers. However, there are going to be many questions throughout the semester, so there’s is no need to stress about each and every PLQ as you do in exams. (See the next section.)

Why you should not be stressed about it

While the PLQ is our exam substitute, making it as important as exams, do not make the mistake of stressing over it as you do for an exam. The main difference is that this is taken in tiny chunks throughout the semester. There are a few important things to learn about the format to avoid stressing about the PLQs:

• Assuming a PLQ in every class, we’re likely to drop two or more at the end of the semester. This is a good way to “smooth out” the numbers and avoid outliers due to missing class, feeling bad, or temporarily blanking out on one.
• “PLQ” means “PL Question” — this is not an exam or a quiz. It’s true that it’s a format that is similar to a quiz, but the important difference is that it’s a collection of many PLQs throughout the semester, so missing one is not necessarily leading to a grade penalty. If you’re missing too many, I’ll adjust the number of auto-drops for you so there’s still enough information to make the grade stable.
• For the same reason, I try to ask several questions rather than just one. This further smoothes the occasional local mistake. Thinking about the PLQ grades as data that is used to measure your performance makes it clear that more numbers lead to better correlation and therefore a grade that reflects your performance.
• The grading of each PLQ is not done in a clinical vacuum: I do pay attention to how the whole class did. If there’s a question where most people chose some wrong answer, I will usually note it and give it a higher grade and/or lower its overall weight. This serves as protection from confusing or badly phrased answers.
• Generally speaking, I pay more attention to the overall curve and your position on it, rather than looking at the absolute numbers. This means that to really mess up (or really shine), you need to fail when most people didn’t (or succeed when most didn’t).
• At the end of the semester, I look at the overall picture and adjust the weights (among the PLQs, the homework, and the weights between both) to get a normal looking curve and grade range. So to check your status you should see how it fits on the class chart before you reach any conclusions.
• Just to be extra clear: You are not expected to get them all correctly! The whole point is that this is a statistical device that evaluates your performance relative to the curve. (And that does not mean that there’s some fixed percentile that will fail: it’s likely (and common!) for the whole class to do pretty well.)

And as usual, when in doubt, see me at office hours or schedule a meeting. I will go over the numbers with you and give your my interpretation and impression.

Avoiding stress

People are often too stressed about their PLQ performance. In addition to the above, see the “stress” question in the FAQ.

Suggesting PLQ questions

If you think that your PLQ performance (or your overall performance) is too bad, or if you just want to learn the material better, you should consider making a PLQ Suggestion.

The idea is to give the students some more control over the questions that are asked on the PLQs. This is intended for two main reasons: first, it prevents questions that we think should be reasonable but turn out to be harder than usual (maybe because some subject wasn’t explained properly), and second, if you do make such suggestions, it will generally be a good way to boost your overall performance. As further motivation to suggest questions, karma cookies are given for suggested questions (at a value that corresponds to how good it is).

When you think about a question, please keep in mind the style of questions so far, what you like and don’t like about them. Keep in mind the considerations and preferences of other students too — for example, most students prefer a question about the material from the last class. (But you can also suggest a question about the upcoming class if you go over the material in advance.)

Please include a few options for answers too, including the right one (or the one you think is correct). If it’s a question that you want to ask but don’t know the answer to then that’s fine to: if it’s a good question it’ll get included and we will come up with the answer(s). You can also meet me at office hours and we can go over it together. Also, feel free to include comments like the purpose of the question (e.g., a recap-style question), or maybe a note that it’s a question that sound good to you but you think that it’s too hard as-is and maybe we can manage to water it down a bit, or any other notes you might have on it.

Also, of course, make it an actual question rather than something that is unusable like silly questions, obvious trivia, unrelated questions, or a suggestion for a question rather than a question. Again, if your question is used (and often even if it doesn’t) you’ll get a corresponding karma cookie too. Also, we’re likely to tweak the question and/or the answers, but you’ll probably still have the advantage of being the question’s author, and having researched the material to write it, you should still be in a good position to answer it.

Bear in mind that I pay attention to how students perform on PLQs, which means that things should be fine even if your suggestion was used but some disaster happened (e.g., nobody got it right, or it didn’t have the correct answer). In such cases it will be graded leniently and/or its weight will be adjusted accordingly.

Writing a good PLQ

Writing a good PLQ is difficult. Here are some tips to keep in mind:

Intention

A PLQ should affirm knowledge of some subset of the covered material. This means that the goal of a PLQ is not to test minutiae, trivia, or memorization. A good PLQ should be related to major concepts of the material, or details and nuances of a concept that has a meaningful significance.

You should avoid writing PLQs that require specific knowledge of one sentence in the class notes. Instead, try to summarize the main points: things like major implementation concepts, fundamental requirements, the overall context, significant drawbacks, problems, or features. With a list like this distilled, think about questions about these concepts. How can you show a specific drawback’s behavior? What is unique about the changes we’ve seen? What requirements are necessary for things to function as they are supposed to for a new feature to be well behaved?

Time Limit

PLQs are typically short questions, with 1-3 minutes allocated for answering them. Considering this, a question that takes a significant chunk of that time to parse and understand is not the best. Code snippets should be as small as you can get to minimally display the behavior that the question is getting at. While wrapping the question in layers of indirection might make the question trickier to answer, it only succeeds in masking out the core concept the question is getting at, therefore missing the point. It shifts the question’s goal from core understanding question towards quick comprehension.

Answers

When you write a question, you usually will have a particular answer in mind. This is great, but when writing other answers — the wrong answers — be mindful of how those options might differ from both the correct answer, and the material at large. The same rules in the first point should apply. The difference in answers are part of the question, so it’s best to avoid trivia and memorized bits of text, and instead should be related to the fundamental concept you’re demonstrating. In the case of a “choose the best answer” question, answers should play on each other in meaningful ways. With all of this in mind, it is worth noting that you do not need to know the answer to a PLQ to ask the question — sometimes that can even be more valuable. (Eli will go over the questions and will adjust/correct answers when needed.)

Closing Notes

Repeating the above, writing a PLQ betters your own knowledge of the material. So when writing a PLQ, consider writing about the aspects of the material that you found confusing. Exploring these topics through writing PLQs will necessarily leave you with a better understanding of that material, and potentially reveal things that can be massaged into a good PLQ question.

Mini PLQ FAQ

• Where/how do I make a suggestion?

  The place to make these suggestions is the PLQ Suggestion page. Note that you need to be logged in (as with the PLQ itself) to be able to submit a suggestion. (If you encounter any problems with this page, please let me know. Also, write your question elsewhere first, so if all else fails you can just email it.)

• I don’t know the material well enough, what can I suggest?

  The idea is that just by trying to suggest a question, you’re already doing some re-reasing of the material in a different mindset which should help you do better even if no actual question comes out of it.

• I have a question, but I’m not sure about the correct answer…?

  Suggestions are not just used as-is, so don’t worry about whether the answers are correct or not. If your question is used, and even if you weren’t sure about the answer, you’d still be in a better position since you’ve reviewed the relevant material.

• The PLQs are hard, I don’t want to make more of them

  The PLQs are not based only on student suggestions, so the number of questions does not depend on the number of suggestions. A better way to think about it is that student suggestions generally tend to be easier than the questions that I would come up with instead.

Other alternatives?

I decided to avoid exams for various reasons, and the PLQ is the best substitute that I found for them. I know that it has some issues, but all other alternatives have bigger problems. If you think that you have a better alternative, feel free to email me — but keep in mind that I have been actively looking for alternatives for a long time, including asking students for ideas. (So please avoid non-solutions like using only homework.)