Problem/Motivation
The quiz scoring system is very convoluted. Some of the code dates back to Quiz 3.0, especially the manual grading routines. The question scoring forms differ depending on what context they are used in (a single answer context vs. per-result context).
Additionally, the max score, score, and weight make zero sense when looking at it cold. The weighting calculations are also set in multiple places, so there is no reliable way to get the weighted or unweighted score.
Proposed resolution
- Rip out redundant forms in procedural form functions (short_answer_score_form_submit) and use the already provided report forms
- Clean up global single-answer grading (Entity API changes)
- Clean up quiz_calculate_score (eliminate non-Entity API db_* operations)
- Clean up quiz_end_scoring
- Clean up quiz_store_question_result
- Improve/document QuizQuestionResponse::getMaxScore, QuizQuestion::getMaximumScore, QuizQuestionResponse::getScore
- Add QuizQuestionResponse::getWeightedRatio and use it instead of local calculations
Remaining tasks
None
User interface changes
None
API changes
- Added QuizQuestionResponse::getWeightedRatio
Data model changes
None
| Comment | File | Size | Author |
|---|---|---|---|
| #46 | scoring_system_incorrect_percentage-2470103-10.patch | 3.11 KB | bonrita |
| #44 | scoring_system_can-2470103-44.patch | 2.05 KB | bonrita |
| #39 | refactor_scoring_system-2470103-39.patch | 48.88 KB | djdevin |
Comments
Comment #1
FreekVR commentedComment #2
djdevinI wonder if this is related to #2438421: Correct score is shown on quiz but incorrect score is shown in Quiz results table
Comment #3
FreekVR commentedProbably, the issue seems to be the value thats inserted into the "score" column of "quiz_node_results". It calculates a percentage based on the values returned by the question classes, ignoring custom points configured by the admin. The "quiz_node_results_answers" does however show the correct amount of points awarded for a specific quiz result set.
The percentage that's stored in quiz_node_results is not being used to determine whether the quiz was completed successfully. This leads to a use case where I have a quiz with a pass rate of 60%, which I've answered a 100% correct in terms of points. When walking trough the quiz I get the correct message at the end telling me I've passed - but when looking at the results page (or the quiz_node_results table) I'm getting a pass rate of 46%.
In this case it's being caused by there being ~50 multichoice questions, of which I've only added a score to 10 of them - the others are basically "warming up" questions that should not be taken into account when determining if a user has passed.
The quiz function "quiz_is_passed()" uses the percentage directly from "quiz_node_results" returning an incorrect boolean wherever it's called.
Comment #4
FreekVR commentedOk, so my confusion stems from the fact that a question can have two maximum scores: One determined by the question itself (stored in quiz_question_properties) and one for a specific quiz relationship (stored in quiz_node_relationship).
In the following situations the method getMaximumScore is called, which will return a (partially) hardcoded value
There are tons of references everywhere and various methods for getting values from different tables, making this confusing to debug. I'll try to create a patch to fix the "correct percentage" as stored in the results. There are probably other issues with scoring but I don't yet understand the module enough to make any kind of non-breaking patch. A suggestion would be to use object properties with getters/setters which would make everything easier to understand, but this would require a major refactor of, well, everything :P
Comment #5
FreekVR commentedHere's a patch to correctly use the value, if available, from the question relationship to a specific quiz
Comment #6
FreekVR commentedComment #7
FreekVR commentedComment #8
djdevinWell, it didn't break anything :) Thanks!
Since the bug wasn't covered in the tests, let's get one written that fails without this patch.
Comment #9
FreekVR commentedComment #10
rodgey commentedI am running into a problem related to this as well.
Since some users have been testing and I manually reviewed all results, it seems I am able to locate the problem in my case:
The calculated score was only inconsistent for one test user who didn't finish a quiz in time (timed quiz).
Because the user still has to 'submit' the current current question, after the timer reached it's end, it seems very plausible the last question which this user submitted was evaluated as correct for the result generated in the second point from the list above, but the last submitted question didn't count as correct (as desired) for the first point form the list above.
Since I am not a coder (yet;) I am not sure if this situation is already covered by the currently listed issues:
Hope this might help.
Comment #11
djdevinI confirmed this is broken as it came up in a use case.
#5 fixes it, I'll try to write a test.
Comment #12
djdevinI noticed it doesn't work in weighted points but maybe that isn't a fix here but elsewhere.
For example
One quiz with
MCQ 1 (1 point)
Answer 1 correct 3pts
Answer 2 correct 3pts
Answer 3 correct 3pts
Multiple answers
Simple score off
Results in a quiz result score of 300%
Comment #13
djdevinComment #14
djdevinSTR
Create a quiz with three True/False questions
Set question 1 to be worth 1 point
Set question 2 to be worth 5 points
Set question 3 to be worth 20 points
Take the quiz and answer the last question correct.
On the results page you should see (correct)
You got 20 of 26 possible points.
Your score: 77%
On the manage results page you should see 33% (incorrect)
The quiz_node_results_answers is correct (20/26 points)
quiz_node_results table is wrong (score 33)
Comment #15
djdevinThere was a lot of legacy code from Quiz 3 lying around so I removed it and replaced it with the Entity approach.
The score weighting wasn't being checked in every case.
There were also a bunch of different places where the weight was being calculated, so I'm trying to find all of those places and replace them with API calls.
Comment #16
djdevinComment #18
djdevinComment #22
djdevinComment #25
djdevinComment #28
djdevinComment #29
djdevinComment #30
djdevinComment #31
djdevinAdded back support for weighted random questions. Replaced old blocks of code.
Comment #32
djdevinComment #33
djdevinComment #34
djdevinComment #39
djdevinComment #41
djdevinThis is fixed, with tests.
Thanks!
Comment #43
bonrita commentedRewriting the patch to be compatible with version 5.5
Comment #44
bonrita commentedRewriting the patch to be compatible with version 5.5
Comment #45
bonrita commentedComment #46
bonrita commentedThis Patch that is compatible with version 5.0-alpha10