Problem/Motivation

The quiz scoring system is very convoluted. Some of the code dates back to Quiz 3.0, especially the manual grading routines. The question scoring forms differ depending on what context they are used in (a single answer context vs. per-result context).

Additionally, the max score, score, and weight make zero sense when looking at it cold. The weighting calculations are also set in multiple places, so there is no reliable way to get the weighted or unweighted score.

Proposed resolution

- Rip out redundant forms in procedural form functions (short_answer_score_form_submit) and use the already provided report forms
- Clean up global single-answer grading (Entity API changes)
- Clean up quiz_calculate_score (eliminate non-Entity API db_* operations)
- Clean up quiz_end_scoring
- Clean up quiz_store_question_result
- Improve/document QuizQuestionResponse::getMaxScore, QuizQuestion::getMaximumScore, QuizQuestionResponse::getScore
- Add QuizQuestionResponse::getWeightedRatio and use it instead of local calculations

Remaining tasks

None

User interface changes

None

API changes

- Added QuizQuestionResponse::getWeightedRatio

Data model changes

None

Comments

FreekVR’s picture

Issue summary: View changes
djdevin’s picture

FreekVR’s picture

Probably, the issue seems to be the value thats inserted into the "score" column of "quiz_node_results". It calculates a percentage based on the values returned by the question classes, ignoring custom points configured by the admin. The "quiz_node_results_answers" does however show the correct amount of points awarded for a specific quiz result set.

The percentage that's stored in quiz_node_results is not being used to determine whether the quiz was completed successfully. This leads to a use case where I have a quiz with a pass rate of 60%, which I've answered a 100% correct in terms of points. When walking trough the quiz I get the correct message at the end telling me I've passed - but when looking at the results page (or the quiz_node_results table) I'm getting a pass rate of 46%.

In this case it's being caused by there being ~50 multichoice questions, of which I've only added a score to 10 of them - the others are basically "warming up" questions that should not be taken into account when determining if a user has passed.

The quiz function "quiz_is_passed()" uses the percentage directly from "quiz_node_results" returning an incorrect boolean wherever it's called.

FreekVR’s picture

Ok, so my confusion stems from the fact that a question can have two maximum scores: One determined by the question itself (stored in quiz_question_properties) and one for a specific quiz relationship (stored in quiz_node_relationship).

In the following situations the method getMaximumScore is called, which will return a (partially) hardcoded value

  1. quizQuestion->save() (‘max_score’ in quiz_question_properties table)
  2. quizQuestion->saveRelationships() (‘max_score’ insert/update into quiz_question_relationship)
  3. quizQuestion->getMaxScore (sets question->max_score property)
  4. quiz_question_quiz_question_score() (sets score->possible property)

There are tons of references everywhere and various methods for getting values from different tables, making this confusing to debug. I'll try to create a patch to fix the "correct percentage" as stored in the results. There are probably other issues with scoring but I don't yet understand the module enough to make any kind of non-breaking patch. A suggestion would be to use object properties with getters/setters which would make everything easier to understand, but this would require a major refactor of, well, everything :P

FreekVR’s picture

StatusFileSize
new3.09 KB

Here's a patch to correctly use the value, if available, from the question relationship to a specific quiz

FreekVR’s picture

Status: Active » Needs review
djdevin’s picture

Well, it didn't break anything :) Thanks!

Since the bug wasn't covered in the tests, let's get one written that fails without this patch.

FreekVR’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests
rodgey’s picture

I am running into a problem related to this as well.

Since some users have been testing and I manually reviewed all results, it seems I am able to locate the problem in my case:

The calculated score was only inconsistent for one test user who didn't finish a quiz in time (timed quiz).

  • In the quiz report (node/%/quiz/results/%/view) the user has 17 of 30 possible points, which is calculated correctly to a score of 57%
  • In a view where the calculated scores are generated (field => 'Quiz result: Score') for the logged in users(contextual filter => 'Quiz result: Uid'), for this user a score of '60% correct' is generated.

Because the user still has to 'submit' the current current question, after the timer reached it's end, it seems very plausible the last question which this user submitted was evaluated as correct for the result generated in the second point from the list above, but the last submitted question didn't count as correct (as desired) for the first point form the list above.

Since I am not a coder (yet;) I am not sure if this situation is already covered by the currently listed issues:

  • Using a quiz with weighted scores, or optional questions with no points awarded even when correct - will NOT save a correct "percentage" of points earned.
  • When finishing the quiz, in quiz_calculate_score, the function "quiz_question_quiz_question_score" is called. This function will return:
  • When no quiz and no result ID are available, the maximum score for the question nid from the database
    When a result ID is available the maximum score as reported by the implementing QuizQuestionResponce class (which seems to in turn be calling the getMaximumScore() method of the question class)
  • When the quiz is available, but not the result ID, the maximum score as reported by the implementing question class

Hope this might help.

djdevin’s picture

I confirmed this is broken as it came up in a use case.

#5 fixes it, I'll try to write a test.

djdevin’s picture

I noticed it doesn't work in weighted points but maybe that isn't a fix here but elsewhere.

For example

One quiz with

MCQ 1 (1 point)
Answer 1 correct 3pts
Answer 2 correct 3pts
Answer 3 correct 3pts

Multiple answers
Simple score off

Results in a quiz result score of 300%

djdevin’s picture

djdevin’s picture

STR

Create a quiz with three True/False questions

Set question 1 to be worth 1 point
Set question 2 to be worth 5 points
Set question 3 to be worth 20 points

Take the quiz and answer the last question correct.

On the results page you should see (correct)

You got 20 of 26 possible points.
Your score: 77%

On the manage results page you should see 33% (incorrect)

The quiz_node_results_answers is correct (20/26 points)

quiz_node_results table is wrong (score 33)

djdevin’s picture

Status: Needs work » Needs review
StatusFileSize
new9.65 KB

There was a lot of legacy code from Quiz 3 lying around so I removed it and replaced it with the Entity approach.

The score weighting wasn't being checked in every case.

There were also a bunch of different places where the weight was being calculated, so I'm trying to find all of those places and replace them with API calls.

djdevin’s picture

StatusFileSize
new27.56 KB

Status: Needs review » Needs work

The last submitted patch, 16: scoring_system_can-2470103-16.patch, failed testing.

djdevin’s picture

Status: Needs work » Needs review
StatusFileSize
new29.15 KB

The last submitted patch, 16: scoring_system_can-2470103-16.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 18: scoring_system_can-2470103-18.patch, failed testing.

The last submitted patch, 18: scoring_system_can-2470103-18.patch, failed testing.

djdevin’s picture

Status: Needs work » Needs review
StatusFileSize
new40.03 KB

Status: Needs review » Needs work

The last submitted patch, 22: scoring_system_can-2470103-22.patch, failed testing.

The last submitted patch, 22: scoring_system_can-2470103-22.patch, failed testing.

djdevin’s picture

Status: Needs work » Needs review
StatusFileSize
new40.04 KB

Status: Needs review » Needs work

The last submitted patch, 25: scoring_system_can-2470103-25.patch, failed testing.

The last submitted patch, 25: scoring_system_can-2470103-25.patch, failed testing.

djdevin’s picture

Status: Needs work » Needs review
StatusFileSize
new40.04 KB
djdevin’s picture

Title: Scoring system can return inconsistent values » Refactor scoring system to use Entity and Question API
Issue summary: View changes
djdevin’s picture

Issue summary: View changes
djdevin’s picture

StatusFileSize
new39.95 KB

Added back support for weighted random questions. Replaced old blocks of code.

djdevin’s picture

StatusFileSize
new5.52 KB
djdevin’s picture

StatusFileSize
new5.97 KB
djdevin’s picture

StatusFileSize
new6.81 KB
new48.63 KB

The last submitted patch, 34: fail.patch, failed testing.

The last submitted patch, 34: refactor_scoring_system-2470103-34.patch, failed testing.

The last submitted patch, 34: fail.patch, failed testing.

The last submitted patch, 34: refactor_scoring_system-2470103-34.patch, failed testing.

djdevin’s picture

StatusFileSize
new48.88 KB

  • djdevin committed 6d2ba45 on 7.x-5.x
    Issue #2470103 by djdevin, FreekVR: Refactor scoring system to use...
djdevin’s picture

Status: Needs review » Fixed
Issue tags: -Needs tests

This is fixed, with tests.

Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

bonrita’s picture

StatusFileSize
new2.29 KB

Rewriting the patch to be compatible with version 5.5

bonrita’s picture

StatusFileSize
new2.05 KB

Rewriting the patch to be compatible with version 5.5

bonrita’s picture

bonrita’s picture

This Patch that is compatible with version 5.0-alpha10