I needed a while to set up some order quantity restrictions in Drupal Commerce and I've seen many different questions and issues on how to do such things. So here's what I did:

Tasks

  1. Set a line items order limit (quantity of same product)
  2. Set a product order limit (quantity of different products)

Requirements

1. Set a line items order limit (quantity of same product)

Usually, there are two places where customers can change the quantity of line items: in the add to cart form and the cart itself. So I build the following two rules to set a limit of 5 items per product.

This is for the add to cart form:

{ "rules_limit_atcf" : {
    "LABEL" : "Limit ATCF",
    "PLUGIN" : "reaction rule",
    "OWNER" : "rules",
    "REQUIRES" : [ "rules", "commerce_cart" ],
    "ON" : { "commerce_cart_product_add" : [] },
    "IF" : [
      { "data_is" : {
          "data" : [ "commerce-line-item:quantity" ],
          "op" : "\u003E",
          "value" : "5"
        }
      }
    ],
    "DO" : [
      { "data_set" : { "data" : [ "commerce-line-item:quantity" ], "value" : "5" } },
      { "entity_save" : { "data" : [ "commerce-line-item:order" ] } },
      { "drupal_message" : {
          "message" : "Enough is enough.",
          "type" : "warning"
        }
      }
    ]
  }
}

This is for the cart:

{ "rules_limit_cart" : {
    "LABEL" : "Limit cart",
    "PLUGIN" : "reaction rule",
    "OWNER" : "rules",
    "REQUIRES" : [ "rules", "entity" ],
    "ON" : { "commerce_line_item_update" : [] },
    "IF" : [
      { "data_is" : {
          "data" : [ "commerce-line-item:quantity" ],
          "op" : "\u003E",
          "value" : "5"
        }
      }
    ],
    "DO" : [
      { "data_set" : { "data" : [ "commerce-line-item:quantity" ], "value" : "5" } },
      { "drupal_message" : {
          "message" : "Enough is enough.",
          "type" : "warning"
        }
      }
    ]
  }
}

2. Set a product order limit (quantity of different products)

If you also want to limit the quantity of different products that a user can add to his cart, I found it best to show that this option isn't any longer available for the customer. This rule allows to add products to the cart till a limit of 10 is reached.

Disable the add to cart form:

{ "rules_disable_add_to_cart" : {
    "LABEL" : "Disable add to cart",
    "PLUGIN" : "reaction rule",
    "OWNER" : "rules",
    "REQUIRES" : [ "commerce_rules_extra", "commerce_stock" ],
    "ON" : { "commerce_stock_check_add_to_cart_form_state" : [] },
    "IF" : [
      { "commerce_rules_extra_compare_line_item_count" : {
          "order" : [ "site:current-cart-order" ],
          "operator" : "\u003E=",
          "value" : "10"
        }
      }
    ],
    "DO" : [
      { "commerce_stock_set_add_to_cart_form_state" : { "disabled" : "1", "text" : "Nope, that's it", "class_name" : "product-limit-disabled" } }
    ]
  }
}

Now let's validate the above rule in order to ensure that already opened tabs where the form is still enabled don't contradict it:

{ "rules_validate_disable_add_to_cart" : {
    "LABEL" : "Validate Disable add to cart",
    "PLUGIN" : "reaction rule",
    "OWNER" : "rules",
    "REQUIRES" : [ "commerce_rules_extra", "rules", "commerce_cart" ],
    "ON" : { "commerce_cart_product_add" : [] },
    "IF" : [
      { "commerce_rules_extra_compare_line_item_count" : { "order" : [ "commerce-order" ], "operator" : "\u003E=", "value" : "11" } }
    ],
    "DO" : [
      { "entity_delete" : { "data" : [ "commerce-line-item" ] } },
      { "drupal_message" : {
          "message" : "There's already more than you need in your cart.",
          "type" : "warning"
        }
      }
    ]
  }
}

Result

The result of this 2x2 rules should be that a user can order a max. of 10 different products, and max. 5 of each one.