limit the number of groups that user can join simultaneously

I can't code every use case I get a support request for, but It wouldn't be that hard to check for existing memberships:

if (count(group_membership_load_by_user($uid)) > 0) {
  // The user already has a membership somewhere, run your logic here.
}

As soon as I get some time on my hands, I'll add [user:group-memberships] to the available tokens so you can use Rules as well.

How can I prevent adding user to second group with this code? Is there some hook for adding or inviting user to a group?

If a person joins a group in any way a GroupMembership entity is created and saved.

So you should be able to react on hook_entity_insert or hook_group_membership_insert and remove the newly created membership if they user has reached his threshold. Quite the overhead for something you'd want to prevent, not clean up after the fact, but it should work.