Attachments can be downloaded by annonomous users if they know the url.

I have a Drupal commons Version: 7.x-3.38 with og_access. On both an invitation only community and a Joining requires admin approval Hide contributed content from non-members I am able to still reach attachments to those communities if I copy and paste the address into the URL. The community cannot be reached but my concern is people knowing the url are still able to reach the document even if you are not part of the community or even logged in at all.
I have tried setting up a private file system path outside of root which protects the files but only the owner can download the file.
How can I configure Drupal commons to have attachments follow the community settings.