How do I set up a view for a user flag to make a follower/following block

Hi, I am trying to set up a user flag view to display who a user is following and who is following said user. I am using the flag module to do this. I have set up the "following" view like this.
1. Created new user view (I called it follow).
2. Created new block display (following block) with row style fields.
3. Under relationships I added "Flags:User flag", labeled it "following" , chose my "follow" flag, checked "include only flagged content", and selected by current user.
4. Added another relationship "Flags:User", gave it the relationship to "following" (my first relationship) and checked require this relationship.
5.Added the field for "User:name" and for "Flags:link.

I think that this is working correctly to display all the people that the currently logged in user is "following".

I also need to make a display to show who is following the currently logged in user. This is where im getting stuck.
Here is what I have so far.

1. Created new block display (followers block) in my "follow" view, with row style fields and distinct checked.
2. Under relationships I added "Flags:User flag", labeled it "follower" , chose my "follow" flag, checked "include only flagged content", and selected by any user.
3. Added another relationship "Flags:User's flagged content", checked include only flagged content and chose my "follow" flag.
4. Added an argument for "User:Id", checked provide default argument and checked "User id from logged in user", lastly checked exclude from view ( I set up this argument because the logged in user's name was showing in the list.
5.Added the field for "User:name".

This view gives me a list of all the people that are following the logged in user, but it also shows certain (but not all) people that another user is following. Also, if the logged in user chooses to follow certain (not all) people it show them being followed by said people. This is no good. What am I missing here? Also, is there a better way for me to do my "following" view? Sorry if this is confusing. Thanks.