You are assigning permissions at wrong place. you are giving extra privilege by adding user under site setting. That user can do anything in the the report manager.
It should provide permission at Report or Folder Level
Go To properties of report > Security > Provide Report builder access (if user is supposed to modify in place in report manager)
Also, as per your comment, check if the mentioned domain user can/supposed to connect to your report manger.