Is there a config way to set this up without having to specify which branch?
Git already only pulls the current branch. If you have branch set up as a tracking branch, you do not need to specify the remote branch.
git branch --set-upstream localbranch reponame/remotebranch will set up the tracking relationship. You then issue
git pull [--rebase] and only that branch will be updated.
Of course, all remote tracking branches and all refs for the remote will be updated, but only your local tracking branch will be modified.