Git bash looks for an ssh private key in
HOME being set by the git bash to
Start PuTTY, go to Connection->SSH->Auth and select your key
In order for TortoiseGit to use ssh instead of putty, as the OP Jason comments, you need to specify
ssh.exe in the network settings.
That is true especially if you have generated ssh keys with openssh (
id_rsa.pub), and not with
(Image from blog post "Setting up Git and TortoiseGit with Bitbucket, step by step" by Guganeshan.T)