I'm trying to build the R openssl package and link it to an openssl library that is a newer version than the one in the systemwide lib64 directory.
The Makevars file simply has:
R CMD INSTALL
gcc -std=gnu99 -shared -L/usr/local/lib64 -o openssl.so [...objects...] -Lmy/path/here -lssl -lcrypto
I solved this by passing arguments to R CMD INSTALL. I.e.:
R CMD INSTALL . --configure-vars="INCLUDE_DIR=$HOME/local/include LIB_DIR=$HOME/local/lib64"
But this varies for each package. There's no general solution. Instead, read config and src/Makefile.in.