Question says it all really. This is more for my own curiosity to see if anyone knows of a particular reason why this should be avoided!
The only thought I have had is that perhaps, even if no library functions are called, the compiler will still generate loading code and load the library even though it will not be used. I'm guessing here, would be interested to know what the implications actually are!
At program start up you should expect that the library will be loaded. This is a memory usage and start up time impact (I am not saying how small/big an impact). This is the standard case on Windows with the Microsoft tool chain and, as I understand this answer to another question, is also the case on Linux.
Note that this is different than using LoadLibrary on Windows. LoadLibrary is a Windows facility to explicitly load and use a dynamic library at run time without linking at build time.
The Microsoft tool chain does allow you to optionally specify that a Dll is to be delay loaded. If you do this, then the dll will not actually be loaded until/unless you use the dll. The tool chain does this by substituting build time dll linking with run time loading on demand.