Ruxi Ruxi - 2 years ago 94
C++ Question

How to reduce the huge time cost (10 seconds) by cublasCreate()?

I am implementing a conjugate gradient (CG) solver using the cuSPARSE_v2/cuBLAS_v2 libraries to cope with a large sparse matrix in my research. The weird thing I observed is the huge time cost by cublasCreate() function, ~ 10 seconds. I am aware that the library initialization cost is usually large, but by searching forums I found the usual time cost of cublasCreate is of ~100 ms scale, not as much as 10s. While the whole CG iteration part only cost 0.6 ~ 1 second. I also implemented CG solvers using CUSP library, which performed quite well - with the total code time of ~ 0.5 second.

So how to reduce the time cost by cublasCreate()? Also, if large as 10s a must-have for cuda library initialization, why CUSP library performs much better, with a nearly neglectable initialization cost?

I am using CUDA-7.5 on GTX 980 Ti. Here is my code snippet with timing:

// Timing begin
struct timeval begin, end;
gettimeofday(&begin, 0);

cublasStatus = cublasCreate(&cublasHandle);

// Timing end
gettimeofday(&end, 0);
float cgtime = (end.tv_sec - begin.tv_sec) * 1000.0 + (end.tv_usec - begin.tv_usec) / 1000.0;
printf("\nTime elapse: %f ms.\n", cgtime);

Thanks a lot!

Answer Source

I finally found the cause - our main server node didn't function well and couldn't communicate with GPU nodes normally, which somehow caused the dynamic linking of cuBLAS library hindered. A reboot recovered all.

So there is no problem with cublasCreate() at this point. I post it here as an answer in case anyone encounters a similar situation (though low probability).

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download