Unstable delay when running on GPU T4

[xaiocaibi]

Describe the bug
When I use pycuda to execute the compiled trt model, there will be regular delay glitches in the program. When locating the problem, it was found that the location where the delay became high was applying for memory. Our guess is that the machine is performing certain tasks on a regular basis to cause this problem.
When nvidia-smi is executed while stress testing, it is found that the delay is unstable. Can you help me with a question?

The figure below is the phenomenon of time delay：

When executing the nvida-smi command：

I'm sure this problem is caused by executing the nvidia-smi command, when I remove nvidia-smi, the delay returns to normal.
But this problem still needs to be solved. Can you help me with a question?
Code with latency fluctuations：

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Environment (please complete the following information):

• OS: centos
• CUDA version:11.4
• CUDA driver version: [e.g. 470.86]
• PyCUDA version: 2021.1
• Python version: 3.7
• GPU T4

Additional context
Add any other context about the problem here.

[inducer]

Maybe try using a memory pool? Memory allocations, particularly repeated ones, are quite expensive on GPUs.