[ASK] The Training using a GPU is very slow for NCF model

[orrimoch]

I've set up an virtual environment based on the following commands:
!conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1
!pip install recommenders[gpu]

Moreover I've added the following commands in the code:

tf.debugging.set_log_device_placement(True)
gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.set_visible_devices([], 'CPU') # hide the CPU
tf.config.set_visible_devices(gpus[0], 'GPU') # unhide potentially hidden GPU
tf.config.get_visible_devices()

Set the environment to use GPU

physical_devices = tf.config.list_physical_devices('GPU')
if physical_devices:
tf.config.experimental.set_memory_growth(physical_devices[0], True)

Non of the steps above helped me to increase the utilization of the gpu.
Any suggestion would be helpful.

Thanks,
Or