Unable to build Intel gptj Docker image due to installation errors

[eltimen]

Hi all,
I'm trying to run Intel benchmarks on my server. I have the following error when trying to build gptj Docker image using this script:

#12 [conda 1/1] RUN curl -fsSL -v -o ~/miniconda.sh -O  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh  &&     chmod +x ~/miniconda.sh &&     ~/miniconda.sh -b -p /opt/conda &&     rm ~/miniconda.sh &&     /opt/conda/bin/conda install -y python=3.9 cmake==3.22.1 &&     /opt/conda/bin/conda install -c conda-forge -y jemalloc==5.2.1 gperftools==2.10 pybind11==2.10.4 &&     /opt/conda/bin/conda install -c intel -y mkl==2023.1.0 mkl-include==2023.1.0 intel-openmp==2023.1.0 &&     /opt/conda/bin/conda clean -ya  &&     pip3 install --pre torch==2.1.0.dev20230711+cpu torchvision==0.16.0.dev20230711+cpu torchaudio==2.1.0.dev20230711+cpu --index-url https://download.pytorch.org/whl/nightly/cpu &&     pip install setuptools==58.2.0 &&     pip install cpuid nltk evaluate protobuf absl-py rouge-score==0.1.2 tqdm numpy cython sentencepiece accelerate &&     pip install git+https://github.com/intel/neural-compressor.git@a2931eaa4052eec195be3c79a13f7bfa23e54473

...

ERROR: Could not find a version that satisfies the requirement torch==2.1.0.dev20230711+cpu (from versions: 2.2.0.dev20231010+cpu, 2.2.0.dev20231204+cpu, 2.2.0.dev20231205+cpu, 2.2.0.dev20231206+cpu, 2.2.0.dev20231207+cpu, 2.2.0.dev20231208+cpu, 2.2.0.dev20231209+cpu, 2.2.0.dev20231210+cpu, 2.2.0.dev20231211+cpu, 2.2.0.dev20231212+cpu, 2.2.0.dev20231213+cpu, 2.3.0.dev20231214+cpu, 2.3.0.dev20231215+cpu, 2.3.0.dev20231216+cpu, 2.3.0.dev20231217+cpu, 2.3.0.dev20231218+cpu, 2.3.0.dev20231219+cpu, 2.3.0.dev20231220+cpu, 2.3.0.dev20231221+cpu, 2.3.0.dev20231222+cpu, 2.3.0.dev20231223+cpu, 2.3.0.dev20231224+cpu, 2.3.0.dev20231225+cpu, 2.3.0.dev20231226+cpu, 2.3.0.dev20231227+cpu, 2.3.0.dev20231228+cpu, 2.3.0.dev20231229+cpu, 2.3.0.dev20231230+cpu, 2.3.0.dev20231231+cpu, 2.3.0.dev20240101+cpu, 2.3.0.dev20240102+cpu, 2.3.0.dev20240103+cpu, 2.3.0.dev20240104+cpu, 2.3.0.dev20240105+cpu, 2.3.0.dev20240106+cpu, 2.3.0.dev20240107+cpu, 2.3.0.dev20240108+cpu, 2.3.0.dev20240109+cpu, 2.3.0.dev20240110+cpu, 2.3.0.dev20240111+cpu, 2.3.0.dev20240113+cpu, 2.3.0.dev20240114+cpu, 2.3.0.dev20240115+cpu, 2.3.0.dev20240116+cpu, 2.3.0.dev20240117+cpu, 2.3.0.dev20240118+cpu, 2.3.0.dev20240119+cpu, 2.3.0.dev20240120+cpu, 2.3.0.dev20240121+cpu, 2.3.0.dev20240122+cpu, 2.3.0.dev20240123+cpu, 2.3.0.dev20240124+cpu, 2.3.0.dev20240125+cpu, 2.3.0.dev20240126+cpu, 2.3.0.dev20240127+cpu, 2.3.0.dev20240128+cpu)
#12 167.7 ERROR: No matching distribution found for torch==2.1.0.dev20230711+cpu

Can anybody provide a solution/workaround for this? Can changing the torch version to a latest dev version affect the validity of the benchmark results?

I also had to edit the Dockerfile by adding a -y to some conda install commands to prevent a hang when waiting for user confirmation.

[arjunsuresh]

We had a fix for this which was to build pytorch from the git hash corresponding to the wheel file. Will just double check everything and share the recipe shortly.