Thumbnailing

[will-moore]

Since the current data have only 1 multiscales resolution, generation of thumbnails in IDR takes a long time as the full-scale images must be retrieved to generate thumbnails.

The script at https://github.com/IDR/idr0125-way-cellpainting/blob/ffe378766685f418774f43ee5b78913ac57c679a/scripts/add_downsampling.py is in development as a way of adding down-sampled resolutions.

To test, lets download a full plate...
On idr-ftp server...

conda activate aws
cd /data/idr0125/
mkdir SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr

aws s3 sync --no-sign-request s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr

[will-moore]

Looks like full plate downloaded

$ find ./ -name .zattrs | wc
   3842    3842  270564

 python add_downsampling.py SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr 8

[will-moore]

Then upload to bucket..

$ /home/wmoore/mc cp -r SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr  uk1s3/idr0125/SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr

[will-moore]

Looks like I got the mc cp command wrong - added an extra directory level:
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0125/SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr/SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr

I must have run python add_downsampling.py SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr 8 twice, since each image has 2 extra resolutions.

[will-moore]

Repeat with a new Plate...

mkdir SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
aws s3 sync --no-sign-request s3://cellpainting-gallery/cpg0004-lincs/broad/images/2016_04_01_a549_48hr_batch1/images_zarr/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

time python add_downsampling.py SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr 8
...
real    40m16.263s
user    31m5.552s
sys     3m26.322s


/home/wmoore/mc cp -r SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr uk1s3/idr0125

Validate that this plate has a single 8-factor downsampling...
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

Views well in Vizarr:

Zoom-in gives good resolutions of thumbnails (plate view):

Single image looks good too (no issues with 8 x factor scaling):

[will-moore]

To test import and thumbnailing of this Plate above, I'll use released ZarrReader on idr0125-pilot (to avoid black-image and import-failure issues with ZarrReader performance improvements branch)

Copy local OMEZarrReader jar to idr0125-pilot...

rsync -rvP --progress ~/Downloads/OMEZarrReader-0.3.1-jar-with-dependencies.jar idr-pilot.openmicroscopy.org:/home/wmoore

ssh -A idr-pilot.openmicroscopy.org
rsync -rvP --progress OMEZarrReader-0.3.1-jar-with-dependencies.jar idr0125-omeroreadwrite:/home/wmoore

ssh idr0125-omeroreadwrite
sudo cp OMEZarrReader-0.3.1-jar-with-dependencies.jar OMEZarrReader.jar
sudo chown omero-server OMEZarrReader.jar
sudo cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
sudo cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/
sudo service omero-server restart

Going to try....

1. Use a tried-and-tested regular import of plate-without-chunks, then update symlinks to include chunks.
2. Also try import of METADATA.ome.xml and use mkngff to create Fileset from full Plate.

screen -S idr0125_aws_sync
cd idr0125
mkdir SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
aws s3 sync --no-sign-request --exclude '*' --include "*/.z*" --include "*.xml" --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3://idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

# copied to /ngff/idr0125 and chown to omero-server, then...

$ omero import --transfer=ln_s --depth=100 --name=SQ00014812__2016-05-23T20_44_31-Measurement1 --skip=all SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr --file /tmp/idr0125_20230914.log --errs /tmp/idr0125_20230914.err

...
2023-09-14 14:52:03,766 16610619   [l.Client-4] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 52235482
2023-09-14 14:52:03,791 16610644   [l.Client-5] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 52235482
2023-09-14 14:52:03,832 16610685   [l.Client-4] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 52235482
2023-09-14 14:52:03,911 16610764   [l.Client-5] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 52235482
2023-09-14 14:52:05,000 16611853   [l.Client-4] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /ngff/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/OME/METADATA.ome.xml
Other imported objects:
Fileset:5287574

==> Summary
14613 files uploaded, 1 fileset, 1 plate created, 3456 images imported, 0 errors in 4:36:34.647

http://localhost:1080/webclient/?show=plate-10601

On idr0125-pilot, mount s3 bucket...

sudo mkdir /idr0125 && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0125 /idr0125

$ ls /idr0125
SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr  SQ00015098__2016-06-08T18_43_42-Measurement1.ome.zarr

Create symlink

sudo -u omero-server -s
rm -rf /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-09/14/10-15-31.104/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr
ln -s /idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-7/2023-09/14/10-15-31.104/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr

This allowed chunks to be viewed and on "Save to all" thumbnails were generated quickly - about a minute for the whole plate:

[will-moore]

cell-paininting data now has 2nd, lower resolution, e.g. https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0125/SQ00014812__2016-05-23T20_44_31-Measurement1.ome.zarr/A/1/0/