Unable to use MONAILabel in Digital Slide Archive (DSA) (pathology)

[M1n9X]

I'm trying to setup MONAILabel in DSA.

What i have done

• Setup DSA
  by following the official docs under https://github.com/DigitalSlideArchive/digital_slide_archive/tree/master/devops/dsa i have successfully built DSA docker and DSA is now accessible under http://10.100.37.100:8080/
  i did not modify the original DSA docker-compose file.
• Setup MONAILabel server using following command
  monailabel start_server --app apps/pathology --studies http://10.100.37.100:8080/api/v1
  now MONAILabel api is available (port mapping in docker) under http://10.100.37.100:38000/
• Import MONAILabel plugin docker image for DSA
  in DSA Admin console -> Slicer CLI Web -> Import Docker Image CLI Tasks, i used image projectmonai/monailabel-dsa:latest
  now i can also access MONAILabel tool in DSA, and i have configured the MONAILabel Address to http://10.100.37.100:38000/

What actually happened

Unlike what i did with QuPath, when i draw a rectangle with Analysis ROI tool and press Submit button or the MONAILabelAnnotation under Analyses drop-down menu, nothing happend.
With further check, i found no response from MONAILabel server (should have printed some logs in console), so i suspect that communication between DSA and MONAILabel server is still not available.

Any suggestion?

Did I miss any step or make wrong configurations? I would appreciate it if more detailed installation info could be provided. Thanks.

[diazandr3s]

Hi @M1n9X,

Thanks for the detailed discussion here.
It happens sometimes with DSA. For some reason, jobs started even before you submit the ROI to the MONAI Label server. I'd recommend you cancel previous jobs in the jobs tab. Just leave running the latest one.

In this video I show how you can do that:

dsa-annotation.mp4

This is something we need to improve. We need to make sure previous empty jobs don't start on DSA.

Hope this helps,

[M1n9X]

Thanks for your quick reply.
I tried with no luck. 😢

The issue i came across is that even i cancelled previous job (as you described in the tutorial video), the remaining job was still not in RUNNING status, shwoing INACTIVE instead. Besides, there are no logs.

I did not notice obvious differences according to your video, and i even installed the dsarchive/histomicstk:latest docker image, which has been successfully imported. I also tried inside docker working environment in case port mapping matters.

Anything else i can do to find more debug info?

[diazandr3s]

Hi @M1n9X,

Thanks for the follow-up.

I've faced a similar issue before. I restarted both DSA docker and MONAI Label server to kill previous jobs.

Can you please try that and let us know how that goes?

@manthey, any thought as to why this is happening on DSA?

[M1n9X]

Hi @diazandr3s

I tried to restart both DSA docker and MONAI Label server, but it still did not work.
When starting DSA docker i noticed the following console log

girder_1 | 172.16.17.164 - - [18/Jul/2022:03:02:52] "GET /api/v1/folder?parentId=62cfabf4cb7ee21f491548d8&parentType=collection&offset=0&limit=50 HTTP/1.1" 200 463 "" "python-requests/2.27.1"
girder_1 | 172.16.17.164 - - [18/Jul/2022:03:02:52] "GET /api/v1/item?folderId=62cfabf4cb7ee21f491548d9&limit=0 HTTP/1.1" 200 473 "" "python-requests/2.27.1"
worker_1 | The user root' is already a member of root'.
worker_1 | The user root' is already a member of tox'.
worker_1 | Running a worker with superuser privileges when the
worker_1 | worker accepts messages serialized with pickle is a very bad idea!
worker_1 |
worker_1 | If you really want to continue then you have to set the C_FORCE_ROOT
worker_1 | environment variable (but please think about this before you do).
worker_1 |
worker_1 | User information: uid=0 euid=0 gid=0 egid=0
worker_1 |
dsa_worker_1 exited with code 1
mongodb_1 | {"t":{"$date":"2022-07-18T03:03:10.698+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"JournalFlusher","msg":"WiredTiger message","attr":{"message":"[1658113390:698837][1:0x7fc7fd575700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 309, snapshot max: 309 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 13837"}}
mongodb_1 | {"t":{"$date":"2022-07-18T03:03:10.917+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1658113390:917478][1:0x7fc7fcd74700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 311, snapshot max: 311 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 13837"}}
worker_1 | The user root' is already a member of root'.
worker_1 | The user root' is already a member of tox'.
worker_1 | Running a worker with superuser privileges when the
worker_1 | worker accepts messages serialized with pickle is a very bad idea!
worker_1 |
worker_1 | If you really want to continue then you have to set the C_FORCE_ROOT
worker_1 | environment variable (but please think about this before you do).
worker_1 |
worker_1 | User information: uid=0 euid=0 gid=0 egid=0
worker_1 |
dsa_worker_1 exited with code 1
mongodb_1 | {"t":{"$date":"2022-07-18T03:04:10.920+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1658113450:920799][1:0x7fc7fcd74700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 314, snapshot max: 314 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 13837"}}

Note that dsa_worker_1 exited with code 1, i used root account inside docker, i think this might matter.
What i have further tried is using the folloing command to start dsa service
C_FORCE_ROOT=True DSA_USER=$(id -u):$(id -g) docker-compose up
But it was still not working.

Hope the above info can help you guys for further investigation.

[SachidanandAlle]

cd plugins/dsa
docker build -t projectmonai/monailabel-dsa:latest .

Try to build the docker locally (just in case if the dockerhub version is outdated and needs to be refreshed)

Before that, instead of running MONAILabel CLI Task.. try to run Nuclei Detection.. and see if the job is able to schedule and run properly.. if that is also not working.. then I suggest to check with DSA support team on their github..

[M1n9X]

Hi, @SachidanandAlle, thanks for the reply.

I tried to build the image in Windows, but failed to import the docker image into DSA.

Reimport the official image in dockerhub is ok. FYI.

[SachidanandAlle]

DSA.. we recommend to try on ubuntu.. so far we never tried/tested on windows for DSA... so it's kind of not guaranteed..
please try on ubuntu

[M1n9X]

DSA is in fact running in Ubuntu, as you have mentioned, it may not work in Windows/WSL, and it indeed not, which i have already tested.

Because of some company policy, i can not connect to external network in Ubuntu server. What we usually do is, first build docker images in Windows/WSL and then transfer and load them in Linux. Most time it's working fine.

But just as you suggested, this could still be a reason. I will try to build the docker image later in a clean linux environment.