Benchmarking MODE code - where is it slowest?

[Jeff-Duda]

My question is pretty succinctly stated in the title. Some background, though:

I frequently run MODE on high-resolution data sets (CAMs like HRRR and HREF members). The existing builds I have access to can only run MODE in serial, so that is a hearty task. MODE tends to take a long time to run on these datasets, especially when there are a lot of objects in the data (e.g., higher base rate). What I'm curious about is where most of the processing time in MODE is spent. I have a suspicion that a lot of time is spent checking the centroid distances between pairs of objects - even though there is a user-defined threshold telling MODE not to output stat lines for object pairs that are too far apart, I wonder if MODE is still spending time calculating metrics for said object pairs.

Anyway, any hints at the percentage of total processing time MODE spends on its broader submodules (i.e., I/O, classifying objects, object comparison, stat computation, error checking) would help. Thanks.

[jprestop]

Hi @Jeff-Duda. Thanks for your question and the background information.

Can you please tell us what version you are running? We're wondering if you're running 11.1 or a beta version of 12.0.

Our developers have worked on the MET issue Enhance MODE to use OpenMP to make the convolution step faster #2724 (added for 12.0.0-beta2), which should enable a quicker convolution step. If you are currently, using 11.1, we'd love to have you try using 12.0.0-beta6 and learn what, if any improvement, you see. To see the list of machines that beta6 (or really beta2 or great) is installed on, you can visit the Existing Builds section of our METplus User's Guide.

Additionally, setting max_centroid_dist well can make MODE run more efficiently by skipping unrealistic object comparisons.

I also wanted to make you aware of MET issue Review MODE implementation to look for more computational efficiencies #2727 describes more work in this area, so there is still room for improvement.

[Jeff-Duda]

Asking about v11.1.

I will look into the other links. Thanks.

[Jeff-Duda]

This looks very promising! I will begin using met12.0.0-beta6 on Jet/Hera to examine the impact of these changes.

[jprestop]

Thank you @Jeff-Duda. We look forward to hearing how it goes.