Failing dark montior

[bhilbert4]

The dark monitor is failing after the recent shared tasks updates in #1262

There are warnings in the log file such as:
WARNING: Input data contains invalid values (NaNs or infs), which were automatically clipped.

and:
08/01/2023 02:19:41 AM INFO: Name of mean slope image: miri_mirim_full_59897.333649982626_to_60157.291934589884_mean_slope_image.png
08/01/2023 02:19:41 AM WARNING: Input data contains invalid values (NaNs or infs), which were automatically clipped.
08/01/2023 02:19:41 AM WARNING: BokehUserWarning: ColumnDataSource's columns must be of the same length. Current lengths: ('pixels_x', 130), ('pixels_y', 130), ('values', 76)

08/01/2023 02:19:42 AM CRITICAL: ERROR: Out of range float values are not JSON compliant
08/01/2023 02:19:42 AM CRITICAL: Traceback (most recent call last):
File "logging_functions.py", line 291, in wrapped
func(*args, **kwargs)
File "logging_functions.py", line 256, in wrapped
func(*args, **kwargs)
File "shared_tasks.py", line 169, in _caller
ret_value = run_func(*args, **kwargs)
File "dark_monitor.py", line 1040, in run
self.process(dark_files)
File "dark_monitor.py", line 841, in process
raise e
File "dark_monitor.py", line 819, in process
self.create_mean_slope_figure(slope_image, len(slope_files), hotxy=new_hot_pix, deadxy=new_dead_pix,
File "dark_monitor.py", line 353, in create_mean_slope_figure
export_png(self.plot, filename=output_filename)
File "bokeh/io/export.py", line 111, in export_png
image = get_screenshot_as_png(obj, width=width, height=height, driver=webdriver, timeout=timeout)
File "bokeh/io/export.py", line 232, in get_screenshot_as_png
html = get_layout_html(obj, resources=resources, width=width, height=height)
File "bokeh/io/export.py", line 313, in get_layout_html
return html()
ValueError: Out of range float values are not JSON compliant

I've confirmed that the mean slope image in this case contained pixels with values of NaN. It looks like this is a problem when trying to save a png version of the image.

We probably need to replace NaN values with something less offensive, like 0.

[bhilbert4]

I just tried creating a png using an array with multiple nan pixels in it. The png was created successfully and looks fine. The one thing I did not do was to overplot the bad pixels onto the image. So now I'm wondering if the error above about the columns of differing lengths is the real problem. Maybe the values for some of these bad pixels are NaNs and those are being ignored by bokeh?

[bhilbert4]

My test case suggests that the above is the problem. I don't see the warning about the column lengths, but when I provide a list of hot pixels to overplot, and any of those hot pixels have a value of NaN in the image, then I get the error same error as above: ValueError: Out of range float values are not JSON compliant

[bhilbert4]

This does beg the question: How can a pixel be determined to be hot, but have a value of NaN in the mean dark rate image? It might be worth digging deeper into the pipeline outputs to make sure there aren't any unexpected issues in the outputs.

And we should also fix the overplot_bad_pix() function so that any NaN pixels are given finite values before adding them to the legend.

[bhilbert4]

The discrepancy is coming from the newly-found dead pixels. 130 were found, but some of those have values of NaN in the mean dark rate image. Those NaN values were being ignored by the ColumnDataSource when trying to overplot the points onto the image. So there were 130 sets of coordinates, but only 76 pixel values because the other 54 pixels had a value of NaN.