generate_gantt_chart fails to convert strings to datetime objects

[hjmjohnson]

Summary

The log file for resource monitoring stores the date-time object as a string.

These objects result in errors like the following:

    generate_gantt_chart(callback_log_path, cores=n_procs)
  File "/home/johnsonhj/.venv/py38botimage/lib/python3.8/site-packages/nipype/utils/draw_gantt_chart.py", line 518, in generate_gantt_chart
    duration = (last_node["finish"] - start_node["start"]).total_seconds()
TypeError: unsupported operand type(s) for -: 'str' and 'str'

Actual behavior

No generated gantt chart and python crashes.

Expected behavior

Gantt chart generated with date strings converted back to date objects

document

How to replicate the behavior

run with the nipype.utils.draw_gantt_chart function passing the following json log file objects in :

{"name": "Coarse_P", "id": "Coarse_Segment_Prostate", "start": "2022-10-26T22:56:01.788467", "finish": "2022-10-26T22:56:06.238882", "duration": 4.450415, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "NormA_Intensities", "id": "Normalize_ADC_Intensities", "start": "2022-10-26T22:56:06.455242", "finish": "2022-10-26T22:56:06.493943", "duration": 0.038701, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "w_Crop_to_Mask", "id": "T2w_Crop_to_Mask", "start": "2022-10-26T22:56:06.772535", "finish": "2022-10-26T22:56:06.873257", "duration": 0.100722, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "cal_ROI", "id": "Create_Spherical_ROI", "start": "2022-10-26T22:56:07.018926", "finish": "2022-10-26T22:56:07.038770", "duration": 0.019844, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "_Glasd", "id": "Segment_Prostate_and_Central_Gland", "start": "2022-10-26T22:56:07.184228", "finish": "2022-10-26T22:56:11.056042", "duration": 3.871814, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "Regies", "id": "Register_Volumes", "start": "2022-10-26T22:56:11.431874", "finish": "2022-10-26T22:57:44.150974", "duration": 92.7191, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "NormTIntensities", "id": "Normalize_T2w_Intensities", "start": "2022-10-26T22:57:44.316520", "finish": "2022-10-26T22:57:44.841228", "duration": 0.524708, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "NorwFoV", "id": "Normalize_T2w_FoV", "start": "2022-10-26T22:57:45.083709", "finish": "2022-10-26T22:57:45.315151", "duration": 0.231442, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "AMatchTFoV", "id": "ADC_Match_T2w_FoV", "start": "2022-10-26T22:57:47.106713", "finish": "2022-10-26T22:57:47.392528", "duration": 0.285815, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "bMT2FoV", "id": "bVal_Match_T2w_FoV", "start": "2022-10-26T22:57:47.523598", "finish": "2022-10-26T22:57:47.794880", "duration": 0.271282, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "Match2FoV", "id": "Prostate_Match_T2w_FoV", "start": "2022-10-26T22:57:47.922329", "finish": "2022-10-26T22:57:47.993250", "duration": 0.070921, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "nd_Match_TFoV", "id": "Central_Gland_Match_T2w_FoV", "start": "2022-10-26T22:57:48.120202", "finish": "2022-10-26T22:57:48.187301", "duration": 0.067099, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "lize_l_Intensities", "id": "Normalize_bVal_Intensities", "start": "2022-10-26T22:57:49.113437", "finish": "2022-10-26T22:57:49.646035", "duration": 0.532598, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "te_tate_Mask", "id": "Update_Prostate_Mask", "start": "2022-10-26T22:57:49.773739", "finish": "2022-10-26T22:57:49.836341", "duration": 0.062602, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "eatures", "id": "Calculate_Features", "start": "2022-10-26T22:57:51.111196", "finish": "2022-10-26T22:58:10.643069", "duration": 19.531873, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "fier", "id": "Classifier", "start": "2022-10-26T22:58:10.973538", "finish": "2022-10-26T22:58:35.878028", "duration": 24.90449, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "Color_Map", "id": "Create_Color_Map", "start": "2022-10-26T22:58:36.206685", "finish": "2022-10-26T22:58:36.544164", "duration": 0.337479, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "Overlay", "id": "Create_Overlay", "start": "2022-10-26T22:58:38.252586", "finish": "2022-10-26T22:58:39.006377", "duration": 0.753791, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "PDF", "id": "Generate_PDF_Report", "start": "2022-10-26T22:58:39.174119", "finish": "2022-10-26T22:59:08.496506", "duration": 29.322387, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "DICOMWrite", "id": "DICOM_Write", "start": "2022-10-26T22:59:08.821098", "finish": "2022-10-26T22:59:09.013734", "duration": 0.192636, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "DICOMPDF", "id": "DICOM_Encapsulate_PDF", "start": "2022-10-26T22:59:09.145558", "finish": "2022-10-26T22:59:09.252677", "duration": 0.107119, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}
{"name": "WriteOutput", "id": "Write_Output", "start": "2022-10-26T22:59:10.900479", "finish": "2022-10-26T22:59:10.905279", "duration": 0.0048, "runtime_threads": "N/A", "runtime_memory_gb": "N/A", "estimated_memory_gb": 0.2, "num_threads": 1}

Execution environment

• My python environment outside container.

[hjmjohnson]

Related to #3528

[hjmjohnson]

Followup: Note that

"runtime_threads": "N/A", "runtime_memory_gb": "N/A"

in all cases. I do have psutil installed. and this is running on a linux computer (Ubuntu 20.04). Not sure what is going on.

[hjmjohnson]

It appears that the function 'get_max_resources_used' is not used anywhere. Perhaps that is the crux of the N/A problem.

[hjmjohnson]

@oesteban Let me know if I can assist with fixing this. I think the initial bug problem of converting the json strings back to date objects is pretty straight forward.

The larger issue is that all the probes for threads and memory use are coming back as N/A. I don't know where to track that bug down.