nwm_client Azure blob storage tests failing.

[jarq6c]

Looks like recent NWM data my no longer get pushed to Azure. Older data are still available, but currently it's unclear when the archive ends.

[aaraney]

Assuming these are the test failure you are referencing?

Failing Test Output

Github deletes actions runner output after I think 30 days? Below is the output for historical purposes:

============================= test session starts ==============================
platform linux -- Python 3.10.15, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/runner/work/hydrotools/hydrotools/python/nwm_client
configfile: pytest.ini
collected 26 items

python/nwm_client/tests/test_FileDownloader.py ...
python/nwm_client/tests/test_NWMClient.py .....
python/nwm_client/tests/test_NWMFileCatalog.py ...F
python/nwm_client/tests/test_NWMFileProcessor.py ...
python/nwm_client/tests/test_ParquetStore.py ........
python/nwm_client/tests/test_UnitHandler.py ...

=================================== FAILURES ===================================
____________________________ test_azure_list_blobs _____________________________

setup_azure = <hydrotools.nwm_client.AzureFileCatalog.AzureFileCatalog object at 0x7f85934a38e0>

    @pytest.mark.slow
    def test_azure_list_blobs(setup_azure):
        blobs = setup_azure.list_blobs(
            configuration="analysis_assim",
            reference_time=pd.Timestamp(reference_time)
        )
>       assert len(blobs) == 3
E       assert 0 == 3
E        +  where 0 = len([])

python/nwm_client/tests/test_NWMFileCatalog.py:59: AssertionError
=============================== warnings summary ===============================
../../../../../opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pydantic/_internal/_config.py:295
  /opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pydantic/_internal/_config.py:295: PydanticDeprecatedSince20: Support for class-based `config` is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.10/migration/
    warnings.warn(DEPRECATION_MESSAGE, DeprecationWarning)

tests/test_NWMClient.py::test_gcp_client
  <frozen importlib._bootstrap>:241: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED python/nwm_client/tests/test_NWMFileCatalog.py::test_azure_list_blobs - assert 0 == 3
 +  where 0 = len([])
================== 1 failed, 25 passed, 2 warnings in [13](https://github.com/NOAA-OWP/hydrotools/actions/runs/12545078249/job/35070711979#step:6:14).69s ===================

[aaraney]

It seems that the last uploads occurred on the 2024-12-22 (at least for the channel routing files).

az storage blob list \ 
  --account-name noaanwm \ 
  --container-name nwm \ 
  --prefix nwm.20241222 \
  --output table

For others trying to reproduce this, using the azure docker image may be helpful: docker pull mcr.microsoft.com/azure-cli.

[jarq6c]

Looks like test_azure_blobs is failing for the second week. I'm leaning towards hard coding the last known good date and adding a note to the README and docs.

[jarq6c]

All weekly slow tests are passing again, including AzureFileCatalog. However, it does not appear that missing data were back-filled.

hydrotools.nwm_client.NWMClient.QueryError: No data found for configuration 'analysis_assim' and reference time '2025-01-01 01:00:00+00:00'

[jarq6c]

The package does raise a QueryError as designed, so I'm not sure there's much to be done.