libmamba Could not parse state file: Could not load cache state

[nick-youngblut]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

Installed miniconda:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

Installed mamba:

conda install mamba

Versions:

mamba 1.3.1
conda 23.1.0

When I create or update a conda env, I get the following:

warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null

conda/mamba seem to run correctly, but the warning is troubling.

Conda Info

active environment : base
    active env location : /home/nyoungblut-lab/miniconda3
            shell level : 1
       user config file : /home/nyoungblut-lab/.condarc
 populated config files : /home/nyoungblut-lab/.condarc
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.10.9.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.35=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /home/nyoungblut-lab/miniconda3  (writable)
      conda av data dir : /home/nyoungblut-lab/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://conda.anaconda.org/pytorch/linux-64
                          https://conda.anaconda.org/pytorch/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/qiime2/linux-64
                          https://conda.anaconda.org/qiime2/noarch
          package cache : /home/nyoungblut-lab/miniconda3/pkgs
                          /home/nyoungblut-lab/.conda/pkgs
       envs directories : /home/nyoungblut-lab/miniconda3/envs
                          /home/nyoungblut-lab/.conda/envs
               platform : linux-64
             user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.9 Linux/5.15.0-58-generic ubuntu/22.04.1 glibc/2.35
                UID:GID : 1003:1003
             netrc file : None
           offline mode : False

### Conda Config

```shell
channels:
  - conda-forge
  - bioconda
  - pytorch
  - defaults
  - qiime2

### Conda list

```shell
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
brotlipy                  0.7.0           py310h7f8727e_1002    defaults
bzip2                     1.0.8                h7b6447c_0    defaults
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h5eee18b_3    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
conda                     23.1.0          py310hff52083_0    conda-forge
conda-content-trust       0.1.3           py310h06a4308_0    defaults
conda-package-handling    2.0.2           py310h06a4308_0    defaults
conda-package-streaming   0.7.0           py310h06a4308_0    defaults
cryptography              38.0.4          py310h9ce1e76_0    defaults
fmt                       9.1.0                h924138e_0    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4             py310h06a4308_0    defaults
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
ld_impl_linux-64          2.38                 h1181459_1    defaults
libarchive                3.6.2                h3d51595_0    conda-forge
libcurl                   7.88.1               hdc1c0ab_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h6a678d5_6    defaults
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libmamba                  1.3.1                hcea66bb_1    conda-forge
libmambapy                1.3.1           py310h1428755_1    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsolv                   0.7.23               h3eb15da_0    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libssh2                   1.10.0               hf14f497_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libxml2                   2.10.3               h7463322_0    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
llvm-openmp               15.0.7               h0cdce71_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.3.1           py310h51d5547_1    conda-forge
ncurses                   6.4                  h6a678d5_0    defaults
openssl                   3.0.8                h0b41bf4_0    conda-forge
pip                       22.3.1          py310h06a4308_0    defaults
pluggy                    1.0.0           py310h06a4308_1    defaults
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.4           py310h5eee18b_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pyopenssl                 22.0.0             pyhd3eb1b0_0    defaults
pysocks                   1.7.1           py310h06a4308_0    defaults
python                    3.10.9          he550d4f_0_cpython    conda-forge
python_abi                3.10                    3_cp310    conda-forge
readline                  8.2                  h5eee18b_0    defaults
reproc                    14.2.4               h0b41bf4_0    conda-forge
reproc-cpp                14.2.4               hcb278e6_0    conda-forge
requests                  2.28.1          py310h06a4308_0    defaults
ruamel.yaml               0.17.21         py310h5eee18b_0    defaults
ruamel.yaml.clib          0.2.6           py310h5eee18b_1    defaults
setuptools                65.6.3          py310h06a4308_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
sqlite                    3.40.1               h5082296_0    defaults
tk                        8.6.12               h1ccaba5_0    defaults
toolz                     0.12.0          py310h06a4308_0    defaults
tqdm                      4.64.1          py310h06a4308_0    defaults
tzdata                    2022g                h04d1e81_0    defaults
urllib3                   1.26.14         py310h06a4308_0    defaults
wheel                     0.37.1             pyhd3eb1b0_0    defaults
xz                        5.2.10               h5eee18b_1    defaults
yaml-cpp                  0.7.0                h27087fc_2    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
zstandard                 0.18.0          py310h5eee18b_0    defaults
zstd                      1.5.2                h3eb15da_6    conda-forge

### Additional Context

OS: `Ubuntu 22.04.1`
Shell: `5.1.16(1)-release`

[wolfv]

Ouch that sounds like the new repodata state file is not created correctly. Did you run micromamba with repodata_use_zst: true before?

Otherwise we should probably properly align with conda (if conda is writing out state files nowadays).

[jaimergp]

I am also seeing some similar errors on Windows CI:

warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not remove state file "C:\\conda_bin\\envs\\conda-test-env\\pkgs\\cache\\59ba4880.state.json": The process cannot access the file because it is being used by another process.
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not remove state file "C:\\conda_bin\\envs\\conda-test-env\\pkgs\\cache\\3e39a7aa.state.json": The process cannot access the file because it is being used by another process.
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not remove state file "C:\\conda_bin\\envs\\conda-test-env\\pkgs\\cache\\920c960f.state.json": The process cannot access the file because it is being used by another process.
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not remove state file "C:\\conda_bin\\envs\\conda-test-env\\pkgs\\cache\\4ea078d6.state.json": The process cannot access the file because it is being used by another process.
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not remove state file "C:\\conda_bin\\envs\\conda-test-env\\pkgs\\cache\\5ca77eed.state.json": The process cannot access the file because it is being used by another process.
warning  libmamba Could not parse state file: Could not load cache state: [json.exception.type_error.302] type must be string, but is null
warning  libmamba Could not remove state file "C:\\conda_bin\\envs\\conda-test-env\\pkgs\\cache\\c4a505b4.state.json": The process cannot access the file because it is being used by another process.

@dholth - have you seen this?

[dholth]

23.1 produces a pre-CEP .state.json file, the format is probably different than mamba's.

[jaimergp]

You mean this CEP, right?

[dholth]

The conda 23.3 release produces a state.json that is much more similar to mamba's (the keys are no longer _underscore-prefixed)

{
  "url": "https://repo.anaconda.com/pkgs/main/osx-arm64",
  "etag": "W/\"7cba3a7265b9feb756f943cb057fea77\"",
  "mod": "Wed, 29 Mar 2023 10:01:09 GMT",
  "cache_control": "public, max-age=30",
  "mtime_ns": 1680097158248444702,
  "size": 9280134,
  "refresh_ns": 1680097158251096000
}

[jaimergp]

#65 alleviates this, so conda-libmamba-solver 23.3 will have better compliance. We have run some local tests and didn't see any state warnings.

Feel free to reopen if you see them again. Thanks!

[mharradon]

I continue to see this with updated versions on linux-64. I'm fairly certain my caches have been cleaned since these updates. I also get this mixed with output of the form Cache file "XXX.json" was modified by another program.

[dholth]

So conda is producing these error messages without other programs trying to share the cache? Are you using main/conda-forge only or are you also using anaconda.org channels? I assume you are using conda-libmamba-solver, is --experimental=jlap turned on or off?

[jaimergp]

Can you share your conda list -n base and conda config --show-sources, as well as a command that shows that output? Thanks!

[mharradon]

I see this issue in a docker-based recipe build (using mamba build). Solving steps for build and test environments across multiple recipes both produce these warnings consistently - previously Mamba was installed via miniconda, but I just updated to installing via the new direct script and I'm seeing similar results. We have a few other channels in our sources, mainly robostack, but most solves just produce conda-forge packages (and warnings are emitted prior).

The pkgs directory is stored via a Docker cache mount, but I don't expect that to cause issues, as clearing the cache mount does not remove the warnings. Maybe some issue with filesystem permissions, but I wouldn't expect it to error in this way. I can confirm the cache mount gets populated with expected packages.

Before building we run this in the base environment: mamba install patchelf conda-build conda-verify conda-libmamba-solver boa. Some of this may not be necessary anymore.

We do not call anything with the flag--experimental=jlap.

I'll follow up with some environment output details.

[dholth]

I see, there is a mix of mamba and conda. We have seen bugs where mamba's .state.json parsing looks for what it thinks is a mandatory field, which conda's .state.json can omit or set to null. The cache_control field can be null for example if the server doesn't send that header.

Is it possible to list the contents of ~/miniconda3/pkgs/cache/*.state.json or wherever conda is installed?

The upcoming conda does have code to enforce stringiness on these fields: https://github.com/conda/conda/blob/main/conda/gateways/repodata/__init__.py#L362

[jaimergp]

If you are using conda mambabuild then you are not importing conda-libmamba-solver. That subcommand is provided by mamba-org/boa.

You could try CONDA_SOLVER=libmamba conda build ... to make sure you are using conda-limamba-solver. Our support for conda-build is still in the works (check #194), though.

[mharradon]

Thank you all for the helpful feedback. I believe part of my previous report was incorrect - it seems that switching from conda mambabuild to mamba build managed to make the cache state warnings go away.

We have some large dev-facing env builds, so I like to have both the mamba solving and the package caching working reliably :)

Just discovering that the install instructions have updated is great - hoping to try out micromamba soon!