Cannot update project if source copier.yml contains {{ _copier_conf|to_json }}

[anjos]

Describe the problem

Running copier update --trust --defaults --vcs-ref=HEAD new-foo will lead to:

Traceback (most recent call last):
  File "/Users/andre/.local/share/pixi/envs/python/bin/copier", line 10, in <module>
    sys.exit(CopierApp.run())
             ~~~~~~~~~~~~~^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/plumbum/cli/application.py", line 640, in run
    inst, retcode = subapp.run(argv, exit=False)
                    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/plumbum/cli/application.py", line 635, in run
    retcode = inst.main(*tailargs)
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/cli.py", line 425, in main
    return _handle_exceptions(inner)
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/cli.py", line 70, in _handle_exceptions
    method()
    ~~~~~~^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/cli.py", line 415, in inner
    with self._worker(
         ~~~~~~~~~~~~^
        dst_path=destination_path,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
        overwrite=True,
        ^^^^^^^^^^^^^^^
    ) as worker:
    ^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/main.py", line 237, in __exit__
    raise value
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/cli.py", line 423, in inner
    worker.run_update()
    ~~~~~~~~~~~~~~~~~^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/main.py", line 914, in run_update
    self._apply_update()
    ~~~~~~~~~~~~~~~~~~^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/main.py", line 1003, in _apply_update
    with replace(
         ~~~~~~~^
        self,
        ^^^^^
    ...<5 lines>...
        exclude=exclude_plus_removed,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as new_worker:
    ^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/main.py", line 237, in __exit__
    raise value
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/main.py", line 1012, in _apply_update
    new_worker.run_copy()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/main.py", line 847, in run_copy
    self._execute_tasks(self.template.tasks)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/main.py", line 311, in _execute_tasks
    self._render_string(str(part), extra_context) for part in task_cmd
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/copier/main.py", line 768, in _render_string
    return tpl.render(**self._render_context(), **(extra_context or {}))
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/jinja2/environment.py", line 1295, in render
    self.environment.handle_exception()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/jinja2/environment.py", line 942, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/site-packages/jinja2_ansible_filters/core_filters.py", line 103, in to_json
    return json.dumps(a, *args, **kw)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ~~~~~~^^^^^
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/andre/.local/share/pixi/envs/python/lib/python3.13/json/encoder.py", line 261, in iterencode
    return _iterencode(o, 0)
pydantic_core._pydantic_core.PydanticSerializationError: Unable to serialize unknown type: <class 'function'>

Template

_min_copier_version: '9'

_tasks:
  - ['{{ _copier_python }}', '{{ _copier_conf.src_path }}/tasks.py', '{{ _copier_conf|to_json }}']

project_name:
  type: str
  default: new-foo
  help: A short (slug-style) name for your project

To Reproduce

1. Create a minimal source project called foo with a template containing {{ _copier_conf|to_json }}, like the one above. Initialise the git repo and commit all changes. I also added a tasks.py file that just echoes the input to screen.
2. Run copier copy --trust --defaults $(PWD)/foo --vcs-ref=HEAD new-foo to create a copy called new-foo. This copy should contain .copier-answers.yml (with the project-name and control variables for the source git repo), and a copy of tasks.py (since I did not ignore it).
3. Initialise a git repo on new-foo and commit the files, so that the update will work next.
4. Run copier update --trust --defaults --vcs-ref=HEAD new-foo to reproduce the traceback above.

After exploring the source code, the reason for this relates to:

a. The combined property of objects of type AnswerMap on user_data.py (line 98 on copier v9.4.1) contains references to two functions coming from the DEFAULT_DATA global variable on that module (now and make_secret):

DEFAULT_DATA: AnyByStrDict = {
    "now": _now,
    "make_secret": _make_secret,
}

b. On line 1012 of main.py (copier v9.4.1, method _apply_update()), a run_copy() is called with data = self.answers.combined (which contains the functions above aside from all other stuff already present on _copier_conf)

c. As a consequence, Jinja2 to_json converter will choke on trying to convert _copier_conf to JSON (as pydantic's to_jsonable_python will not filter those).

Logs

No response

Expected behavior

I'd expect that the update runs smoothly.

Screenshots/screencasts/logs

No response

Operating system

macOS

Operating system distribution and version

macOS 15

Copier version

9.4.1

Python version

3.13

Installation method

local build

Additional context

No response

[pawamoy]

Related:

• json dump of _copier_conf not working #1481
• feat: Make the to_json filter support the conf object #508

[pawamoy]

With Pydantic v1, we used pydantic_encoder, which seems to simply ignore unsupported types:

https://github.com/pydantic/pydantic/blob/4e055d56471651bca5ce8907c671bc6339d309fe/pydantic/v1/json.py#L84-L87

With Pydantic v2, we now use to_jsonable_python with default arguments. Setting serialize_unknown=True could resolve this issue:

serialize_unknown: Attempt to serialize unknown types, str(value) will be used, if that fails "<Unserializable {value_type} object>" will be used.

I'm not sure why this issue is not caught by our test suite though.

[anjos]

Could this be related to the version of pydantic I have on my environment? For completeness, here they are (not sure which of the 2 is applicable):

pydantic                     2.10.4          pyh3cfb1c2_0        289.6 KiB
pydantic-core                2.27.2          py313hdde674f_0     1.5 MiB

[anjos]

For completeness, and those that have a similar problem, you can get around the issue with the following (extremely complex) template construct, which removes the faulty entries from config:

  - ['{{ _copier_python }}', '{{ _copier_conf.src_path }}/tasks.py', '{{ dict(_copier_conf.items() | rejectattr(0, "in", ["data", "answers"]) | list + [("answers", dict(_copier_conf.answers.items() | rejectattr(0, "equalto", "init")))]) | to_json }}']

[anjos]

A naïve question: why do you need to add these 2 entries/functions on the configuration object?

[pawamoy]

Backward compatibility 🙂 To be honest I think we could have removed them already but we probably forgot. I would say it's time, but this will need a major bump.