Cbor2 serialization for function calling [SDK-584]

[freider]

Adds opt-in support for cbor as a payload serialization format instead of pickle.

• If the input format is cbor, the corresponding output will also be encoded as such by the runtime
• Requires cbor2 to be installed in the container environment to work. This happens automatically for image builder 2025.06 and newer (if users absolutely need to use older image builder they can manually add cbor2 as a dependency to their image to make this work)
• The python SDK only uses cbor for input serialization when the MODAL_PAYLOAD_FORMAT=cbor env var is set
• libmodal beta is intended to remove the pickle serde for function calling altogether, relying on remote functions being deployed with cbor support to be called
• Functions can be optionally marked as restricted to cbor output (_experimental_restrict_output=True for now). This makes output cbor regardless if the input is pickle (good for security in the receiver, or if you know the consumer will be on libmodal even if the input creator is in Python using .spawn)

TODO

• Respect supported_input_formats in the client by sending cbor if pickle isn't supported by the Function definition. There is currently no syntactic way to restrict the input to a function when deploying it, but if we were to add this in the future it's good if the client respects it already for forwards-compatibility.
• Add _experimental_restrict_output to either the @app.cls or @modal.method decorators + add test that it's respected

Not in this patch (yet?):

• Tracebacks and python representations of exceptions are only provided for pickle calls - custom serialization of tracebacks could be a future improvement. libmodal only looks at the exception string repr of errors anyways at the moment. Will not do for this initial version - opting to leave these fields empty in the response payload for cbor inputs.
• We don't enforce client-side that a function call that's performed using cbor input requires cbor output. For untrusted functions it would be good to enforce this in order to not allow "unpickle attacks" by the functions. This is hard since FunctionCall.from_id() exists so retrieval doesn't necessarily "know" what output format to expect. Better to implement this on the backend and reject outputs that don't conform

---

Compatibility checklist

Check these boxes or delete any item (or this section) if not relevant for this PR.

• Client+Server: this change is compatible with old servers
• Client forward compatibility: this change ensures client can accept data intended for later versions of itself

Note on protobuf: protobuf message changes in one place may have impact to
multiple entities (client, server, worker, database). See points above.

---

Changelog

• [Internal] Adds support for calling into deployed functions using a new cbor based serialization format used by beta versions of libmodal-ts and libmodal-go

[ehdr]

Look good to me! Just minor comments on implementation, though also pending the larger discussion we started yesterday I guess.

[ehdr]

Debugging leftovers?

[freider]

Yup

[freider]

TODO @freider : Add test for cbor generators