Protobuf support

[Sh4d1]

Hello! Is it currently possible to convert the data to protobuf before sending to Kafka ? Don't seem that way from the docs.
If not, is it planned?

Thanks!

[dosubot]

Hi @Sh4d1! I'm Dosu and I’m helping the sequin team.

Sequin currently only supports sending messages to Kafka in JSON format—Protobuf serialization is not available or configurable at this time, and there’s no indication in the docs or codebase that Protobuf support is planned or under development. You can see this explicitly mentioned in the Kafka sink documentation: Sequin sends messages to Kafka as JSON, not Protobuf, and provides a reference to the JSON message shape here. The Kafka sink implementation also defines messages as strings and does not reference Protobuf serialization here.

There isn’t any documented extension point or workaround for custom serialization like Protobuf within Sequin itself, so if you need Protobuf, you’d need to handle the transformation outside of Sequin.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[davoclavo]

Hello @Sh4d1, thanks a lot for your question. Currently only JSON is supported, but Protobuf and Avro have been requested multiple times so they definitely will be considered in the roadmap!

Do you have any current requirements that you could share? They would be helpful to know what to tackle first.

For instance, in order to know what schema to use for a given payload, which one is the option you would pick:

• Schema upload via Sequin configuration (UI or YAML)
• Schema registry connection settings (e.g. Confluent Schema Registry)
• Dynamically generated schema
• Some other alternative?

[Sh4d1]

Good to know! Right know our need would be pretty basic, so uploading a schéma, or through a schema registry cour work! Reflection might also be an idea here!
Ideally, including a wasm plugin might give the maximum flexibility and performance imho.

We're also looking at https://docs.redpanda.com/redpanda-connect/components/processors/protobuf/ which could give you some ideas as well!