feat(kafka_native): new module

[strowk]

What does this PR do?

Adding new module: kafka_native, that should be a much faster alternative to existing kafka module based on confluent kafka.

Why is it important?

Put simply Kafka Native is blazingly fast. It is based on GraalVM in native mode, which compiles Java code into native binary. This translates into very fast starting time and starting time is what always important in tests (unlike in runtime) and what is normally very slow for Java. In my relatively simple tests on my relatively powerful machine, the improvement is from 16.6 seconds to 3.6 seconds. Translate this into say 100 tests and the improvement can be measured in minutes if not hours for big projects.

This change also can fix this issue due to changed way of starting the process: #2206

Related issues

• Relates [Enhancement]: Kafka Module: graceful shutdown #2206

How to test this PR

Explain here how this PR will be tested by the reviewer: commands, dependencies, steps, etc.

cd modules/kafka_native && go test -timeout 30s -count 1 -run '^TestKafka$' github.com/testcontainers/testcontainers-go/modules/kafka_native ; cd -

Additional Context

There is also another PR, that attempts to add support for Kafka Native into pre-existing module: #2683

Here is a brief list of differences in approaches:

• this PR does not modify existing module to avoid potentially breaking backwards compatibility for users depending on details of confluent implementation
• this PR does not attempt to add SASL support and instead only focuses on using Kafka Native
• this PR does not remove starting script (advertising needs to detect correct hostname, at least for docker in Windows) and instead just adds "exec" before starting Kafka Native process, this should help with related issue linked above because exec would replace process instead of starting a new one and piping streams, so signals would go to the right process properly

[netlify]

✅ Deploy Preview for testcontainers-go ready!

Name	Link
🔨 Latest commit	1ef2f84
🔍 Latest deploy log	https://app.netlify.com/projects/testcontainers-go/deploys/688ee41d63b8fb000875afdc
😎 Deploy Preview	https://deploy-preview-3249--testcontainers-go.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[strowk]

For comparison: on my machine currently, TestKafka in kafka_native runs in 3.6 seconds, while original module test runs in 16.6 seconds. On big codebases and weaker machines this could translate in significant speed up!

$ cd modules/kafka_native && go test -timeout 30s -count 1 -run '^TestKafka$' github.com/testcontainers/testcontainers-go/modules/kafka_native ; cd -
ok      github.com/testcontainers/testcontainers-go/modules/kafka_native        3.693s

$ cd modules/kafka && go test -timeout 30s -count 1 -run '^TestKafka$' github.com/testcontainers/testcontainers-go/modules/kafka ; cd -
ok      github.com/testcontainers/testcontainers-go/modules/kafka       16.654s

Tests themselves are identical, only different images and less confluent-dependant configs.

[mdelapenya]

Hi @strowk thanks for adding this! I'm currently taking a look, also discussing internally with the other TC maintainers if this code should live as a new sub-package under the kafka module. What are your thoughts on this?

[strowk]

Sounds ok, though not sure what would interface look like

[mdelapenya]

Hmmm thinking more in depth about the submodule, will the existing kafka options apply to this native image? If so, then we should probably find them, although it can be probably done in a follow-up.