fix: move DB setup to happen after Connect.init #1533

edgurgel · 2025-09-15T02:42:40Z

What kind of change does this PR introduce?

This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition.

I've also removed StartCounters as it's not needed anymore. We start the RateCounters where they are handled.

What is the current behavior?

Connect.init/1 can take minutes blocking other Connect processes from starting up if they share the same partition supervisor.

What is the new behavior?

Change Connect.init/1 to be quick and delay connecting to handle_continue

Additional context

Add any other context or screenshots.

vercel · 2025-09-15T02:42:45Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Preview	Comments	Updated (UTC)
realtime-demo	Ignored	Preview		Sep 15, 2025 2:58am

This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition

edgurgel · 2025-09-15T03:44:38Z

lib/realtime/tenants/connect/check_connection.ex

-    case Database.check_tenant_connection(tenant) do
+    case Realtime.Database.check_tenant_connection(tenant) do
      {:ok, conn} ->
-        Process.link(conn)


No need to link as Database.check_tenant_connection calls Postgrex.start_link which links the process.

edgurgel · 2025-09-15T04:37:37Z

lib/realtime/tenants/connect/check_connection.ex

+    case Realtime.Database.check_tenant_connection(tenant) do
      {:ok, conn} ->
-        Process.link(conn)
        db_conn_reference = Process.monitor(conn)


In fact this Process.monitor is also not doing much given that they are linked. Most likely won't have time to react to the DOWN message as the linked process will crash Connect

filipecabaco

does this mean that , for some amount of time, users will be able to start connecting and then we will disconnect them?

part of the reason we had this to be more blockng / sync was to actually be sure that the database was fully ready and up to move forward with accepting socket connections so I'm wondering what will be the side effects of having this more async vs just having more partitions

edgurgel · 2025-09-15T19:06:39Z

does this mean that , for some amount of time, users will be able to start connecting and then we will disconnect them?
part of the reason we had this to be more blockng / sync was to actually be sure that the database was fully ready and up to move forward with accepting socket connections so I'm wondering what will be the side effects of having this more async vs just having more partitions

We only return a conn when syn has the conn metadata which is after the DB connection is set-up. This is how it works today already and this PR has not changed this.

This PR just also changes so that we also capture the shutdown while connecting. We won't block on the Supervisor call anymore but we will block waiting for the syn message that the conn is ready (which we already do today).

Previously if we had 5 websockets from different nodes calling Connect.lookup_or..:

The first one would block calling Supervisor.start_child
The other 4 websockets would block waiting for the syn message saying that conn metadata has been set.

This PR changes so that all 5 websockets wait on syn

`

filipecabaco · 2025-09-15T21:20:31Z

makes sense ship it 👍

kiwicopple · 2025-09-15T21:43:02Z

🎉 This PR is included in version 2.48.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

* fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <[email protected]> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Bradley Haljendi <[email protected]>

@h0lybyte

* 🔄 Sync with upstream changes (#2) * chore: fix couple of flaky tests (supabase#1517) * fix: Improve runtime setup logic (supabase#1511) Cleanup runtime.exs logic to be more organized and easier to mantain * fix: runtime setup error (supabase#1520) --------- Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Filipe Cabaço <[email protected]> * 🔄 Sync with upstream changes (#4) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) --------- Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Bradley Haljendi <[email protected]> * 🔄 Sync with upstream changes (#6) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) --------- Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> * 🔄 Sync with upstream changes (#7) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <[email protected]> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Bradley Haljendi <[email protected]> --------- Co-authored-by: Al @h0lybyte <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Bradley Haljendi <[email protected]>

* fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <[email protected]> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Bradley Haljendi <[email protected]>

@h0lybyte

* 🔄 Sync with upstream changes (#2) * chore: fix couple of flaky tests (supabase#1517) * fix: Improve runtime setup logic (supabase#1511) Cleanup runtime.exs logic to be more organized and easier to mantain * fix: runtime setup error (supabase#1520) --------- Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Filipe Cabaço <[email protected]> * 🔄 Sync with upstream changes (#4) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) --------- Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Bradley Haljendi <[email protected]> * 🔄 Sync with upstream changes (#6) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) --------- Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> * 🔄 Sync with upstream changes (#7) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <[email protected]> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Bradley Haljendi <[email protected]> * 🔄 Sync with upstream changes (#9) * fix: runtime setup error (supabase#1520) * fix: use primary instead of replica on rename_settings_field (supabase#1521) * feat: upgrade cowboy & ranch (supabase#1523) * fix: Fix GenRpc to not try to connect to nodes that are not alive (supabase#1525) * fix: enable presence on track message (supabase#1527) currently the user would need to have enabled from the beginning of the channel. this will enable users to enable presence later in the flow by sending a track message which will enable presence messages for them * fix: set cowboy active_n=100 as cowboy 2.12.0 (supabase#1530) cowboy 2.13.0 set the default active_n=1 * fix: provide error_code metadata on RealtimeChannel.Logging (supabase#1531) * feat: disable UTF8 validation on websocket frames (supabase#1532) Currently all text frames as handled only with JSON which already requires UTF-8 * fix: move DB setup to happen after Connect.init (supabase#1533) This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition * fix: handle wal bloat (supabase#1528) Verify that replication connection is able to reconnect when faced with WAL bloat issues * feat: replay realtime.messages (supabase#1526) A new index was created on inserted_at DESC, topic WHERE private IS TRUE AND extension = "broadast" The hardcoded limit is 25 for now. * feat: gen_rpc pub sub adapter (supabase#1529) Add a PubSub adapter that uses gen_rpc to send messages to other nodes. It uses :gen_rpc.abcast/3 instead of :erlang.send/2 The adapter works very similarly to the PG2 adapter. It consists of multiple workers that forward to the local node using PubSub.local_broadcast. The way to choose the worker to be used is based on the sending process just like PG2 adapter does The number of workers is controlled by `:pool_size` or `:broadcast_pool_size`. This distinction exists because Phoenix.PubSub uses `:pool_size` to define how many partitions the PubSub registry will use. It's possible to control them separately by using `:broadcast_pool_size` * fix: ensure message id doesn't raise on non-map payloads (supabase#1534) * fix: match error on Connect (supabase#1536) --------- Co-authored-by: Eduardo Gurgel Pinho <[email protected]> * feat: websocket max heap size configuration (supabase#1538) * fix: set max process heap size to 500MB instead of 8GB * feat: set websocket transport max heap size WEBSOCKET_MAX_HEAP_SIZE can be used to configure it * fix: update gen_rpc to fix gen_rpc_dispatcher issues (supabase#1537) Issues: * Single gen_rpc_dispatcher that can be a bottleneck if the connecting takes some time * Many calls can land on the dispatcher but the node might be gone already. If we don't validate the node it might keep trying to connect until it times out instead of quickly giving up due to not being an actively connected node. * fix: improve ErlSysMon logging for processes (supabase#1540) Include initial_call, ancestors, registered_name, message_queue_len and total_heap_size Also bump long_schedule and long_gc * fix: make pubsub adapter configurable (supabase#1539) --------- Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Bradley Haljendi <[email protected]> --------- Co-authored-by: Al @h0lybyte <[email protected]> Co-authored-by: Eduardo Gurgel <[email protected]> Co-authored-by: Filipe Cabaço <[email protected]> Co-authored-by: Bradley Haljendi <[email protected]>

fix: move DB setup to happen after Connect.init

98ba205

This change reduces the impact of slow DB setup impacting other tenants trying to connect at the same time that landed on the same partition

edgurgel force-pushed the fix/connect-init branch from e2f2a58 to 98ba205 Compare September 15, 2025 02:58

edgurgel commented Sep 15, 2025

View reviewed changes

filipecabaco reviewed Sep 15, 2025

View reviewed changes

filipecabaco approved these changes Sep 15, 2025

View reviewed changes

edgurgel merged commit 70339c7 into main Sep 15, 2025
5 checks passed

edgurgel deleted the fix/connect-init branch September 15, 2025 21:42

kiwicopple added the released label Sep 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix: move DB setup to happen after Connect.init #1533

fix: move DB setup to happen after Connect.init #1533

Uh oh!

edgurgel commented Sep 15, 2025

Uh oh!

vercel bot commented Sep 15, 2025 •

edited

Loading

Uh oh!

edgurgel Sep 15, 2025 •

edited

Loading

Uh oh!

edgurgel Sep 15, 2025

Uh oh!

filipecabaco left a comment

Uh oh!

edgurgel commented Sep 15, 2025 •

edited

Loading

Uh oh!

filipecabaco commented Sep 15, 2025

Uh oh!

Uh oh!

kiwicopple commented Sep 15, 2025

Uh oh!

Uh oh!

Uh oh!

fix: move DB setup to happen after Connect.init #1533

fix: move DB setup to happen after Connect.init #1533

Uh oh!

Conversation

edgurgel commented Sep 15, 2025

What kind of change does this PR introduce?

What is the current behavior?

What is the new behavior?

Additional context

Uh oh!

vercel bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edgurgel Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edgurgel Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

filipecabaco left a comment

Choose a reason for hiding this comment

Uh oh!

edgurgel commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

filipecabaco commented Sep 15, 2025

Uh oh!

Uh oh!

kiwicopple commented Sep 15, 2025

Uh oh!

Uh oh!

vercel bot commented Sep 15, 2025 •

edited

Loading

edgurgel Sep 15, 2025 •

edited

Loading

edgurgel commented Sep 15, 2025 •

edited

Loading