Runners stuck in restart loop as host machine cannot SSH into VM

[kengfai]

What happened?

No issue until latest macOS 15.2 update. No configuration change on VM or tartlet or Github. Thereafter, the log file in ~/Library/Application Support says it cannot SSH into the VM. Checked VM config and Remote Login is enabled.

What are the steps to reproduce?

1: Run Tartelet
2: VM reboots in endless loop
3. No fix

What is the expected behavior?

I expect that VM would run with Tartlet. To note only macOS updated in host machine. macOS on VM not updated. Was running fine previously until now.

[MikkelIJ]

I have the same issue on M2 Pro version 15.2

[tlow92]

Same issue here. Problem seems to be this Failed obtaining IP address of virtual machine named sequoia-15.4-1: The operation couldn’t be completed. (Swift.CancellationError error 1.)

Might be related cirruslabs/tart#998

[annuh]

I'm having the same issue, after a MacOS update.

Here are the errors from the debug log:

13:49:59.733 INFO: Start virtual machine named runner-1
13:50:09.781 INFO: Failed connecting to runner-1 on 192.168.64.29: The operation couldn’t be completed. (NIOPosix.NIOConnectionError error 1.)
13:50:09.785 INFO: Attempt 1 out of 3 to establish an SSH connection to the virtual machine named runner-1 failed.
13:50:11.881 INFO: Failed connecting to runner-1 on 192.168.64.29: The operation couldn’t be completed. (NIOPosix.NIOConnectionError error 1.)
13:50:11.882 INFO: Attempt 2 out of 3 to establish an SSH connection to the virtual machine named runner-1 failed.
13:50:13.957 INFO: Failed connecting to runner-1 on 192.168.64.29: The operation couldn’t be completed. (NIOPosix.NIOConnectionError error 1.)
13:50:13.959 INFO: Attempt 3 out of 3 to establish an SSH connection to the virtual machine named runner-1 failed.
13:50:13.960 INFO: Last attempt to establish an SSH connection to virtual machine named runner-1 failed.
13:50:13.961 INFO: Failed establishing connection to virtual machine after retrying a number of times.
13:50:13.962 INFO: Could not connect to the virtual machine over SSH, so the virtual machine will be shut down.
13:50:13.967 INFO: Virtual machine named runner-1 stopped with message: Failed establishing SSH connection
13:50:14.779 INFO: Start virtual machine named runner-1
13:50:18.108 INFO: Failed obtaining IP address of virtual machine named runner-1: The operation couldn’t be completed. (Swift.CancellationError error 1.)
13:50:18.110 INFO: Did stop virtual machine named runner-1
13:50:18.806 INFO: Did delete virtual machine named runner-1
13:50:18.807 INFO: Task running virtual machine named runner-1 was cancelled.

Booting into the 'editor' mode (Virtual Machine → Edit Virtual Machine) works. I can then also SSH into the virtual machine.
I tried reinstalling Tartelet and the virtual machine, but that didn't help.

If there is something else I can try or test, let me know 🙂

[kuhnroyal]

Try to manually SSH into the machine parallel to Tartelet and see if you can get a more specific error?

[annuh]

Try to manually SSH into the machine parallel to Tartelet and see if you can get a more specific error?

That works, but you have to very quick since the runner is restarted immediately. Here is some example output:

❯ ssh -o StrictHostKeyChecking=accept-new runner@192.168.64.28 "hostname"
Warning: Permanently added '192.168.64.28' (ED25519) to the list of known hosts.
runners-Virtual-Machine.local

[kuhnroyal]

Hmm then tartlet should be able to SSH as well. Verify your SSH credentials in Tartelet?

[annuh]

Hmm then tartlet should be able to SSH as well. Verify your SSH credentials in Tartelet?

I just retyped the SSH password and now... it works! 🥳 Thanks!
Must have changed the value there, by accident. @kengfai could you check if this works for you as well?

[hisaac]

I got stuck on this today after reinstalling Tartelet using the new Homebrew cask. What finally fixed it for me was toggling off/on the setting to allow Tartelet to connect to local network devices.

System Settings > Privacy & Security > Local Network > Toggle Tartelet switch

[quasivivo]

If it helps anyone, I encountered this just now after creating a new Github app, but then forgetting to install it on an org. The Tartelet logs showed everything necessary to fix, but it appeared to be a SSH failure until I found the logs.

15:41:38.320 INFO: The GitHub app has not been installed. Please install it from the developer settings.
15:41:38.322 INFO: Could not connect to the virtual machine over SSH, so the virtual machine will be shut down.
15:41:38.326 INFO: Virtual machine named runner-1 stopped with message: Failed establishing SSH connection

[pavel4766]

I got stuck on this today after reinstalling Tartelet using the new Homebrew cask. What finally fixed it for me was toggling off/on the setting to allow Tartelet to connect to local network devices.

System Settings > Privacy & Security > Local Network > Toggle Tartelet switch

This fixed it for me. Thank you.