/claim #7724

Problem

Coolify sporadically sends the wrong SSH key when connecting to servers, causing Permission denied (publickey,password) errors. Regenerating the key temporarily fixes it, then it breaks again.

Root Cause Analysis

I identified three interrelated bugs in SshMultiplexingHelper.php that together explain the sporadic behavior:

1. validateSshKey() only checks file existence, not content (primary cause)

When a user regenerates their SSH private key, the new key is saved to the database but validateSshKey() only checks if the key file exists on disk:

// Before: only checks existence
$checkKeyCommand = "ls $keyLocation 2>/dev/null";
$keyCheckProcess = Process::run($checkKeyCommand);
if ($keyCheckProcess->exitCode() !== 0) {
    $privateKey->storeInFileSystem();
}

If the old key file still exists (which it does — same UUID means same filename), the stale file is never updated. SSH then uses the old key from disk while the server expects the new key from the database.

This explains why regenerating the key “temporarily fixes it” — the changePrivateKey flow calls refresh_server_connection() → removeMuxFile() and storeInFileSystem(), which writes the correct key. But subsequent operations that go through validateSshKey() only check existence and never detect the stale content.

2. Missing -o IdentitiesOnly=yes SSH option

Without IdentitiesOnly=yes, SSH will try all available identity files (e.g., ~/.ssh/id_rsa, ~/.ssh/id_ed25519) in addition to the one specified with -i. This can:

• Exhaust authentication attempts with wrong keys before trying the correct one
• Trigger Too many authentication failures or Permission denied if the server has a low MaxAuthTries

3. Stale multiplexed connections not invalidated on key change

When validateSshKey() detects and rewrites a stale key file, the existing SSH multiplexed (mux) connection — which was authenticated with the old key — is not torn down. Subsequent SSH commands reuse this stale mux connection, which effectively uses wrong credentials.

Fix

All changes are in app/Helpers/SshMultiplexingHelper.php:

1. validateSshKey() now compares file content against the database — reads the key file and compares it to the decrypted value from the database. Only rewrites when content differs (or file is missing). Also uses native file_exists() instead of shelling out to ls.

2. Added -o IdentitiesOnly=yes to getCommonSshOptions() — ensures SSH only offers the specified key file, preventing interference from any other keys on the system.

3. Mux connection is invalidated when a stale key is detected — if validateSshKey() had to rewrite the key file, removeMuxFile() is called before establishing a new connection, ensuring the new connection uses the correct key.

Testing

This is a race-condition/state-desync bug that requires a multi-server Coolify Cloud environment to reproduce naturally. The fix is defensive — it ensures key file content is always validated before use and stale connections are always torn down.