/claim #7724

Problem

Sporadic Permission denied (publickey,password) errors occur because the SSH subsystem can silently use a stale private key to authenticate. Three independent failure modes combine to cause this:

1. Key file validation only checks existence, not content

validateSshKey() runs ls to confirm the file exists on disk, but never verifies its content matches the database. When a user rotates their SSH key (updates private_key in the DB), the old key file remains on disk and SSH happily authenticates with it — until it doesn’t.

2. Multiplexed connections survive key rotation

Even if the key file is eventually refreshed, an existing mux socket stays open and continues authenticating with the key that was active when the connection was first established. There is no mechanism to detect that the underlying key has changed.

3. No file permission enforcement

SSH clients refuse private keys that are group- or world-readable (0644). Laravel’s local Storage driver’s put() does not guarantee 0600 permissions on every write, especially after container restarts or volume remounts.

Root Cause

// Before: only checks file existence
private static function validateSshKey(PrivateKey $privateKey): void
{
    $keyLocation = $privateKey->getKeyLocation();
    $checkKeyCommand = "ls $keyLocation 2>/dev/null";
    $keyCheckProcess = Process::run($checkKeyCommand);

    if ($keyCheckProcess->exitCode() !== 0) {
        $privateKey->storeInFileSystem();
    }
}

This passes as long as any file exists at that path — even if it contains a completely different (old) key.

Solution

SshMultiplexingHelper::validateSshKey() — content-aware validation

• Refreshes the PrivateKey model from the database (PrivateKey::find()) to bypass Eloquent’s in-memory relationship cache that may hold a stale version from earlier in the request
• Compares file content against the DB value; re-writes the file on mismatch
• Invalidates all multiplexed connections for servers using that key when a mismatch is detected
• Enforces 0600 permissions on every invocation via ensureKeyPermissions()

SshMultiplexingHelper::ensureMultiplexedConnection() — fingerprint tracking

• storeConnectionMetadata() now caches the SSH key fingerprint alongside the connection timestamp
• isKeyStaleForConnection() compares the cached fingerprint against the current key; triggers refreshMultiplexedConnection() on mismatch
• clearConnectionMetadata() cleans up both cache entries
• Gracefully handles pre-upgrade connections (no cached fingerprint → no forced refresh)

Server::saved() — immediate mux teardown on key reassignment

• Detects wasChanged('private_key_id') in the saved hook
• Calls SshMultiplexingHelper::removeMuxFile() so the next SSH command establishes a fresh connection with the new key

Files Changed

Defense in Depth

The fix operates at three layers to ensure no stale key can be used:

1. File layer — every generateSshCommand() call validates the key file content and permissions
2. Mux layer — every ensureMultiplexedConnection() call checks the key fingerprint against the one used when the mux was established
3. Model layer — changing a server’s SSH key immediately tears down the mux socket

Backward Compatibility

• All changes are additive; no config keys, database migrations, or API changes
• Pre-upgrade mux connections without cached fingerprints are left alone (no unnecessary churn)
• The existing retry mechanism (SshRetryable trait) continues to work as a complementary safety net