Summary

Fixes #7724 - Sporadic “Permission denied (publickey,password)” SSH errors

Problem

SSH connections fail sporadically with “Permission denied” errors when:

1. SSH key files on disk become stale/out-of-sync with database content
2. Multiplexed SSH connections continue using cached connections made with old keys
3. Eloquent relationship caching returns stale PrivateKey data

Root Cause Analysis

The existing validateSshKey() method only checked if the key file exists (using ls), but never verified that its content matched the database. When keys were updated:

• The database was updated correctly
• The file system might still have old key content
• Multiplexed connections continued using cached connections authenticated with the old key
• This caused sporadic failures when the old key was no longer valid on the server

Solution

This PR implements a comprehensive fix addressing all identified root causes:

1. SSH Key Content Validation

• validateSshKey() now uses Laravel’s Storage::disk('ssh-keys') to compare file content with database
• Refreshes PrivateKey from database to bypass Eloquent’s in-memory caching
• Re-stores key file when content mismatch detected

2. Multiplexed Connection Invalidation

• When key content mismatch detected, invalidates all mux connections using that key
• New isKeyMismatch() method detects when mux connection was made with a different key fingerprint
• ensureMultiplexedConnection() now checks for key mismatches before reusing connections

3. Key Fingerprint Tracking

• storeConnectionMetadata() now stores key fingerprint alongside connection time
• Enables detection of key changes for existing multiplexed connections
• clearConnectionMetadata() clears fingerprint cache when connection is closed

4. Server Model Enhancement

• saved event now invalidates mux connection when private_key_id changes
• Handles the case where a server is switched to use a different SSH key

Changes

• app/Helpers/SshMultiplexingHelper.php: Enhanced key validation, fingerprint tracking, mux invalidation
• app/Models/Server.php: Added mux invalidation when private_key_id changes
• tests/Unit/SshKeyValidationTest.php: Added comprehensive unit tests

Testing

• Unit tests added for all new functionality
• Tests verify implementation patterns (Storage disk usage, logging, fingerprint tracking)
• Syntax validation passed
• Manual testing (demo video can be added per bounty requirements)

Improvements Over Previous PRs

This solution improves upon PR #7727 and #8125 by:

• Using Laravel’s Storage disk instead of shell commands for security
• Refreshing PrivateKey from database to bypass Eloquent caching
• Tracking key fingerprints to detect mux connection key mismatches
• Handling private_key_id changes in Server model
• Comprehensive logging for debugging sporadic issues

/claim #7724