Proposed Changes

Fixes #819

/claim #819

Three separate timeout bugs cause tlsx to hang indefinitely when scanning hosts with problematic TLS configurations (accepting TCP but never completing the TLS handshake). All three are fixed in this PR.

Bug 1: Broken select in ztls tlsHandshakeWithTimeout()

The original code evaluated tlsConn.Handshake() synchronously inside the select case expression:

// BEFORE: Handshake() blocks here; ctx.Done() is never evaluated
select {
case <-ctx.Done():
    return error
case errChan <- tlsConn.Handshake(): // blocks forever
}

The fix moves the handshake into a goroutine so the select can properly race between handshake completion and context cancellation. On timeout, the connection is force-closed to unblock the goroutine’s pending I/O:

// AFTER: goroutine races against ctx.Done()
go func() { errChan <- tlsConn.Handshake() }()
select {
case <-ctx.Done():
    _ = tlsConn.Close()
    return timeout error
case err := <-errChan:
    ...
}

Bug 2: ztls cipher enumeration used context.TODO() (no timeout)

Each per-cipher handshake attempt during EnumerateCiphers was called with context.TODO(), which has no deadline. Replaced with a context.WithTimeout derived from the configured --timeout value (default 10s).

Bug 3: ctls cipher enumeration used bare Handshake() (no context)

The standard crypto/tls EnumerateCiphers path used conn.Handshake() which has no way to cancel. Changed to conn.HandshakeContext(ctx) with a proper timeout context (default 5s).

Additional fixes

• Config race condition: Both ctls and ztls cipher enumeration loops mutated baseCfg.CipherSuites in-place during iteration. Since the ztls handshake now runs in a goroutine, this creates a data race. Fixed by cloning the config per cipher iteration.
• File descriptor leak: file_writer.Close() returned early when Flush() failed, skipping file.Close() and leaking the fd. Now always releases the file.
• OpenSSL dial timeout: openssl.ConnectWithOptions used context.TODO() for its Dial call, which could block indefinitely on unreachable hosts. Now uses a bounded context.

Proof

Regression test (TestHandshakeTimeout) starts a local TCP listener that accepts connections but never speaks TLS — exactly the scenario from the bug report. Verifies the handshake returns within the configured timeout:

=== RUN   TestHandshakeTimeout
    timeout_test.go:68: handshake correctly timed out in 2.001s with error: ...
--- PASS: TestHandshakeTimeout (2.00s)
PASS

Without this fix the test hangs indefinitely.

Checklist

• PR created against the dev branch
• Tests added that prove the fix is effective
• Minimal, focused diff — only touches timeout/handshake paths and the related fd leak
• No unrelated changes