A bug let anyone log in with any password.
Mindfront was not exposed.

For ten years, OpenSSH on Alpine Linux could let anyone into an account meant never to take a password, using any password over 256 characters. Mindfront runs on Alpine, and four layers of defence kept the bug out.

A default Alpine server
was one password away.

Özgün Kültekin of Wakefield published *sshole on 5 November and traced the bug back to 2016. An Alpine server meets most of its conditions out of the box.

  • sshd runs on muslAlpineAlpine, OpenWrt, Buildroot, Yocto and many Docker images
  • sshd skips PAMAlpine default
  • Password login is onOpenSSH default
  • An account is locked with *Cloud imagesKey-only users, LDAP users, and the default user on Alpine's AWS and Azure images
  • Another account has a real passwordOne is enoughsshd borrows its salt to check the * account
  • Any password of 257 to 1024 characters logs in to the * account.

Conditions from Alpine's advisory and the write-up.

Illustrative: an exposed Alpine server, not ours

$ ssh alpine@server
alpine@server's password:
  # typed: 300 × "A"
server:~$ doas id
uid=0(root) gid=0(root)

On Alpine's AWS and Azure images, the default user can become root without a password.

Mindfront's servers were not exposed. They run Alpine without PAM, so they meet the first two conditions, but password login is off. Four layers of defence stand in the way, and two of them each stop the bug alone. Six minutes after the write-up reached us, our first servers had confirmed they take keys only, and by that evening every cloud server had. Our customers had nothing to do.

Two sensible choices
made * match *.

An account that should never take a password has * where its hash would be. Nothing hashes to *, so no password should ever match it.

  1. The locked account service:*:20397:0:99999:7::: Illustrative /etc/shadow line. The * means no password can match.
  2. The attacker sends 256 bytes 300 300 characters from an ordinary SSH client. musl refuses anything past 256 bytes.
  3. musl answers crypt(password, salt) "*" When musl refuses, it answers * instead of nothing.
  4. OpenSSH compares "*" == "*" logged in OpenSSH counts only a missing answer as failure.

OpenSSH 10.6p2 fixes the comparison: a stored hash shorter than four characters never matches.

Four layers
stand in its way.

An attacker has to get through all four, in order. The first two turn away anyone outside our operations network. The last two each stop the bug alone, from anywhere.

  1. 01

    Cloud firewall

    The cloud provider drops SSH from anywhere but our operations network, before it reaches the server.

    allowtcp:22from our operations network
    droptcp:22from everywhere else

    On its own: internet traffic never reaches sshd.

  2. 02

    Server firewall

    Each server runs its own firewall with the same rule, so one mistake in the cloud console cannot open SSH. Appliances on site also accept their local network, behind the site's router.

    allowtcp:22from our operations network, rate-limited
    dropallother incoming traffic
    blocksourceafter repeated failed logins

    On its own: the server drops the connection even if the cloud rule is wrong.

  3. 03

    Keys only

    sshd accepts one way in: a key held on a hardware security key. The private key never leaves the device, and every new login needs a person to touch it.

    AuthenticationMethods    publickey
    PubkeyAcceptedAlgorithms sk-ssh-ed25519@openssh.com,
                             sk-ssh-ed25519-cert-v01@openssh.com

    On its own: sshd never considers a password.

  4. 04

    Password login off

    sshd does not offer password login and turns away any client that tries. Keyboard-interactive, the other way to ask for a password, is off too.

    PasswordAuthentication       no
    KbdInteractiveAuthentication no

    What every cloud server answered on 5 November

    $ ssh -v -o PubkeyAuthentication=no server
    debug1: Authentications that can continue: publickey

    On its own: the code with the bug never runs. Even if asked, sshd's privileged half refuses to check a password while password login is off.

Every deploy
checks the layers.

sshd keeps the first value it reads for each setting, so a correct line in the wrong place changes nothing.

sshd_config, illustrative

  1. 12PasswordAuthentication yessshd uses this one
  2. …
  3. 97PasswordAuthentication noignored
  ✓ sshd refuses passwords
  ✓ sshd requires a public key
  ✓ sshd refuses root
  ✓ root account locked
  ✓ firewall admits SSH only from our network
  ✓ every administrator key is hardware-backed
host security verified
FATAL: sshd did not apply the managed policy: passwordauthentication no
Previous SSH configuration restored.
Host security verification failed; deployment stopped.

Adapted from our deploy check's output.

Each deploy writes our settings where sshd reads them first, then runs sshd -T, which prints what sshd will enforce. If anything differs, the deploy restores the previous files and stops, so that server cannot take a release. The same check confirms the firewall admits SSH only from our operations network, and that root is locked.

Six minutes
to a first answer.

SSH tells a client which login methods it accepts before anyone logs in, so we could ask each server without using a key.

  1. The write-up reaches us.

  2. We ask our servers which logins they accept.

  3. We read every cloud firewall rule and find one stale address.

  4. First answer: every server that replied takes keys only.

  5. A person approves, and the stale address is gone half a minute later.

  6. The last cloud servers answer: keys only.

5 November 2026, Cape Town time

With the other layers standing, there was no emergency. Every deploy upgrades each server's packages, OpenSSH included, so the fix goes on with the next release.

What any team
can take from this.

  1. No single point of failure, even in software you trust.

    We trust OpenSSH, and it still shipped this bug, in the seam between two components that each work as designed. Plan for every layer to fail one day, and make sure the next one does not depend on it.

    So we do not run musl everywhere. We lay our machines out as a matrix and keep the overlap between them small. The only way in from outside is a jump host that does nothing else, hardened and on an unlisted domain away from the rest of our infrastructure. It shares almost nothing with the servers behind it, so reaching them from the internet means breaking two different systems.

    Part Jump hostThe only way in from outside Overlap Cloud serversBehind it
    Distribution Mainstream, long-term support Different Alpine
    C library glibc Different musl*sshole
    Password check None: hardware keys only Different sshd's own, switched off*sshole
    Hardware Its own ARM board Different Cloud VMs, ARM and x86
    Updates Daily and automatic, reboots itself Different Scheduled by us
    SSH server OpenSSH Same project, different release OpenSSH, another release
    Kernel Linux Same project, different release Linux, another series
    *sshole needs musl and sshd's own password check together. The jump host has neither and accepts hardware keys only, so the bug could not take the way in.
  2. Turn off what you do not use.

    Nobody at Mindfront logs in to a server with a password, so password login is off. The bug sat in code our servers never run.

  3. Keep SSH off the open internet.

    When the internet cannot reach SSH, an SSH bug becomes a routine update. Keep a short list of sources you can name, and review it: ours had one stale entry, which we found on the day.

  4. Let the deploy check what the server enforces.

    Config files have includes, defaults and first-match rules, and sshd -T prints what sshd will actually do. Put that check in the deploy and let it stop the release, because a check someone has to remember gets skipped on a busy day.

Safe when AI
does the work.

Attackers can hand a published advisory to a model and get an attack back, so assume attacks start the day a bug is published.

13 of 15
real, published vulnerabilities that GPT-4 exploited on its own from their public descriptions, in a 2024 study. Every other model and scanner tested exploited none. Fang et al. ↗

We use the same tools on defence. Mindfront engineering works AI-first: on 5 November our AI coding tool did the checking, within its permission rules, and a person approved the one change.

Our AI coding tool's actions on 5 November

  1. Read the OpenSSH 10.6p2 changeAllowedread only
  2. Ask each server which logins it acceptsAllowedno key, no login
  3. Drop a stale address from a firewall ruleApprovedby a person
  4. Probe with host key checks turned offRefusedby its permission rules

None of the four layers depends on a model being right. Firewall rules, sshd settings, hardware keys and the deploy check give the same answer every time, and all of them were in place months before *sshole was published.

If you run
Alpine.

If both lines come back like this, you are exposed. Upgrade with apk upgrade -U and restart sshd, or set PasswordAuthentication no.

On your server

$ doas sshd -T | grep -E 'passwordauthentication|usepam'
passwordauthentication yes
usepam no

To see whether anyone already got in, search /var/log/messages for Accepted password on an account that has no password.

Layers guard
the AI too.

Mindfront runs on a private connection, the server enforces who can see what, and an action policy limits what the AI can do.