CVE-2026-11789: 389 Directory Server SMD5 Integer Underflow DoS Vulnerability

Technical summary

CVE-2026-11789 is an integer underflow vulnerability (CWE-191) in the SMD5 password storage plugin of 389 Directory Server. When the plugin processes a password hash shorter than 16 bytes, it performs an unsigned integer underflow while computing salt length, resulting in an out-of-bounds buffer over-read. This memory access violation crashes the LDAP server process during authentication attempts, causing service unavailability. The vulnerability requires high-privilege authentication to exploit, limiting its attack surface to authenticated users within trusted networks.

Business impact

Directory server downtime directly impacts identity and access management infrastructure. An attacker with valid high-privilege credentials can repeatedly crash the LDAP authentication service, forcing manual intervention to restore availability. In environments where 389 Directory Server is the primary identity provider, this denial of service can cascade to downstream applications that depend on authentication and directory queries. Recovery requires server restart, incurring operational overhead and potentially delaying legitimate user access.

Affected systems

Red Hat Directory Server (all listed versions), Red Hat 389 Directory Server, and Red Hat Enterprise Linux installations that include or depend on the affected directory server components are vulnerable. The vulnerability affects the authentication pathway at the point where password hashes are validated, so any environment relying on LDAP authentication through these products is exposed.

Exploitability

Exploitation requires an attacker to possess valid high-privilege authentication credentials and network access to the LDAP service. The attack is straightforward—submit a password hash shorter than 16 bytes during authentication—but the prerequisite of legitimate credentials significantly constrains the threat landscape. The vulnerability is not remotely exploitable by unauthenticated attackers. CVSS 3.1 score of 4.9 (MEDIUM severity) reflects the requirement for authenticated access and the limited impact (availability only, no confidentiality or integrity breach). The vulnerability is not currently tracked in CISA's Known Exploited Vulnerabilities (KEV) catalog.

Remediation

Apply security patches from Red Hat when available for affected Directory Server and Enterprise Linux versions. Until patches are deployed, restrict high-privilege account access to trusted administrators and monitor LDAP server logs for authentication failures or unexpected crashes. Implement network segmentation to limit who can reach LDAP services, and consider temporary rate-limiting on authentication attempts to slow potential attack attempts. Verify patch applicability with Red Hat's security advisories to confirm version compatibility.

Patch guidance

Check Red Hat's security advisory for CVE-2026-11789 to obtain patched versions of 389 Directory Server and Enterprise Linux packages. Test patches in a staging environment that mirrors your production LDAP setup before rolling out. Monitor authentication performance and server stability during and after patching. If your environment uses a third-party LDAP frontend or load balancer, verify that it does not cache or reuse connections in ways that could mask the fix.

Detection guidance

Monitor 389 Directory Server process behavior for unexpected crashes or frequent restarts correlating with authentication spikes. LDAP logs may show malformed password hash entries or authentication errors preceding crashes. Implement alerting on LDAP server restarts and sudden drops in authentication throughput. Endpoint Detection and Response (EDR) tools can flag repetitive authentication failures from the same high-privilege account. Review recent access logs for any high-privilege accounts attempting unusual operations immediately before crashes.

Why prioritize this

While the CVSS score is moderate (4.9), the vulnerability should be addressed in regular patch cycles because it directly impacts service availability and exploitability is trivial once a high-privilege account is compromised. Organizations should prioritize patching based on their reliance on 389 Directory Server for production authentication. However, this is not a critical emergency if access controls restrict high-privilege accounts and network segmentation is in place.

Risk score, explained

The CVSS 3.1 score of 4.9 reflects a network-accessible service flaw with low attack complexity but a mandatory requirement for authenticated, high-privilege access. The impact is limited to availability (server crash), with no confidentiality or integrity compromise. Unsigned integer underflow bugs are well-understood in the security community, but the prerequisite of valid high-privilege credentials prevents this from being a critical threat. The score appropriately balances the severity of DoS impact against the practical barrier to exploitation.

Frequently asked questions