TL; DR; If your source code is written in Go and it uses one-way or mutual TLS authentication, you are vulnerable to CPU denial of service (DoS) attacks. The attacker can formulate inputs in a way that makes the verification algorithm in Go’s crypto/x509 standard library hog all available CPU resources as it tries to verify the TLS certificate chain the client has provided.
To protect your services, upgrade immediately to Go v1.10.6 or later, or v1.11.3 or later.
Why we were interested
The backend of the API Security platform by 42Crunch has been implemented using a microservices architecture, with the microservices written in Go. The microservices communicate with each other over gRPC and have a REST API gateway for external invocations. To ensure security, we follow the “TLS everywhere” mantra, extensively relying on the mutual TLS authentication.
Go provides native SSL/TLS support in its standard library, as well as an extensive set of x509 and TLS primitives for manipulating connections, verifications, authentication, certificates, and so forth. This native support avoids external dependencies and reduces the risks by using a standard vetted and maintained TLS implementation.
It naturally follows that 42Crunch could potentially be affected, was curious about this TLS vulnerability and had to understand it to ensure the security of the 42Crunch platform.
The following analysis and details on this CVE are provided by the 42Crunch security team.
Problem
The DoS issue in TLS chain validation was originally found and reported by Netflix, as described in Golang’s issue tracker:
“Package crypto/x509 parses and validates X.509-encoded keys and certificates. It’s supposed to handle certificate chains provided by an attacker with reasonable resource use.
The crypto/x509 package does not limit the amount of work performed for each chain verification, which might allow attackers to craft pathological inputs leading to a CPU denial of service. Go TLS servers accepting client certificates and TLS clients verifying certificates are affected.“
The issue lies along the call path crypto/x509 Certificate.Verify() function, which is responsible for authenticating and verifying certificates.
Description
To simplify and stay concise, we explain only an example where a TLS client connects to a TLS server verifying the client certificate.
A TLS server is listening to port 8080 for TLS clients and verifying client certificates against one trusted certificate authority (CA):
caPool := x509.NewCertPool()
ok := caPool.AppendCertsFromPEM(caCert)
if !ok {
panic(errors.New("could not add to CA pool"))
}
tlsConfig := &tls.Config{
ClientCAs: caPool,
ClientAuth: tls.RequireAndVerifyClientCert,
}
//tlsConfig.BuildNameToCertificate()
server := &http.Server{
Addr: ":8080",
TLSConfig: tlsConfig,
}
server.ListenAndServeTLS(certWeb, keyWeb)In a standard TLS verification scenario, the TLS client connects to the TLS server on port 8080 and provides its “trust chain” that includes the client certificate, the root CA certificate, and all the intermediate CA certificates. The TLS server handles the TLS handshake and to verify the client certificate, checks if the client is trusted (the client certificate is signed by a CA the server trusts). The following diagram shows a simplified flow how the TLS handshake normally goes:
Following through the Go’s crypto/x509 library, you end up in x509/tls/handshake_server.go:doFullHandshake() :
...
if c.config.ClientAuth >= RequestClientCert {
if certMsg, ok = msg.(*certificateMsg); !ok {
c.sendAlert(alertUnexpectedMessage)
return unexpectedMessageError(certMsg, msg)
}
hs.finishedHash.Write(certMsg.marshal())
if len(certMsg.certificates) == 0 {
// The client didn't actually send a certificate
switch c.config.ClientAuth {
case RequireAnyClientCert, RequireAndVerifyClientCert:
c.sendAlert(alertBadCertificate)
return errors.New("tls: client didn't provide a certificate")
}
}
pub, err = hs.processCertsFromClient(certMsg.certificates)
if err != nil {
return err
}
msg, err = c.readHandshake()
if err != nil {
return err
}
}
...Here, the server processes the client certificate it received and calls the function x509/tls/handshake_server.go:processCertsFromClient(). If the client certificate has to be verified, the server creates a VerifyOptions structure that contains:
- The root CA pool, a list of trusted CAs configured to verify clients certificate (cotrolled by the server)
- The intermediate CA pool, a list of received intermediate CAs (controlled by the client)
- The signed client certificate (controlled by the client)
- Other fields (optional)
if c.config.ClientAuth >= VerifyClientCertIfGiven && len(certs) > 0 {
opts := x509.VerifyOptions{
Roots: c.config.ClientCAs,
CurrentTime: c.config.time(),
Intermediates: x509.NewCertPool(),
KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
}
for _, cert := range certs[1:] {
opts.Intermediates.AddCert(cert)
}
chains, err := certs[0].Verify(opts)
if err != nil {
c.sendAlert(alertBadCertificate)
return nil, errors.New("tls: failed to verify client's certificate: " + err.Error())
}
c.verifiedChains = chains
}To understand what went wrong, you need to understand how certificate pools are organised to allow certificate verification in an efficient manner. A certificate pool, simply put, is a list of certificates, that are accessible through three different ways depending on the need. The following diagram shows an example of this: Certificates grouped in a pool are accessible through an indexed array (named “Certs”) and are hashed by CN, IssuerName, SubjectKeyId.
Verification
The server calls the function Verify() with VerifyOptions on the client certificate (the first certificate in the chain:certs[0]).
Then, Verify() takes the client certificate to be verified against the provided chain. However, first the verification chain must be built and checked using the buildChains() function:
var candidateChains [][]*Certificate
if opts.Roots.contains(c) {
candidateChains = append(candidateChains, []*Certificate{c})
} else {
if candidateChains, err = c.buildChains(make(map[int][][]*Certificate), []*Certificate{c}, &opts); err != nil {
return nil, err
}
}The buildChains() function in turn calls some CPU-expensive functions sequentially and recursively calls itself on each element of the chain it finds.
The buildChains() function relied on a helper function findVerifiedParents() that identifies the parent certificate, using map access of the certificate pool by IssuerName or AuthorityKeyId, and returns the index of the certificate candidate that then gets verified against the client controlled pool.
In normal conditions, IssuerName and AuthorityKeyId are populated and expected to be unique, which returns only one certificate to verify:
func (s *CertPool) findVerifiedParents(cert *Certificate) (parents []int, errCert *Certificate, err error) {
if s == nil {
return
}
var candidates []int
if len(cert.AuthorityKeyId) > 0 {
candidates = s.bySubjectKeyId[string(cert.AuthorityKeyId)]
}
if len(candidates) == 0 {
candidates = s.byName[string(cert.RawIssuer)]
}
for _, c := range candidates {
if err = cert.CheckSignatureFrom(s.certs[c]); err == nil {
parents = append(parents, c)
} else {
errCert = s.certs[c]
}
}
return
}The buildChains() function calls the following on the whole certificate chain the client sent to the TLS server:
-
findVerifiedParents(client_certificate)on the root CA pool (server-side) to locate the signing authority (if it’s the root CA) of the verified certificate and check the signatures of all candidates for the certificateAuthorityKeyId(if not nil) or the raw issuer value (if nil) -
findVerifiedParents(client_certificate)on the intermediate CA pool (client-provided) to locate the signing authority (if it’s the root CA) of the verified certificate and check the signatures of all candidates for the certificateAuthorityKeyId(if not nil) or the raw issuer value (if nil), - Get the signing intermediate parent
- Call
buildChains()with the newly-found intermediate parent, which repeats the whole set of signature checks previous described
The DoS attack
The main CPU DoS is triggered by buildChains() and findVerifiedParent() functions in the unexpected conditions where all intermediate CA certificates share the same name and have a nil AuthKeyId value. The findVerifiedParent() function returns all certificates matching that name, which is the entire pool, and then checks signatures against all the certificates. Once that is done, the buildchains() function is again called for the found parent recursively until it reaches the root CA, each time verifying against the entire intermediate CA pool ,and hence consuming all available CPU for only one TLS connection!
Impact
An attacker can construct a certificate chain that makes the client certificate verification consume all the CPU resources and thus making the host less responsive. This has been implemented with only one connection. In accordance with Go scheduler rules, only two CPU cores were affected and used at 100%, creating a new connection and forcing the scheduler to allocate more resources to process the signature check, which in turn can lead to an unresponsive service or host.
Remediation
The Go language community has fixed the issue by implementing the following changes:
- Moving signature checking out of the
findVerifiedParent()certificate pool lookup - Limit the number of signature checks to the maximum of 100 intermediate CAs (which is an unrealistic trust chain)
To get the fixes, upgrade immediately to Go v1.10.6 or later, or v1.11.3 or later.
Get API Security news directly in your Inbox.
By clicking Subscribe you agree to our Data Policy
