SSL fingerprint inconsistency: what does it mean?

This probably depends on dns resolution + caching that might happen also at the notary servers. If you resolve www.facebook.com you usually get one IP address. But this IP changes over time and may lead you to somewhere different.

I just did a quick check and indeed the www.facebook.com IP address I was given changed, and also the certificate information was different.

Over the course of a few minutes, I received 3 different IP addresses for www.facebook.com: 69.171.229.16, 69.171.234.80, 69.171.228.40

Getting the SSL certificate for each of those IPs produced different certificates. However the certificates weren't consistent even for the same IP address it seems. Over those few minutes I received at least two certificates:

-----BEGIN CERTIFICATE-----
MIIDzDCCAzWgAwIBAgIQPAjP7r6f68QrsT7gPWIL3zANBgkqhkiG9w0BAQUFADCB
ujEfMB0GA1UEChMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazEXMBUGA1UECxMOVmVy
aVNpZ24sIEluYy4xMzAxBgNVBAsTKlZlcmlTaWduIEludGVybmF0aW9uYWwgU2Vy
dmVyIENBIC0gQ2xhc3MgMzFJMEcGA1UECxNAd3d3LnZlcmlzaWduLmNvbS9DUFMg
SW5jb3JwLmJ5IFJlZi4gTElBQklMSVRZIExURC4oYyk5NyBWZXJpU2lnbjAeFw0x
MTExMTcwMDAwMDBaFw0xMjA3MTMyMzU5NTlaMGoxCzAJBgNVBAYTAlVTMRMwEQYD
VQQIEwpDYWxpZm9ybmlhMRIwEAYDVQQHEwlQYWxvIEFsdG8xFzAVBgNVBAoTDkZh
Y2Vib29rLCBJbmMuMRkwFwYDVQQDExB3d3cuZmFjZWJvb2suY29tMIGfMA0GCSqG
SIb3DQEBAQUAA4GNADCBiQKBgQC4e9C0eD3zy0YR829bH7fd3PGGDp/3Rd7UR65Q
+jcsRoLZaer9k9tPEOd5ZmWR1MTzwVEmZ94fhoWf219K2Nx/v7fQaWYh5U0DETUo
bkDfR4zBAe+oMuFDIGrhEkUEdlWUOcpvScrtzRjRLyikTc4twjRlpB5RdnGGFopw
CRTNMwIDAQABo4IBIDCCARwwCQYDVR0TBAIwADBEBgNVHSAEPTA7MDkGC2CGSAGG
+EUBBxcDMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9y
cGEwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL1NWUkludGwtY3JsLnZlcmlzaWdu
LmNvbS9TVlJJbnRsLmNybDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
CwYDVR0PBAQDAgWgMCkGA1UdEQQiMCCCEHd3dy5mYWNlYm9vay5jb22CDGZhY2Vi
b29rLmNvbTA0BggrBgEFBQcBAQQoMCYwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LnZlcmlzaWduLmNvbTANBgkqhkiG9w0BAQUFAAOBgQANiGfuAUQqkUZiD2cozCmb
7+e6vK5yzc/3o/zAd+QBydo7dkPo/t8nIHUfGxcAKUICjAzH/j3FDykK3bupFvBW
D4GoU6qVvNFgH8ucYWMNbxsknN/s3lcFryIGZYFcsYuPEB/rbBEEseKTilUSR7vt
pUEsLwSGdWwTNtKgy/COcQ==
-----END CERTIFICATE-----

and

-----BEGIN CERTIFICATE-----
MIIDnzCCAwigAwIBAgIQCSGX4cDpzQPaNSQ2VhCGgTANBgkqhkiG9w0BAQUFADCB
ujEfMB0GA1UEChMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazEXMBUGA1UECxMOVmVy
aVNpZ24sIEluYy4xMzAxBgNVBAsTKlZlcmlTaWduIEludGVybmF0aW9uYWwgU2Vy
dmVyIENBIC0gQ2xhc3MgMzFJMEcGA1UECxNAd3d3LnZlcmlzaWduLmNvbS9DUFMg
SW5jb3JwLmJ5IFJlZi4gTElBQklMSVRZIExURC4oYyk5NyBWZXJpU2lnbjAeFw0x
MTA3MTQwMDAwMDBaFw0xMjA3MTMyMzU5NTlaMGoxCzAJBgNVBAYTAlVTMRMwEQYD
VQQIEwpDYWxpZm9ybmlhMRIwEAYDVQQHEwlQYWxvIEFsdG8xFzAVBgNVBAoTDkZh
Y2Vib29rLCBJbmMuMRkwFwYDVQQDExB3d3cuZmFjZWJvb2suY29tMIGfMA0GCSqG
SIb3DQEBAQUAA4GNADCBiQKBgQC4e9C0eD3zy0YR829bH7fd3PGGDp/3Rd7UR65Q
+jcsRoLZaer9k9tPEOd5ZmWR1MTzwVEmZ94fhoWf219K2Nx/v7fQaWYh5U0DETUo
bkDfR4zBAe+oMuFDIGrhEkUEdlWUOcpvScrtzRjRLyikTc4twjRlpB5RdnGGFopw
CRTNMwIDAQABo4H0MIHxMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEFBQcwAYYYaHR0
cDovL29jc3AudmVyaXNpZ24uY29tMAkGA1UdEwQCMAAwRAYDVR0gBD0wOzA5Bgtg
hkgBhvhFAQcXAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3d3dy52ZXJpc2lnbi5j
b20vcnBhMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9TVlJJbnRsLWNybC52ZXJp
c2lnbi5jb20vU1ZSSW50bC5jcmwwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUF
BwMCMAsGA1UdDwQEAwIFoDANBgkqhkiG9w0BAQUFAAOBgQAwUGokl1919Kf6YPDK
uoF+k7CFm+j2YtZqw4ilP9166qkM8IDWyYa87MB084WXSyfS1VnT5FSwzjP37+Qx
gjRaROuWGxfY25KebCQpoBW2PJp3S1JmqHHyxjk4mzr+tzWK0Qn+tlBUy9igtkIh
VybjO+AxBZve1qyJIsVraz8wrw==
-----END CERTIFICATE-----

Facebook might have some priority-list of which IP address the DNS issues and then the SSL endpoint you hit (via their load balancers) when you access any one of the facebook IP addresses. Their infrastructure setup might 'prefer' some endpoints over others (maybe they have stronger hardware on one endpoint over another), which is why you'd mostly see one fingerprint, but occasionally another.

UPDATE:

To answer your additional questions more specifically:

Does this improve security?

I think it does, but primarily if you consider availability rather than confidentiality or integrity when you say 'security'. If one certificate expires/needs to be revoked/needs replacing for any other reason, you still have another one fully-operational, live and ready.

Keeping track of which server has which certificate looks tedious, is this method really worthwhile? It's not even scalable since certificates are different within a single regional cluster

It might be a bit tedious, but I'm sure it's only a fraction of the tedious tasks facebook have to deal with in the grand scheme of things. Is it worth it? I think so, and I'd say facebook do it for a reason (As I mentioned above, primarily availability reasons). I would imagine it's not running on a server, but some beefy dedicated SSL accelerator/endpoint cluster of some sort. Those will handle SSL traffic and then reverse-proxy communication to some front-end web servers (which wouldn't need to worry about certificates).

How about just keeping spare certificates in case of a leak? Keeping them versus deploying them all seems better from a security point of view: if one leaked, why not the others?

Yes, keeping some spare certificates offline is a very good idea, and without knowing I would bet Facebook do that too. Having multiple online certificates is not mutually exclusive to having a few offline too. You are assuming that they deploy all of them, which might not be the case. Also making the logical connection that if one leaked so would the others requires more facts. If they have good security in place to keep things segregated, then one compromise doesn't immediately link to another. They could, for example, use different HSMs from different vendors on different server OS, protected by a separate set of firewalls and managed by a completely independent teams.


Facebook may have deployed multiple SSL certificates for the same endpoint. This is often done for security reasons (a compromised private key only affects a limited number of hosts) or for manageability reasons (I can stagger the validity periods so they don't all need to be replaced at the same time). Different certificates means different serial numbers. As a result, the fingerprints will be different.

It is also possible that the contents of the certificate are different. For example, they could have different subjectAltName extension values to reference a given server by several DNS aliases.