PuppetDB: Failed to submit 'replace facts' command

I got it going, but can't say exactly what steps were necessary or not.

This issue started because authentication on several hosts was slow or hanging, and appeared to be related to domain controller/DNS cache issues. Removing domain mydomain.com entry from /etc/resolv.conf on the puppet master and agents solved the issue, but that created issues with existing puppet certs. I ran puppet cert clean --all on the master to try and recreate all certs, but this did not play well with PuppetDB.

Solution

Clean out old certs on master:

puppet cert clean --all

Clean out old certs on all agents:

rm -rf /var/lib/puppet/ssl

Recreate PuppetDB keystores:

facter fqdn is not available after removing domain foo.com from /etc/resolv.conf. This causes puppetdb-ssl-setup to fail silently.

Edit /usr/sbin/puppetdb-ssl-setup, add a piece of code to use just facter hostname if facter fqdn is empty:

# near line 10
fqdn=`facter fqdn`
# add this "if" section
if [ ! -n "$fqdn" ] ; then
  fqdn=`facter hostname`
fi

Permissions fix:

chown -R puppetdb:puppetdb /etc/puppetdb/ssl

Update passwords in /etc/puppetdb/conf.d/jetty.ini with new keystore/truststore passcode (same pass), which you can get from:

cat /etc/puppetdb/ssl/puppetdb_keystore_pw.txt

Restart puppetdb

service puppetdb restart

Then go to each agent and request new certs and sign each on the master.