certificate verify failed (certificate has expired) even with latest release with letsencrypt

[gionn]

Description

Facing the Letsencrypt Root CA X3 expiration, I hoped that upgrading to latest 16.x (16.16.7) would have solved the issue, but it's not.

Chef Version

16.16.7

Platform Version

System Info:
------------
chef_version=16.16.7
platform=ubuntu
platform_version=18.04
ruby=ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
program_name=/opt/chef/bin/chef-solo
executable=/opt/chef/bin/chef-solo

Replication Case

apt_repository("postgresql_org_repository_13") do
  action [:add]
  uri "https://download.postgresql.org/pub/repos/apt/"
  distribution "bionic-pgdg"
  components ["main", "13"]
  key ["https://download.postgresql.org/pub/repos/apt/ACCC4CF8.asc"]
end

Client Output

OpenSSL::SSL::SSLError
----------------------
apt_repository[postgresql_org_repository_13] (/home/ubuntu/chef/gionn_chef/local-mode-cache/cache/cookbooks/postgresql/resources/repository.rb line 93) had an error: OpenSSL::SSL::SSLError: remote_file[/home/ubuntu/chef/gionn_chef/local-mode-cache/cache/https___download_postgresql_org_pub_repos_apt_ACCC4CF8_asc] (/opt/chef/embedded/lib/ruby/gems/2.7.0/gems/chef-16.16.7/lib/chef/resource/apt_repository.rb line 261) had an error: OpenSSL::SSL::SSLError: SSL Error connecting to https://download.postgresql.org/pub/repos/apt/ACCC4CF8.asc - SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)

Stacktrace

[lotooo]

Same here

      ================================================================================                                                                                                                                                    
      Error executing action `add` on resource 'apt_repository[nginx]'                                                                                                                                                                    
      ================================================================================                                                                                                                                                    
                                                                                                                                                                                                                                          
      OpenSSL::SSL::SSLError                                                                                                                                                                                                              
      ----------------------                                                                                                                                                                                                                    remote_file[/var/chef/cache/https___nginx_org_keys_nginx_signing_key] (/opt/chef/embedded/lib/ruby/gems/2.7.0/gems/chef-16.16.7/lib/chef/resource/apt_repository.rb line 261) had an error: OpenSSL::SSL::SSLError: SSL Error connecting to https://nginx.org/keys/nginx_signing.key - SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)
      
      Resource Declaration:
      ---------------------
      # In /var/chef/cache/cookbooks/nginx/resources/install.rb  
      
       84:       apt_repository 'nginx' do
       85:         uri          repo_url(train: new_resource.repo_train)
       86:         components   %w(nginx)  
       87:         deb_src      true
       88:         key          repo_signing_key
       89:       end
       90:     when 'suse'
                      
      Compiled Resource:     ```

[gionn]

This shed some light: https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

To fix the problem it's ok to upgrade to openssl 1.0.2zb BUT it should also have -DOPENSSL_TRUSTED_FIRST_DEFAULT added at build time.

A workaround, that is working for me, is to manually remove the expired CA from trusted cert:

• open /opt/chef/embedded/ssl/cert.pem with a text editor
• search for DST Root CA X3
• remove the entire cert block
• profit

Another workaround is overriding the bundled chef cacerts via SSL_CERT_FILE environment variable before launching chef-client, on ubuntu platforms I've set SSL_CERT_FILE='/etc/ssl/certs/ca-certificates.crt'.

[l33z3r]

• /opt/chef/embedded/ssl/cert.pem

Hi gionn, thanks for that workaround, but can you elaborate a bit on it. I mean where do you get the ca-certificates.crt file that you use to override the default one?

[rmoriz]

In the meantime https://curl.se/ca/cacert.pem has removed the expired intermediate, so this could be used as a drop-in replacement bundle.

[AmberSpeedyfruit]

For those on windows that are removing the DST Root CA X3 certificate. It will need to be removed from two files:

• C:\cinc-project\cinc\embedded\ssl\cert.pem
• C:\cinc-project\cinc\embedded\ssl\certs\cacert.pem

Whilst I'm using cinc its more than likely the same with chef, you just need to use chef directory names instead of cinc

[lamont-granquist]

The builds yesterday were incorrectly built without the right compile time option to openssl 1.0.2, so there was basically no difference.

The cacert has been updated in the builds now and I'm working on fixing the compile option to be able to release a fixed build later today

[lamont-granquist]

Locking the conversation since that is the relevant information about our process of fixing the builds, and I don't want this to turn into a chat channel about general letsencrypt issues. The #general channel on the chef community slack would be more appropriate for questions.

[lamont-granquist]

The 17.6.18 release on Oct 1st should have fixed this.