# TLS certificates in Infra 

## Common naming convention.

Ansible roles are using the following logic to distribute .key/.cert files

```

  * {{ public_name }}.key : TLS private key

  * {{ public_name }}.crt : signed TLS certificate

  * {{ public_name }}-CAchain.crt : Trusted chain from CA (usually a symlink in pkistore is enough as we have a very few)

```

## Public certificates

### DigiCert

We have some long-term certificates that we use for www/lists/ci/etc .centos.org

Those need to be required through DigiCert Web ui and through internal RH ticket.

Once we have the signed cert back, we can upload it in pkistore and deploy

### Letsencrypt

!!! warning

    This is now the prefered way to retrieve and use TLS certs in the CentOS infra for all public services.

We use one dedicated node to obtain/renew certs for the acme http challenges, and also the same for dns challenges (for internal openshift setup). 

Actually that node is `certbot.rdu2.centos.org`.

#### How to obtain new cert (DNS challenge is the preferred way)

##### For dns challenge

We have automated the delegated dynamic zone needed for acme-challenge update with acme.sh

We just have once to add in our zone (main one, so centos.org)  a CNAME pointing to the delegated zone acme.centos.org

Example (simple) for forums.centos.org : 

```

_acme-challenge.forums IN  CNAME _acme-challenge.forums.acme.centos.org.

```

Now on certbot node, we can just ask acme.sh to dynamically update acme.centos.org with our ddns.key file (already present) that is permitted to update the acme.centos.org and instruct acme.sh that while asking for a record in centos.org, it has to update other TXT record for the 'acme-challenge' record

```

acme.sh --issue --dns dns_nsupdate -d forums.centos.org --challenge-alias forums.acme.centos.org

```

Let's see what this produces on our DNS node, basically updating TXT record:

```

Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: signer "ddns" approved

Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: updating zone 'acme.centos.org/IN': adding an RR at '_acme-challenge.forums.acme.centos.org' TXT "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ"

```

And back on the certbot node, where it just updates, then waits and finalizes the acme validation with letsencrypt servers:

```

[Tue 10 Dec 10:35:56 UTC 2019] Single domain='forums.centos.org'

[Tue 10 Dec 10:35:56 UTC 2019] Getting domain auth token for each domain

[Tue 10 Dec 10:35:57 UTC 2019] Getting webroot for domain='forums.centos.org'

[Tue 10 Dec 10:35:57 UTC 2019] Adding txt value: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain:  _acme-challenge.forums.acme.centos.org

[Tue 10 Dec 10:35:57 UTC 2019] adding _acme-challenge.forums.acme.centos.org. 60 in txt "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ"

[Tue 10 Dec 10:35:57 UTC 2019] The txt record is added: Success.

[Tue 10 Dec 10:35:57 UTC 2019] Let's check each dns records now. Sleep 20 seconds first.

[Tue 10 Dec 10:36:19 UTC 2019] Checking forums.centos.org for _acme-challenge.forums.acme.centos.org

[Tue 10 Dec 10:36:19 UTC 2019] Domain forums.centos.org '_acme-challenge.forums.acme.centos.org' success.

[Tue 10 Dec 10:36:19 UTC 2019] All success, let's return

[Tue 10 Dec 10:36:19 UTC 2019] Verifying: forums.centos.org

[Tue 10 Dec 10:36:22 UTC 2019] Success

[Tue 10 Dec 10:36:22 UTC 2019] Removing DNS records.

[Tue 10 Dec 10:36:22 UTC 2019] Removing txt: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain: _acme-challenge.forums.acme.centos.org

[Tue 10 Dec 10:36:22 UTC 2019] removing _acme-challenge.forums.acme.centos.org. txt

[Tue 10 Dec 10:36:22 UTC 2019] Removed: Success

[Tue 10 Dec 10:36:22 UTC 2019] Verify finished, start to sign.

[Tue 10 Dec 10:36:22 UTC 2019] Lets finalize the order, Le_OrderFinalize: https://acme-v02.api.letsencrypt.org/acme/finalize/52322595/1718928579

[Tue 10 Dec 10:36:23 UTC 2019] Download cert, Le_LinkCert: https://acme-v02.api.letsencrypt.org/acme/cert/04cd1bdeeef194461a56d1323b11691aeccd

[Tue 10 Dec 10:36:24 UTC 2019] Cert success.

```

PS : for a wildcard, just add multiple -d and '*.domain'

```

acme.sh --issue --dns dns_nsupdate -d stg.centos.org -d '*.stg.centos.org' --challenge-alias stg.acme.centos.org

```

All certs/keys obtained through acme are under /root/.acme.sh/{hostname}/ so you'll then have to import those into this pkistore dir

##### For http challenge 

Normally we prefer DNS challenge, but there are corner cases like delegated records for which that would be problematic. That's the case for {buildlogs,cloud,vault}.centos.org nodes (delegated records to pdns/geoip)

You can add multiple SANs in the same certs. Here is one example with mon.centos.org and SAN mon.j7.centos.org, status.centos.org :

```

certbot certonly --webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email sysadmin@centos.org -d mon.centos.org -d mon.j7.centos.org -d status.centos.org

```

All files (certs/keys) are then available under /etc/letsencrypt/live/ (you'll have to import those into this pkistore dir)

#### How to renew existing certs

##### For DNS challenges (existing records)

For each cert/dns record, we have to ask for a renewal

```

acme.sh --renew-all 

```

##### For HTTP challange

For http challenge it's better to run first with --dry-run, then fix eventual issue and then launch it again for real operations

```

time certbot renew --webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email sysadmin@centos.org --dry-run ; echo return code $?

certbot renew --force-renew --webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email sysadmin@centos.org

```

### Deploying through ansible

Don't forget to have pushed the new/renewed certs/keys into this pkistore directory first.

Important too : Please validate that the -CAChain.crt is linked to correct CA chain. As LetsEncrypt is rotating also their CA, please validate that your .crt is correctly being validated with correct CAchain with simple provided tool in this repository : 

```

./validate_public_chain koji.mbox.centos.org.crt 

Validating cert [koji.mbox.centos.org.crt] with CAChain [koji.mbox.centos.org-CAChain.crt] and default trusted ca-bundle

koji.mbox.centos.org.crt: OK

```

Once it's committed/pushed to pkistore git repo, tobisna (ansible bot) will deploy the renewed TLS certs automatically