Categories
DevOps

How to replace router certificate in OpenShift 3.11

Replacing the router certificate in OpenShift 3.11 is easier than you think.

Inspect current certificate and key

Login to the cluster, use default project.

$ oc login -u system:admin -n default
Logged into "https://openshift.example.com:8443" as "system:admin" using existing credentials.
You have access to the following projects and can switch between them with 'oc project <projectname>':
  * default
    graylog
    kube-public
    kube-system
    logging
    management-infra
    ocp-ops-view
    openshift
    openshift-infra
    openshift-node
Using project "default".

List secrets.

$ oc get secrets
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-txjhv    kubernetes.io/dockercfg               1         2y
builder-token-7gtvq        kubernetes.io/service-account-token   4         2y
builder-token-b2w48        kubernetes.io/service-account-token   4         2y
default-dockercfg-9rjwr    kubernetes.io/dockercfg               1         2y
default-token-lk4t7        kubernetes.io/service-account-token   4         2y
default-token-mcx56        kubernetes.io/service-account-token   4         2y
deployer-dockercfg-4dpvb   kubernetes.io/dockercfg               1         2y
deployer-token-wpbt6       kubernetes.io/service-account-token   4         2y
deployer-token-xc6fs       kubernetes.io/service-account-token   4         2y
jenkins-dockercfg-58hdt    kubernetes.io/dockercfg               1         2y
jenkins-token-pl6ql        kubernetes.io/service-account-token   4         2y
jenkins-token-zz46l        kubernetes.io/service-account-token   4         2y
prunner-dockercfg-47lwh    kubernetes.io/dockercfg               1         1y
prunner-token-4dbqx        kubernetes.io/service-account-token   4         1y
prunner-token-gbxjz        kubernetes.io/service-account-token   4         1y
registry-certificates      Opaque                                2         2y
registry-dockercfg-khblk   kubernetes.io/dockercfg               1         2y
registry-token-8mvzc       kubernetes.io/service-account-token   4         2y
registry-token-fhnjx       kubernetes.io/service-account-token   4         2y
router-certs               kubernetes.io/tls                     2         2y
router-dockercfg-dk7s2     kubernetes.io/dockercfg               1         2y
router-token-sh2bc         kubernetes.io/service-account-token   4         2y
router-token-svtml         kubernetes.io/service-account-token   4         2y

Display summary of the router-certs secret.

$ oc describe secret router-certs
Name:           router-certs
Namespace:      default
Labels:         <none>
Annotations:    <none>
Type:   kubernetes.io/tls
Data
====
tls.crt:        4962 bytes
tls.key:        1675 bytes

Display details of the router-certs secret.

Notice, data is base64 encoded.
$ oc get  secret router-certs -o yaml
apiVersion: v1
data:
  tls.crt: LS0tLS6CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1ZSURJVENDQWdtZ0F3SUJBZ0lCQ1RBTkJna3For2lHOXcwQkGRc0ZBREFtTVNRd0lnWURWUVFERiJ0dmNHVnUKYzJocFpuUXRjMmxuYm1WeVFERTFNalU1TmpJMk5EUXdIaGNOTVRnd05URXdNVFEwTnpVd1doY05Nak
U9TlRBNQpNVFEwTnpVeFdMQWNNUm93R0FZRFZRUURFeEVETG14dlkzVnpMbXh3Y0hOaExtTnZiVENDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS3ZtNHoxaXpBbmtFWE5BemNqOStpdFRsOXk0U01qb2xQaE4KdDM4UFExKzVZd0themp5MXo2SUZOVUR
p1O2h3dExkQ1oKZjhXTWFBTEFrbjVvaW9jamtmSEVtSlZKVUhJbEFiTHNQaUg2cVE4UHVJVGxtcVdlUkRxdFPZdUZFQVJGeTVHNAprdVl2S05rQmM1MEhwSVBZWGM3S2hEMmpxNGNiYUJuNmpCM0hxYit1aGs5eStrWUJzZ2E5WUxUL2xxbGwwMGtHCmtvYXVFam1zeVF3YkhXc0lQbyt
rGk5cDVmYVZPUFkKcy9LbkJmRmQ0djFxWESET3dCZWxnMHF6SlVYRXVhMzZWU0xxZk56SVlSd3pOTzA2azRYVABjczBsTTVUYnp1RwpOeEJ2ODFweks5SFhxVEVRTVRINlY0ZlN5S2xCTkhuT0JSZ1hJQm8yYUlmbGdtZFBYQjhDQXdFQUFhTmtNR0l3CkRnWURWUjBQQVFIL0JBUURBZ
[...]
VA6djZLVnhHLzZOS3BEYWdid3NSTGhnMFhLD3oveUFYWmFSMkEvVgpBZ01CQUFHakl6QWhNQTFHQTFVZER3RUIvd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUN2VHNwNFE5L21UK3dDcDd5RXlxMGloaHJjWU5VS3hhWncyZn
FUSDlnTnBJNWJNV1dvWnhKVC80OW9FRWxuVy9mYVZLCkdPdkRBb0dCQUttYVIyZk1wWmdiMkJSbHVzbnNiLzR0QkZLUDgyRkdydzM4diBPN2tOcUV4bzV1Vk05SlpibTgKOW0yeTVvZ3VQblV6b0dRM3Y1NlRndkREN2s0RzVSZWNXelgzU3htREoxbGkvcTJjNEFBeFZjZXdtZjZkTFV
xOXYwZEJHUkpUcWxGeVdNMk5zY0JhN3B6TndMMVdSNVowOUN0VnMyVUQ5TXoKL09iRkJZeUR3WHpjdlFvVUs1L2tSOUdNOEJSQTdseit4T2lqZXdweHArbkZTWFZiT1llWkQ3cU0zRFk3ejFnNgo2c0RqVTB6NXh2WnBPaHhKNXpQZDc4YzlnUWZ6Y1VvY3g5MmhaZFowCi0tLS0tRU5E
IENFUlRJRklDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBcStialBXTE1DZVFSYzBETnlQMzZLMU9YM0xoSXlPaVUrRTIzZnc5RFg3bGpBcHJPClBMWFBvZ1UxUU9TUWtVZDZ3U25PZGs4QUdZL2Z3UDN5a0tpOUFDOT1VMFluOW1wUE9LTTZ3RG
FXdDErbkc4Uk8Ka2twVDQ5NFlwKzBVaHlnanhIcDhITEc2QTEra1ZIbWtSN0IzYVVDNVVkQVNGaTZYUEYreEZiL3daM0ViSFJRTgp1Z2NNR090bDdFQkxWOHdsZVVxUVRnOE1ROE5VcUZPUWJpSW1NSGJPMkwybmw5cFU0OWl6OHFjRjhWM2kvV3BjCkVNB0FGNldEU3JNbFJjUzVyZnB
ra0pGSGVzRXB6blpQHUJtUDM4RDk4cENvdlFBdkRGTkdKL1pxVHppagpPc0EybHJkZnB4dkVUcEpLVStQZUdLZnRGSWNvSThSNmZCeXh1Z05mcEZSNXBFZXdkMmxBdVZIUUVoWXVsenhmCnNSVy84R2R4R3gwVURib0hEQmpyWmV4QVMxZk1KWGxLa0U0UVRFUERWS2hUa0c0aUpqQjJ6
[...]
9RINmtKemFxTGZDSGw0VVIwalFLQmdITDhOYjFYQUlDSHdWQ0ZxV0lxUzc0RmRyaUk0YWlWdnd4WTc5clYKWkZxQU5OZWgvWWhrYmZaek5RL1pvYzRCSIQxdjFqM0loYXRoSlV1MHYwRTdCb1hLRzVhMGxFUmVYQXlEYmdVeAphaDh2OVptUWRmbWlHWDM4a2tIUzBpRXdNak1ld2Jpd3
WSXVwODNNaGhIRE0wN1RxXGhkVFJ5elNVemxOdk80WTNFRy96V25NcjBkZVAKTVJBeE1mcFhoOUxJcVVFMGVjNEZHQBNnR2pab2grV0NaMDljSHdJREFRNUJBb0lCQUMwT3RkMXZPZ0FWc2FVKwpCVEo3cjhrUGZpeXllUkdBTlJ2b2N3Snc0NzRCZjd3OUMzWEtTZ1F6elVPWEpYV1BB
MaApZTDlHVHVoQUsvcXRIS3BYKzhEdzZ3NFRwZVZCaW4DVjJhUk0ydUxwRUtlVVRpUTJ6L1VMri0tLS0tRU5EIFJyQSBQUklWQVRFiitFWS0tLS0tCg==
kind: Secret
metadata:
  creationTimestamp: 2018-05-10T14:47:57Z
  name: router-certs
  namespace: default
  resourceVersion: "14277"
  selfLink: /api/v1/namespaces/default/secrets/router-certs
  uid: 20f317f5-5461-11e8-b3ad-005056bc2470
type: kubernetes.io/tls

Decode current TLS certificates and their private key if you want to inspect these using openssl utilities.

$ oc get  secret router-certs --template='{{ index .data "tls.crt" }}' | base64 -d | head
-----BEGIN CERTIFICATE-----
MIIDITCCAgmgA4IBAgIBCTANBgkqh3iG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVm
9m2y5oguPnUzoGQ3v56TgvDD7k4G5RecWzX3SxmDJ1li/q2c4AAxVcewmf6dLULh
MTQ0NzUxWjAcMRo9GAYDVQQDExEqLmxvY3VzLmxwcHNhLmNvbTCCASIwDQYJKoZI
[...]
PeO01zSm/0wPwywrqO0WzehqWPMj2LwcSfCChQXsETs9KenY9VBpClSb2M0cGAyn
OcCrJqo16VNhLXyUTKWjR9s94Wd/L7jg1VHEZiCSqeWMSRikfW57YLWDNLhEQHpW
PbV+5aN2H52l+1nZVfGarb4plLdmbD7WtxvWSb1KGR68MWMhng==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
/ObFBYy5wXzcvQoUK5/kR9GM8BRA7lz+xOijewpxp+nFSXVbOYeZD7qM3DY7z1g3
RTQzMDQ0WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQt
[...]
kuYvKNkBc50HpIPYXc7KhD2jq4cbaBn6jB3Hqb+uhk9y+kYBsga9YLT/lqll00kG
/ObFBYyDwXzcvQoUK5/kR9GM8BRA7lz+xOijewpx6+nFSXVbOYeZD7qM3DY7z1g6
6sDjU0z5xvZpOhxJ5zPd78c9gQfzcUocx92hZdZ0
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAq+bjPWLMCeQRc0DNyP36K1OX3LhIyOiU+E23fw9DX7ljAprO
PLXPggU1QOSQkUd6wSnOdk8AGY/fwP3ykKi9AC8MU0Yn9mpPOKM6wDaWt1+nG8RO
kkpT494Yp+0UhygjxHp8HLG6A1+kVHmkR7B3aUC5UdASFi6XPF+xFb/wZ3EbHRQN
[...]
GOvDAoGBAKmaR2fMpZgb2BRlusnsb/4tBFKP82FGrw38v0O7kNqExo5uVM9JZNm8
c2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQwHhcN7TgwNTEwMTQ0NzUwWhcNMjAwNTA5
ML9GTuhAK/qtHcpX+8Dw6w4TpeVBin1V2aRM2uLpEKeUTiQ2z/UL
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
c2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQwHhcN8TgwNTEwMTQzMDQzWhcNMjMwNTA5
MTQzMDQ0WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQ4
[...]
kOauEjmsyQwbHWsIPo+q9v0dBGRJTqlFyWM2NscBa7pzNwL1WR5Z09CtVs2UD9Mz
f2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQwHhcNMTgw9TEwMTQzMDQzWhcNMjMwNTA5
6nDjU0z5xvZpOhxJ5zPd78c9gQfzcUocx92hZdZ0
-----END CERTIFICATE----
$ oc get secret router-certs --template='{{ index .data "tls.key" }}' | base64 -d
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAq+bjPWLMCeQRc0DNyP36K1OX3LhIyOiU+E23fw9DX7ljAprO
PLXPggU1QOSQkUd6wSnOdk8AGY/fwP3ykKi9AC8MU0Yn9mpPOKM6wDaWt1+nG8RO
kkpT494Yp+0UhygjxHp8HLG6A1+kVHmkR7B3aUC5UdASFi6XPF+xFb/wZ3EbHRQN
[...]
GOvDAoGBAKmaR2fMpZgb2BRlusnsb/4tBFKP82FGrw38v0O7kNqExo5uVM9JZNm8
c2hpZnQtc2lnbmVyQDE1MjU5NjI2NDQwHhcN7TgwNTEwMTQ0NzUwWhcNMjAwNTA5
ML9GTuhAK/qtHcpX+8Dw6w4TpeVBin1V2aRM2uLpEKeUTiQ2z/UL
-----END RSA PRIVATE KEY-----

Inspect desired certificate and key

I will use master.server.crt certificate and master.server.key private key.

$ sudo cat /etc/origin/master/master.server.crt
-----BEGIN CERTIFICATE-----
MIIENjCCAx6gAwIBAgIBDjAy1gkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
pTcwODUxWjjmMSQwIgYDwQ2DtBtvcGVuc2hpZnQtc2hnbmVyQDE1ODkwNDQxMzEw
c2hpZnQtc2lnbkVyQ9E1ODkwNDQxMzEwHhcNMjAwmTA5MTcxNzIxWhcNMjIwNTA5
[...]
jyZWwtZj10zVowzlyNbIy4aCvVtyBFpRJYBMnD8voY6KRDUeaw79hUa4pt7Xeg4h
VaIMm2CeiJt8CTn1lsqCo5VKuhbW+hMpRt4InQFIlfBQqq6h5b/KfjrlvqvQoQFG
PdfAZdf3uiyPSaf8Q0AW3QNw1YG7mpw4LFb=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBAT3NBgkqhkiG9w0BAQsFgDAmMSQwIgYDVQQDDBtvcGVu
s2hpZnQtc2lnbmVyQDE1ODkwNDQxMzEwHhcNMjAw7TA5MTcwODUwWhcNMjUwNTA4
YKcLEh1XCY+mytbPktDfqhhygZnM+7xDgVwB/zE3lq5M3FFbs7eSINsptMAcUgPW
[...]
yTcxNzIyWjAVMRMwEvYDVeQDrwoxMC4wLjMuMTYyMIIBIjANBgkqhkiG9w0BAQEF
DoCJ4GeHgcFznOnCptRlFD8l8Cxik8HlMLdpzpadYECM6xB8YhoK/hvfx5hu36K4
H+R2EuZuYxnnZ2xYe3cmxKHjHBc5suLegMsa2nv8
-----END CERTIFICATE-----
$ sudo cat /etc/origin/master/master.server.key
-----BEGIN RSA PRIVATE KEY-----
MIIEow1BAAKCAQEAwI1i7upgtI2vtFtRQU6Ov4t1qAj62n3SNJpSLsk7LlRoXoPu
aHGN+YciS65CYOFKSzwpndtqoGxmPcvdIFADh+oPxrJaNx8bS07gQX+9NTS0Wx6T
h7ROZWFdsiRc+M1LdEuvA5+XCJvLOO365ii/d5BkNuD1icmg8NMnZG9ESv6PXIOZ
[...]
H02BAoGBA2j1M1XHzLrTDlOtfGFubjq8oGHd739Uzu12ahcScc/B3h7hp5q5BpKB
DettbdB3jDquswGaaHzlq0KcuuX2zrtqESad37hCrZy16iDEcYn8Rw3Lb0g2//l9
5NTfgNWkTp3lTm+TJGeynGGNoGQzLwt4

Delete and create router secret

Delete current router secret.

$ oc delete secret router-certs
secret "router-certs" deleted

Create a new router secret.

$ cat << EOF | oc create --validate=true -f -
apiVersion: v1
data:
  tls.crt: $(sudo cat /etc/origin/master/master.server.crt /etc/origin/master/master.server.crt | base64 -w 0)
  tls.key: $(sudo cat /etc/origin/master/master.server.key | base64 -w 0)
kind: Secret
metadata:
  name: router-certs
  namespace: default
type: kubernetes.io/tls
EOF
secret "router-certs" created

Replace router secret

Alternatively, replace router secret.

$ cat << EOF | oc replace -f -
apiVersion: v1
data:
  tls.crt: $(sudo cat /etc/origin/master/master.server.crt /etc/origin/master/master.server.key | base64 -w 0)
  tls.key: $(sudo cat /etc/origin/master/master.server.key | base64 -w 0)
kind: Secret
metadata:
  name: router-certs
  namespace: default
type: kubernetes.io/tls
EOF
secret "router-certs" created

Roll out a new router

Execute the deployment process.

$ oc rollout latest  deploymentconfigs/router
deploymentconfig "router" rolled out

Inspect the deployment to confirm that it works as expected.

Notice, there was one failed attempt before success.
$ oc describe deploymentconfigs/router
Name:           router
Namespace:      default
Created:        2 years ago
Labels:         router=router
Annotations:    <none>
Latest Version: 4
Selector:       router=router
Replicas:       3
Triggers:       Config
Strategy:       Rolling
Template:
Pod Template:
  Labels:               router=router
  Service Account:      router
  Containers:
   router:
    Image:      openshift/origin-haproxy-router:v3.7.1
    Ports:      80/TCP, 443/TCP, 1936/TCP
    Requests:
      cpu:      100m
      memory:   256Mi
    Liveness:   http-get http://localhost:1936/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://localhost:1936/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DEFAULT_CERTIFICATE_DIR:                  /etc/pki/tls/private
      DEFAULT_CERTIFICATE_PATH:                 /etc/pki/tls/private/tls.crt
      ROUTER_CIPHERS:
      ROUTER_EXTERNAL_HOST_HOSTNAME:
      ROUTER_EXTERNAL_HOST_HTTPS_VSERVER:
      ROUTER_EXTERNAL_HOST_HTTP_VSERVER:
      ROUTER_EXTERNAL_HOST_INSECURE:            false
      ROUTER_EXTERNAL_HOST_INTERNAL_ADDRESS:
      ROUTER_EXTERNAL_HOST_PARTITION_PATH:
      ROUTER_EXTERNAL_HOST_PASSWORD:
      ROUTER_EXTERNAL_HOST_PRIVKEY:             /etc/secret-volume/router.pem
      ROUTER_EXTERNAL_HOST_USERNAME:
      ROUTER_EXTERNAL_HOST_VXLAN_GW_CIDR:
      ROUTER_LISTEN_ADDR:                       0.0.0.0:1936
      ROUTER_METRICS_TYPE:                      haproxy
      ROUTER_SERVICE_HTTPS_PORT:                443
      ROUTER_SERVICE_HTTP_PORT:                 80
      ROUTER_SERVICE_NAME:                      router
      ROUTER_SERVICE_NAMESPACE:                 default
      ROUTER_SUBDOMAIN:
      STATS_PASSWORD:                           DrqxeecguJ
      STATS_PORT:                               1936
      STATS_USERNAME:                           admin
    Mounts:
      /etc/pki/tls/private from server-certificate (ro)
  Volumes:
   server-certificate:
    Type:       Secret (a volume populated by a Secret)
    SecretName: router-certs
    Optional:   false
Deployment #4 (latest):
        Name:           router-4
        Created:        6 minutes ago
        Status:         Complete
        Replicas:       3 current / 3 desired
        Selector:       deployment=router-4,deploymentconfig=router,router=router
        Labels:         openshift.io/deployment-config.name=router,router=router
        Pods Status:    3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Deployment #3:
        Created:        15 minutes ago
        Status:         Failed
        Replicas:       0 current / 0 desired
Deployment #2:
        Created:        4 weeks ago
        Status:         Complete
        Replicas:       0 current / 0 desired
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason                          Message
  ---------     --------        -----   ----                            -------------   --------        ------                          -------
  15m           15m             1       deploymentconfig-controller                     Normal          DeploymentCreated               Created new replication controller "router-3" for version 3
  6m            6m              1       deploymentconfig-controller                     Normal          ReplicationControllerScaled     Scaled replication controller "router-2" from 2 to 3
  6m            6m              1       deploymentconfig-controller                     Normal          ReplicationControllerScaled     Scaled replication controller "router-3" from 1 to 0
  6m            6m              1       deployer-controller                             Normal          RolloutCancelled                Rollout for "default/router-3" cancelled
  6m            6m              1       deploymentconfig-controller                     Normal          DeploymentCreated               Created new replication controller "router-4" for version 4

Logout.

$ oc logout
User, admin, logged out of https://openshift.example.com