.. _harbor-as-system-app-1d1e3ec59823: ========================= Harbor Container Registry ========================= .. rubric:: |context| Harbor is an open-source container registry with a richer and fuller set of capabilities than the built-in |prod| container registry. |prod| end users can use Harbor to manage their own application container images. Harbor secures artifacts with policies and role-based access control, ensures images are scanned and free from vulnerabilities, and signs images as trusted. Harbor has been evolved to a complete |OCI| compliant cloud-native artifact registry. With Harbor V2.0, you can manage images, manifest lists, Helm charts, |CNABs|, |OPAs| and other artifacts adhering to the |OCI| image specification. It supports operations such as pulling, pushing, deleting, tagging, replicating, and scanning these artifacts. Additionally, you can now sign images and manifest lists. Harbor also supports the replication of images between registries and offers advanced security features such as user management, access control, and activity auditing. See https://goharbor.io/docs/2.0.0/ for more details on Harbor. ------------------- Harbor Installation ------------------- .. rubric:: |prereq| - CephFS backed |PVCs| are recommended for Harbor such that when configuring multiple replicas, for |AIO-DX| or Standard configurations, both Harbor replicas can read and write to the registry. - Create a secret as described below: - Generate certificates and create secret. A |CA| cert and server cert creation procedure using cert-manager is specified below: Create the certificate for Harbor using Cert-Manager and using the local |CA|, system-local-ca, as the issuer. Note that the certificate should be created in the ``harbor-tls SECRET`` in the Harbor ``NAMESPACE``. For example: #. Create the following Harbor certificate yaml configuration file: .. code-block:: none ~(keystone_admin)]$ cat < harbor-certificate.yaml --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: harbor-certificate namespace: harbor spec: secretName: harbor-tls issuerRef: name: system-local-ca kind: ClusterIssuer duration: 2160h # 90 days renewBefore: 360h # 15 days commonName: < oam floating IP Address or FQDN > subject: organizations: - ABC-Company organizationalUnits: - StarlingX-harbor ipAddresses: - < oam floating IP address > dnsNames: - < harbor dns> # e.g. harbor.yourdomian.com - < notary dns > # optional, required only if exposed on ingress e.g. notary.yourdomian.com EOF #. Create the Harbor namespace: .. code-block:: none ~(keystone_admin)]$ kubectl create namespace harbor #. Apply the configuration: .. code-block:: none ~(keystone_admin)]$ kubectl apply -f harbor-certificate.yaml #. Verify the configuration: .. code-block:: none ~(keystone_admin)]$ kubectl get certificate harbor-certificate -n harbor After successful configuration, the certificate's Ready status will be True. .. rubric:: |proc| #. Locate the Harbor system application tarball in ``/usr/local/share/applications/helm``. For example: .. code-block:: none /usr/local/share/applications/helm/harbor-.tgz #. Upload the Harbor application. .. code-block:: none ~(keystone_admin)]$ system application-upload /usr/local/share/applications/helm/harbor-.tgz .. _configure-helm-harbor-step: #. Configure the Helm Overrides for Harbor. #. Expose the Harbor application externally with either nodePort or Ingress. **nodePort** #. Create Harbor using NodePort to expose the service .. note:: The instructions below assume that the NodePorts 30002, 30003, and 30004 are available (i.e., not used by any other applications). If these ports are unavailable, please choose and configure alternative ports that are not in use. #. Put the following nodePort overrides in ``values.yaml``: .. code-block:: none expose: type: nodePort # Type should be nodeport tls: enabled: true certSource: secret secret: # Certificate Source is secret secretName: "harbor-tls" # A secret containing tls.crt and tls.key notarySecretName: "harbor-tls" # A secret containing tls.crt and tls.key nodePort: # The name of NodePort service name: harbor ports: http: # The service port Harbor listens on when serving HTTP port: 80 # The node port Harbor listens on when serving HTTP nodePort: 30002 https: # The service port Harbor listens on when serving HTTPS port: 443 # The node port Harbor listens on when serving HTTPS nodePort: 30003 # Only needed when notary.enabled is set to true notary: # The service port Notary listens on port: 4443 # The node port Notary listens on nodePort: 30004 externalURL: https://harbor.yourdomian.com:30003 # URL of harbor listing on 30003 port **Ingress** #. Create Harbor using Ingress to expose the service. .. note:: The instructions below assume that the URL ``harbor.yourdomain.com`` has been configured in the |DNS| server owning ``yourdomain.com`` as the ``OAM FLOATING IP Address`` of |prod|. #. Put the following Ingress overrides in ``values.yaml``: .. code-block:: none expose: type: ingress. # Type should be ingress tls: enabled: true certSource: secret secret: # Certificate Source is secret secretName: "harbor-tls" # Above created secret name notarySecretName: "harbor-tls" # Above created secret name ingress: hosts: core: harbor.yourdomian.com # Harbor Domain name notary: notary.yourdomian.com # Notary Domain name annotations: kubernetes.io/ingress.class: nginx. # Add ingressclass name. It would be "nginx" if you are using default ingress controller. nginx.org/client-max-body-size: "0". # Add this notation for nginx otherwise nginx will reject the image pull & push externalURL: https://harbor.yourdomian.com # URL of harbor #. For |AIO-DX| and Standard setup, add the following ``storageClass`` and ``accessMode`` overrides for |PVC| used for ``Harbor-Jobservice`` and ``Harbor-Registry`` microservice. .. note:: Set the registry size according to your requirements considering the number and size of images that you will have in this registry. Example for nodePort: .. code-block:: none persistence: enabled: true resourcePolicy: "keep" persistentVolumeClaim: registry: existingClaim: "" storageClass: "cephfs" subPath: "" accessMode: ReadWriteMany size: 100Gi annotations: {} jobservice: jobLog: existingClaim: "" storageClass: "cephfs" subPath: "" accessMode: ReadWriteMany size: 1Gi annotations: {} Example for Ingress: .. code-block:: none persistence: enabled: true resourcePolicy: "keep" persistentVolumeClaim: registry: existingClaim: "" storageClass: "cephfs" subPath: "" accessMode: ReadWriteMany size: 100Gi annotations: {} jobservice: jobLog: existingClaim: "" storageClass: "cephfs" subPath: "" accessMode: ReadWriteMany size: 1Gi annotations: {} #. Update the Helm overrides. .. code-block:: none ~(keystone_admin)]$ system helm-override-update harbor harbor harbor --values values.yaml #. Execute Helm overrides. .. code-block:: none ~(keystone_admin)]$ system helm-override-update harbor harbor harbor --values values.yaml #. Apply/Create the Harbor system application. .. code-block:: none ~(keystone_admin)]$ system application-apply harbor ------------------------------------------------- Configure LDAP Authentication for Harbor Registry ------------------------------------------------- .. rubric:: |prereq| - The URL for accessing the Harbor web interface is the ``externalURL`` set in the Helm override above in the step :ref:`Configure Helm Overrides for Harbor `. - The default admin username is 'admin', and the password is 'Harbor12345'. To configure Harbor to use |prod| Local |LDAP| for authentication, follow the instructions in `Configure LDAP/Active Directory Authentication `__ with the following values. For |prod| local |LDAP|: .. code-block:: none LDAP URL: ldap://controller LDAP search DN: cn=ldapadmin,dc=cgcs,dc=local LDAP Search Password: LDAP Base DN: dc=cgcs,dc=local LDAP UID: cn You can find ```` in ``/etc/ldap/slapd.conf.backup``. -------------------------------------- Push an Image to a in Harbor -------------------------------------- .. note:: Depending on your docker setup, you may be required to run all of the following commands with 'sudo'. #. Docker Login. .. code-block:: none docker login -u Where ```` is either: - for 'Ingress' expose: `harbor.yourdomian.com` - for 'NodePort' expose: `https:// :30003` and ```` is your actual username #. Tag the image. .. code-block:: none docker tag redis:latest //redis:latest #. Push the image. .. code-block:: none docker push //redis:latest ------------------------- Pull an Image from Harbor ------------------------- Use command to pull an image: .. code-block:: none docker pull // redis:latest Where ```` is either: - for ``'Ingress' expose: harbor.yourdomian.com`` - for ``'NodePort' expose: https:// :30003`` ---------------------------------- Push a Helm Chart as an OCI Object ---------------------------------- .. code-block:: none helm package helm push oci://// ----------------- Pull a Helm Chart ----------------- .. code-block:: none helm pull oci:///harbor ------------------ Project Management ------------------ A project in Harbor contains all the repositories (images) of a particular application; e.g. the |prod| project in Harbor would contain all the images from |prod|. For more details on creating and configuring a project see: `Create Projects `__ and `Project Configuration `__. --------------------------------- Use Images from a Remote Registry --------------------------------- A project in a local Harbor registry can be used as a pull-through cache for a remote registry. For example, a Harbor registry project on a |prod| Subcloud can be used as a pull-through cache of a Harbor registry project on its |prod| system controller. This can be achieved using a proxy project which acts as a proxy to the remote registry, or using a replication of the remote registry. .. rubric:: Add a proxy project Proxy cache allows you to use Harbor to proxy and cache images from a target public or private registry. It does not allow to push the image by the user. Below are the steps to create a proxy project: #. Create a registry endpoint. See `Creating Replication Endpoints `__. #. Create a new project using the above created registry endpoint. See `Configure Proxy Cache `__. .. rubric:: Add Replication Rule Setting up pull based replication replicates the images from the remote registry based on a trigger. Trigger can be Manual, scheduled, or event based as shown below. .. image:: figures/new-replication-rule.png #. Create a replication endpoint. See `Creating Replication Endpoints `__. #. Create a new replication rule with replication mode as ``pull based`` as describe in `Creating a Replication Rule `__. ------------------------ Add Users to the Project ------------------------ You can add individual users to a project, for more details see `Assign Users to a Project `__. - |LDAP|/AD users can be added to the project. - |LDAP|/AD groups to projects and assign a role to the group. - You can see the various roles and user permission associated with the roles in `User Permissions By Role `__. ------------------------------- Configure Signed Images Support ------------------------------- For more details see: `Implementing Content Trust `__. #. Select cosign or notary for content trust. Harbor will then only allow verified images to be pulled from the project. .. image:: figures/library-harbor.png #. Enable content trust by setting the following environment variables on the machine on which you run the Docker client. .. code-block:: none export DOCKER_CONTENT_TRUST=1 export DOCKER_CONTENT_TRUST_SERVER=https://: If you push the image for the first time, you will be asked to enter the root key passphrase. ------------------------------ Configure Trivy Scanner Plugin ------------------------------ Trivy is installed and configured as a default scanner. .. image:: figures/interrogation-services.png :width: 800 ----------------------------- Configure Size of Registry DB ----------------------------- #. Registry DB size can be configured by setting following in ``values.yaml`` under: .. code-block:: none persistence: registry: size: 5Gi jobservice: jobLog: size: 1Gi #. Set the the value (Default set to 5Gi). .. code-block:: none system helm-override-update harbor harbor harbor --values values.yaml #. Apply the change: .. code-block:: none system application-apply harbor ------------------------------------------------------ Enforcement of Image Security Policies Using Portieris ------------------------------------------------------ Portieris allows to configure trust policies for an individual namespace or cluster-wide and checks the image against a signed image list on a specified notary server to enforce the configured image policies. An Administrator can enforce |prod| image security policies using the Portieris admission controller. It is required to pull from a registry using a ``docker-registry`` secret. Enforcing trust for anonymous image pulls is not supported. To use portieris, an administrator needs to follow below steps: #. Install portieris as specified in :ref:`install-portieris`. #. Configure image policy to allow images from Harbor registry + notary as specified :ref:`portieris-clusterimagepolicy-and-imagepolicy-configuration`. Below example shows the policy allowing image from Harbor registry. .. code-block:: none apiVersion: portieris.cloud.ibm.com/v1 kind: ImagePolicy metadata: name: allow-custom namespace: harbor spec: repositories: - name: ":30003/*" policy: trust: enabled: true trustServer: "https://:30004" # Optional, custom trust server for repository #. Create a SECRET with a Harbor username and password, to use as an ImagePullSecret in a POD spec. .. code-block:: none kubectl create secret docker-registry \ -n harbor harbor-registry-secret \ --docker-server=:port \ --docker-username=admin \ --docker-password=Test@123 .. note:: If the pod creation with the above secret fails, the user should try with new secret with ``--docker-server`` as ````. #. Pull a signed image from Harbor registry in a pod using ``harbor-secret`` created above. Please note that image policy and pod should be created in the same namespace. .. code-block:: none apiVersion: v1 kind: Pod metadata: name: test-pod-public spec: containers: - command: - sleep - '3600' image: :30003/public-demo/redis:latest imagePullPolicy: Always name: test-pod tolerations: - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule" imagePullSecrets: - name: harbor-registry-secret ---------- Limitation ---------- Harbor application cannot be deployed during bootstrap due to the bootstrap deployment dependencies such as early availability of storage class.