1. Harbor
1.1. 정의
Harbor는 정책 및 역할 기반 액세스 제어로 아티팩트를 보호하고, 이미지를 스캔하고 취약성이 없도록 하며, 이미지를 신뢰할 수 있는 것으로 서명하는 오픈 소스 레지스트리이다. Harbor는 규정 준수, 성능 및 상호 운용성을 제공하여 Kubernetes 및 Docker와 같은 클라우드 네이티브 컴퓨팅 플랫폼에서 아티팩트를 일관되고 안전하게 관리할 수 있도록 지원한다.
1.2. 설치
1) 관리 디렉토리 ch13생성
| [kevin@ubu22-01 ~ (kubernetes-admin@cluster.local:kube-system)]$ mkdir ch13 |
2) helm에 harbor repo추가
| [kevin@ubu22-01 ~ (kubernetes-admin@cluster.local:kube-system)]$ cd ch13 [kevin@ubu22-01 ~/ch13 (kubernetes-admin@cluster.local:kube-system)]$ helm repo add harbor https://helm.goharbor.io "harbor" has been added to your repositories |
3) harbor 이미지 다운로드
| [kevin@ubu22-01 ~/ch13 (kubernetes-admin@cluster.local:kube-system)]$ helm pull harbor/harbor [kevin@ubu22-01 ~/ch13 (kubernetes-admin@cluster.local:kube-system)]$ ll total 56 drwxrwxr-x 2 kevin kevin 4096 8월 23 18:03 ./ drwxrwxr-x 4 kevin kevin 4096 8월 23 18:03 ../ -rw-r--r-- 1 kevin kevin 48932 8월 23 18:03 harbor-1.12.4.tgz |
4) 압축 해제
| [kevin@ubu22-01 ~/ch13 (kubernetes-admin@cluster.local:kube-system)]$ tar xzvf harbor-1.12.4.tgz [kevin@ubu22-01 ~/ch13 (kubernetes-admin@cluster.local:kube-system)]$ ll total 60 drwxrwxr-x 3 kevin kevin 4096 8월 23 18:04 ./ drwxrwxr-x 4 kevin kevin 4096 8월 23 18:03 ../ drwxrwxr-x 4 kevin kevin 4096 8월 23 18:04 harbor/ -rw-r--r-- 1 kevin kevin 48932 8월 23 18:03 harbor-1.12.4.tgz [kevin@ubu22-01 ~/ch13 (kubernetes-admin@cluster.local:kube-system)]$ rm -rf harbor-1.12.4.tgz |
5) harbor설정 파일 변경
| [kevin@ubu22-01 ~/ch13 (kubernetes-admin@cluster.local:kube-system)]$ cd ~/ch13/harbor # 원본파일 보존 [kevin@ubu22-01 ~/ch13 (kubernetes-admin@cluster.local:kube-system)]$ cp values.yaml my-values.yaml # conf 병경 [kevin@ubu22-01 ~/ch13 (kubernetes-admin@cluster.local:kube-system)]$ vi my-values.yaml --- 1 expose: 2 # Set how to expose the service. Set the type as "ingress", "clusterIP", "nodePort" or "loadBalancer" 3 # and fill the information in the corresponding section 4 type: loadBalancer //metallb 설치했기에 lb로 변경 5 tls: ... 20 auto: 21 # The common name used to generate the certificate, it's necessary 22 # when the type isn't "ingress" 23 commonName: "harbor.myweb.io" //접속 도메인 네임 ... 98 loadBalancer: 99 # The name of LoadBalancer service 100 name: harbor 101 # Set the IP if the LoadBalancer supports assigning IP 102 IP: "192.168.100.70" //metallb 대역폭에서 지정 ippools 103 ports: ... 126 # If Harbor is deployed behind the proxy, set it as the URL of proxy 127 externalURL: https://harbor.myweb.io ... 206 # Use the existing PVC which must be created manually before bound, 207 # and specify the "subPath" if the PVC is shared with other components 208 existingClaim: "" 209 # Specify the "storageClass" used to provision the volume. Or the default 210 # StorageClass will be used (the default). 211 # Set it to "-" to disable dynamic provisioning 212 storageClass: "local-path" //로컬 스토리지 지정 213 subPath: "" 214 accessMode: ReadWriteOnce ... |
6) harbor 관리 네임 스페이스 생성
| [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:default)]$ k create ns harbor namespace/harbor created [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:default)]$ k ns harbor Context "kubernetes-admin@cluster.local" modified. Active namespace is "harbor". |
7) 변경한 설정으로 harbor 설치
| [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:harbor)]$ helm install harbor -f my-values.yaml . NAME: harbor LAST DEPLOYED: Mon Aug 21 10:20:57 2023 NAMESPACE: harbor STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Please wait for several minutes for Harbor deployment to complete. Then you should be able to visit the Harbor portal at https://core.harbor.domain For more details, please visit https://github.com/goharbor/harbor |
8) 설치된 harbor 확인
| [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:harbor)]$ kgp NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES harbor-core-646ffc7675-xqlsf 1/1 Running 8 (43m ago) 30h 10.233.88.87 ubu22-02 <none> <none> harbor-database-0 1/1 Running 7 (43m ago) 30h 10.233.88.119 ubu22-02 <none> <none> harbor-jobservice-5899fb8dd7-fgdgt 1/1 Running 20 (41m ago) 30h 10.233.88.112 ubu22-02 <none> <none> harbor-nginx-8bc46f5ff-z9vc2 1/1 Running 6 (43m ago) 30h 10.233.88.105 ubu22-02 <none> <none> harbor-notary-server-5bd87f97d-dqjwd 1/1 Running 13 (43m ago) 30h 10.233.88.103 ubu22-02 <none> <none> harbor-notary-signer-84f77f45df-jxwxc 1/1 Running 11 (41m ago) 30h 10.233.88.78 ubu22-02 <none> <none> harbor-portal-6d694b876b-242wq 1/1 Running 4 (43m ago) 30h 10.233.88.118 ubu22-02 <none> <none> harbor-redis-0 1/1 Running 4 (43m ago) 30h 10.233.88.80 ubu22-02 <none> <none> harbor-registry-78648cd977-24xkz 2/2 Running 10 (43m ago) 30h 10.233.88.79 ubu22-02 <none> <none> harbor-trivy-0 1/1 Running 4 (43m ago) 30h 10.233.88.92 ubu22-02 <none> <none> # 접속 ip확인 [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:harbor)]$ kgs NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR harbor LoadBalancer 10.233.31.181 192.168.100.70 80:31683/TCP,443:32523/TCP,4443:31083/TCP 30h app=harbor,component=nginx,release=harbor harbor-core ClusterIP 10.233.45.126 <none> 80/TCP 30h app=harbor,component=core,release=harbor harbor-database ClusterIP 10.233.44.221 <none> 5432/TCP 30h app=harbor,component=database,release=harbor harbor-jobservice ClusterIP 10.233.57.198 <none> 80/TCP 30h app=harbor,component=jobservice,release=harbor harbor-notary-server ClusterIP 10.233.22.212 <none> 4443/TCP 30h app=harbor,component=notary-server,release=harbor harbor-notary-signer ClusterIP 10.233.21.125 <none> 7899/TCP 30h app=harbor,component=notary-signer,release=harbor harbor-portal ClusterIP 10.233.1.231 <none> 80/TCP 30h app=harbor,component=portal,release=harbor harbor-redis ClusterIP 10.233.39.230 <none> 6379/TCP 30h app=harbor,component=redis,release=harbor harbor-registry ClusterIP 10.233.12.6 <none> 5000/TCP,8080/TCP 30h app=harbor,component=registry,release=harbor harbor-trivy ClusterIP 10.233.48.110 <none> 8080/TCP 30h app=harbor,component=trivy,release=harbor |
1.3. 로컬 저장소 구축
9) 사용할 레지스터리에 harbor 등록 (모든 노드에 설정)
| [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:harbor)]$ sudo vi /etc/containerd/config.toml # 레지스트리 설정 방법 https://github.com/containerd/containerd/blob/main/docs/hosts.md --- version = 2 root = "/var/lib/containerd" state = "/run/containerd" oom_score = 0 [grpc] max_recv_message_size = 16777216 max_send_message_size = 16777216 [debug] level = "info" [metrics] address = "" grpc_histogram = false [plugins] [plugins."io.containerd.grpc.v1.cri"] sandbox_image = "registry.k8s.io/pause:3.9" max_container_log_line_size = -1 enable_unprivileged_ports = false enable_unprivileged_icmp = false [plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc" snapshotter = "overlayfs" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] runtime_type = "io.containerd.runc.v2" runtime_engine = "" runtime_root = "" base_runtime_spec = "/etc/containerd/cri-base.json" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] systemdCgroup = true [plugins."io.containerd.grpc.v1.cri".registry] endpoint = ["https://harbor.myweb.io"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://registry-1.docker.io"] [plugins."io.containerd.grpc.v1.cri".registry.configs] [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.myweb.io".tls] insecure_skip_verify = true ... |
10) /etc/hosts지정
| [kevin@ubu22-01 ~/Desktop (kubernetes-admin@cluster.local:harbor)]$ sudo vi /etc/hosts --- . . . 192.168.100.70 harbor.myweb.io ... |
11) 레지스터리 등록 위치 확인 (모든 노드에 설정)
| # 만약 존재하지 않는다면 생성하면 된다. [kevin@ubu22-01 ~/Desktop (kubernetes-admin@cluster.local:harbor)]$ cd /etc/containerd/certs.d/_default/ [kevin@ubu22-01 /etc/containerd/certs.d/_default (kubernetes-admin@cluster.local:harbor)]$ ll total 12 drwxr-xr-x 2 root root 4096 8월 22 11:25 ./ drwxr-xr-x 3 root root 4096 8월 22 11:25 ../ -rw-r--r-- 1 root root 137 8월 22 11:25 hosts.toml [kevin@ubu22-01 /etc/containerd/certs.d/_default (kubernetes-admin@cluster.local:harbor)]$ vi hosts.toml --- server = "https://harbor.myweb.io" [host."https://harbor.myweb.io"] capablities = ["pull", "resolve", "push"] skip_verify = true ... |
12) 컨테이너 재시작 (모든 노드에 설정)
| [kevin@ubu22-01 /etc/containerd/certs.d/_default (kubernetes-admin@cluster.local:harbor)]$ sudo systemctl restart containerd.service [kevin@ubu22-01 /etc/containerd/certs.d/_default (kubernetes-admin@cluster.local:harbor)]$ sudo systemctl status containerd.service ● containerd.service - containerd container runtime Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2023-08-24 10:23:47 KST; 11min ago |
13) 등록된 레지스터리 확인
| [kevin@ubu22-01 /etc/containerd/certs.d/_default (kubernetes-admin@cluster.local:harbor)]$ sudo nerdctl images REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE grafana/promtail 2.8.3 48e658e500dd 42 hours ago linux/amd64 198.4 MiB 71.0 MiB grafana/promtail <none> 48e658e500dd 42 hours ago linux/amd64 198.4 MiB 71.0 MiB kubernetesui/dashboard v2.7.0 2e500d29e9d5 7 days ago linux/amd64 245.8 MiB 72.3 MiB kubernetesui/metrics-scraper v1.0.8 76049887f07a 7 days ago linux/amd64 41.8 MiB 18.8 MiB nginx latest 104c7c5c54f2 12 days ago linux/amd64 191.8 MiB 67.2 MiB nginx <none> 104c7c5c54f2 2 days ago linux/amd64 191.8 MiB 67.2 MiB nginx <none> 67f9a4f10d14 12 days ago linux/amd64 192.1 MiB 67.3 MiB harbor.myweb.io/library/nginx v1.25 104c7c5c54f2 2 days ago linux/amd64 191.8 MiB 67.2 MiB quay.io/calico/cni v3.25.1 9a2c99f03140 2 weeks ago linux/amd64 192.0 MiB 85.7 MiB quay.io/calico/kube-controllers v3.25.1 02c1232ee4b8 2 weeks ago linux/amd64 69.6 MiB 30.4 MiB quay.io/calico/node v3.25.1 0cd00e83d06b 2 weeks ago linux/amd64 246.2 MiB 84.2 MiB quay.io/calico/pod2daemon-flexvol v3.25.1 66629150669c 2 weeks ago linux/amd64 14.3 MiB 6.8 MiB quay.io/metallb/controller v0.13.9 c9ffd7215dcf 7 days ago linux/amd64 61.7 MiB 26.6 MiB quay.io/metallb/speaker v0.13.9 ed242e213112 7 days ago linux/amd64 109.1 MiB 47.8 MiB quay.io/prometheus/node-exporter v1.6.0 d2e48098c364 43 hours ago linux/amd64 24.0 MiB 11.2 MiB quay.io/prometheus/node-exporter <none> d2e48098c364 43 hours ago linux/amd64 24.0 MiB 11.2 MiB registry.k8s.io/coredns/coredns v1.9.3 8e352a029d30 2 weeks ago linux/amd64 47.0 MiB 14.2 MiB registry.k8s.io/cpa/cluster-proportional-autoscaler v1.8.8 69bf675e3567 2 weeks ago linux/amd64 40.4 MiB 11.1 MiB registry.k8s.io/dns/k8s-dns-node-cache 1.22.20 d2504aceb7db 2 weeks ago linux/amd64 69.0 MiB 29.1 MiB registry.k8s.io/kube-apiserver v1.26.7 c3b8fbd0418e 2 weeks ago linux/amd64 133.4 MiB 34.5 MiB registry.k8s.io/kube-controller-manager v1.26.7 a71663d42f50 2 weeks ago linux/amd64 123.0 MiB 31.4 MiB registry.k8s.io/kube-proxy v1.26.7 8d998d77a1fa 2 weeks ago linux/amd64 67.6 MiB 20.8 MiB registry.k8s.io/kube-scheduler v1.26.7 291877ba8db0 2 weeks ago linux/amd64 58.7 MiB 17.1 MiB registry.k8s.io/pause 3.9 7031c1b28338 2 weeks ago linux/amd64 732.0 KiB 314.0 KiB |
1.4. harbor 이미지 사용
14) harbor에 이미지 등록 및 사용
| # nginx이미지를 다운 받는다. [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:nginx)]$ sudo crictl pull nginx Image is up to date for sha256:eea7b3dcba7ee47c0d16a60cc85d2b977d166be3960541991f3e6294d795ed24 # 받아온 이미지에 tag를 붙인다. [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:nginx)]$ sudo nerdctl tag nginx harbor.myweb.io/library/nginx:v1.25 # tag를 붙인 이미지를 harbod에 업로드 한다. [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:nginx)]$ sudo nerdctl push harbor.myweb.io/library/nginx:v1.25 INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:b2c2d82d177a85cf5594a47c977b5d3fe62264b92558ce0c4f092414003eb9c8) index-sha256:b2c2d82d177a85cf5594a47c977b5d3fe62264b92558ce0c4f092414003eb9c8: done |++++++++++++++++++++++++++++++++++++++| manifest-sha256:48a84a0728cab8ac558f48796f901f6d31d287101bc8b317683678125e0d2d35: done |++++++++++++++++++++++++++++++++++++++| config-sha256:eea7b3dcba7ee47c0d16a60cc85d2b977d166be3960541991f3e6294d795ed24: done |++++++++++++++++++++++++++++++++++++++| elapsed: 0.4 s total: 10.0 K (25.0 KiB/s) # harbor에서 nginx이미지를 run한다. [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:nginx)]$ k run nginx --image harbor.myweb.io/library/nginx:v1.25 pod/nginx created # 생성된 pod확인 [kevin@ubu22-01 ~/Documents/loki (kubernetes-admin@cluster.local:nginx)]$ kgp NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-7d959d6988-8d5ck 2/2 Running 13 (161m ago) 43h 10.233.109.35 ubu22-03 <none> <none> nginx 1/1 Running 0 60m 10.233.88.67 ubu22-02 <none> <none> |
1.5. WEB에서 확인
15) 로그인 정보 확인
| [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:nginx)]$ sudo nerdctl login harbor.myweb.io WARNING: Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded # 암호화된 정보 확인(Base64) [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:harbor)]$ sudo vi /root/.docker/config.json --- { "auths": { "harbor.myweb.io": { "auth": "YWRtaW46SGFyYm9yMTIzNDU=" } } } ... # 디코드로 로그인 정보 확인 [kevin@ubu22-01 ~/ch13/harbor (kubernetes-admin@cluster.local:harbor)]$ echo YWRtaW46SGFyYm9yMTIzNDU= | base64 -d admin:Harbor12345 |
16) 로그인
17) 이미지 확인
18) 이미지 취약점 검사 기능
