使用舵踩坑部署港口1,我的环境1、centos 7.92、k8s集群13、19.034、关闭所有防火墙2、安装配置头盔(k8s-主头盔节点操作)1、安装helm3下载舵的包wget https://get.helm.sh/helm-v3.2.4-linux-amd64.tar.gz解压塔尔-zxf helm-v3.2.4-linux-amd64.tar.gz使用舵命令CP Linux-amd64/helm/usr/local/bin/2、验证赫尔姆$赫尔姆版本版本4 . BuildInfo { version : ' v 3 . 2 . 4 '，git commit : ' 0ad 800 ef 43d 3b 826 f 31 a5 ad 8 dfbb 4 Fe 05d 143688 '，GitTreeState:'clean '，gov : ' go 1。13 .12 ' } 3,命令补全来源（头盔完成猛击)3,安装配置港口1,添加港口图仓库头盔回购添加港口https://头盔。去港口。io舵报告ls2，搜索海港的查特海姆搜索回购港3,拉取赫尔姆拉港/海港-版本1.4.14,解压塔尔白兰地规格港-1.4.1.tgz注意：如果仓库添加不了请使用饭桶克隆方式 　　

　　

　　饭桶克隆https://github.com/goharbor/harbor-helmcd港-赫尔姆吉特结帐1.4.1或https://github.com/goharbor/harbor-helm/releases/tag/v1.4.1直接下载水手包放到服务器进行解压tar xf harbor-helm-1.4.1.tar.gz我是用的下载水手包到服务器的5、修改配置$维姆港-圣盔-1。4 .1/数值。YAML 179坚持：180启用：真181 #将其设置为"保持"以避免在赫尔姆删除182 #操作期间删除聚氯乙烯.将其留空将在图表删除后删除PVC 183资源策略：“保留”184持久卷声明：185注册表：186 #使用绑定前必须手动创建的现有PVC，187 #如果聚氯乙烯与其他组件共享，则指定"子路径“188现有权利要求3360‘harbor-PVC’189 #指定用于配置卷的“存储类”.或者将使用默认值190 #存储类(默认值)。191 #将其设置为"-"以禁用动态资源调配192存储类：""193子路径： "注册表" 194访问模式：读写一次195大小： 20gi 196图表博物馆3360197现有权利要求：'harbor-PVC'198存储类：""199子路径360 　　

2     jobservice:203       existingClaim: "harbor-pvc"204       storageClass: "jobservice"205       subPath: ""206       accessMode: ReadWriteOnce207       size: 20Gi208     # If external database is used, the following settings for database will209     # be ignored210     database:211       existingClaim: "harbor-pvc"212       storageClass: ""213       subPath: "database"214       accessMode: ReadWriteOnce215       size: 20Gi216     # If external Redis is used, the following settings for Redis will217     # be ignored218     redis:219       existingClaim: "harbor-pvc"220       storageClass: ""221       subPath: "redis"222       accessMode: ReadWriteOnce223       size: 20Gi224     trivy:225       existingClaim: "harbor-pvc"226       storageClass: ""227       subPath: "trivy"228       accessMode: ReadWriteOnce229       size: 20Gi522 clair:523   enabled: false552 trivy:553   # enabled the flag to enable Trivy scanner554   enabled: false606 notary:607   enabled: false6、创建pv及PVC1、安装nfs服务端 （k8s集群master节点）yum -y install nfs-utils rpcbindmkdir -p /data/k8s && chmod 755 /data/k8secho '/data/k8s *(insecure,rw,sync,no_root_squash)'>>/etc/exportssystemctl enable rpcbind && systemctl start rpcbind客户端（k8s集群slave节点）yum -y install nfs-utils rpcbindmkdir /nfsdatamount -t nfs 192.168.10.101:/data/k8s /nfsdata2、创建命名空间k8s-master上操作

　　

kubectl create namespace harbor3、创建pvccat >harbor-pvc.yaml <<EOFapiVersion: v1kind: PersistentVolumeClaimmetadata:  name: harbor-pvc  namespace: harborspec:  accessModes:  - ReadWriteOnce  resources:    requests:      storage: 20GiEOFkubectl create -f harbor-pvc.yaml4、创建PVcat >harbor-pv.yaml<<EOFapiVersion: v1kind: PersistentVolumemetadata:  name: harbor-pv  namespace: harborspec:  capacity:    storage: 20Gi  accessModes:  - ReadWriteOnce  persistentVolumeReclaimPolicy: Recycle  nfs:    path: /data/k8s    server: 192.168.10.101EOFkubectl create -f harbor-pv.yaml5、安装harborhelm install harbor ./harbor-helm-1.4.1 -n harbor6、验证查看podkubectl -n harbor get pod 查看ingresskubectl -n harbor get ing4、安装ingressk8s-master

　　

下载wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml修改配置$ grep -n5 nodeSelector mandatory.yaml212-    spec:213-      hostNetwork: true #添加为host模式214-      # wait up to five minutes for the drain of connections215-      terminationGracePeriodSeconds: 300216-      serviceAccountName: nginx-ingress-serviceaccount217:      nodeSelector:218-        ingress: "true" #替换此处，来决定将ingress部署在哪些机器219-      containers:220-        - name: nginx-ingress-controller221-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0222-          args:k8s-master节点添加labelkubectl label node k8s-master ingress=true安装kubectl create -f mandatory.yaml验证kubectl get ns|grep ingkubectl -n ingress-nginx get pod# 如果ingress不能在master节点上运行 状态一直为pending执行以下命令kubectl taint nodes --all node-role.kubernetes.io/master-5、配置hosts解析并访问web192.168.10.101 core.harbor.domainweb 访问https://core.harbor.domain

　　

　　

6、重点来了！！！重点来了！！！重点来了！！！1、踩坑一：发现登陆不进去看报错4011、错误重现登录不进去

　　

　　

发现pod重启了$ kubectl -n harbor get podNAME                                        READY   STATUS    RESTARTS   AGEharbor-harbor-chartmuseum-7f5d97c8d-k9tst   1/1     Running   0          51mharbor-harbor-core-59b76f46bb-shxqz         1/1     Running   3          51mharbor-harbor-database-0                    1/1     Running   0          51mharbor-harbor-jobservice-5bfb764464-625p6   1/1     Running   3          51mharbor-harbor-portal-59c779dd74-5ffg2       1/1     Running   0          51mharbor-harbor-redis-0                       1/1     Running   0          51mharbor-harbor-registry-6b4b74b7b8-vmlxm     2/2     Running   0          51m2、如何排查问题查看日志$ kubectl -n harbor logs -f harbor-harbor-jobservice-5bfb764464-625p6#提示redis有问题2021-06-18T20:10:31Z : Resend hook event error: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.2021-06-18T20:10:36Z : Resend hook event error: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.再去看redis日志$ kubectl -n harbor logs -f harbor-harbor-redis-0# 提示没有权限1:M 18 Jun 20:11:17.152 # Background saving error1:M 18 Jun 20:11:23.096 * 1 changes in 900 seconds. Saving...1:M 18 Jun 20:11:23.096 * Background saving started by pid 153153:C 18 Jun 20:11:23.097 # Failed opening the RDB file dump.rdb (in server root dir /var/lib/redis) for saving: Permission denied1:M 18 Jun 20:11:23.197 # Background saving error1:M 18 Jun 20:11:29.038 * 1 changes in 900 seconds. Saving...1:M 18 Jun 20:11:29.038 * Background saving started by pid 154154:C 18 Jun 20:11:29.039 # Failed opening the RDB file dump.rdb (in server root dir /var/lib/redis) for saving: Permission denied1:M 18 Jun 20:11:29.139 # Background saving error进入到redis容器中查看$ kubectl -n harbor exec -it harbor-harbor-redis-0 bashredis < ~ >$ iduid=999(redis) gid=999(redis) groups=999(redis),redis < ~ >$ ls -l /var/lib/total 4drwxr-xr-x 2 root root    6 2019-05-10 19:07 colordrwxr-xr-x 2 root root    6 2019-05-10 19:07 locatedrwxr-xr-x 2 root root    6 2019-05-10 19:07 miscdrwxr-xr-x 3 root root   28 2020-06-17 02:43 polkit-1drwxr-xr-x 2 root root    6 2021-06-18 19:41 redisdrwxr-xr-x 1 root root 4096 2020-06-17 02:42 rpm再看看postgresql数据库的权限$ kubectl -n harbor exec -it harbor-harbor-database-0 bashpostgres < / >$ iduid=999(postgres) gid=999(postgres) groups=999(postgres)# postgresql权限没问题postgres < / >$ ls -l /var/lib/total 4drwxr-xr-x 2 root root    6 May 10  2019 colordrwxr-xr-x 2 root root    6 Apr 23  2020 hwclockdrwxr-xr-x 2 root root    6 May 10  2019 locatedrwxr-xr-x 2 root root    6 May 10  2019 miscdrwxr-xr-x 3 root root   28 Jun 17  2020 polkit-1drwxr-xr-x 3 root root   18 Jun 17  2020 postgresqldrwxr-xr-x 1 root root 4096 Jun 17  2020 rpm查看postgresql的权限是如何设置的$ vim harbor-helm-1.4.1/templates/database/database-ss.yaml 35       initContainers: 36       - name: "change-permission-of-directory" 37         image: {{ .Values.database.internal.initContainerImage.repository }}:{{ .Values.database.interna    l.initContainerImage.tag }} 38         imagePullPolicy: {{ .Values.imagePullPolicy }} 39         command: <"/bin/sh"> 40         args: <"-c", "chown -R 999:999 /var/lib/postgresql/data"> 41         volumeMounts: 42         - name: database-data 43           mountPath: /var/lib/postgresql/data 44           subPath: {{ $database.subPath }}再看redis服务配置#没有进行初始化容器 少了给存储目录更改权限 36       containers: 37       - name: redis 38         image: {{ .Values.redis.internal.image.repository }}:{{ .Values.redis.internal.image.tag }} 39         imagePullPolicy: {{ .Values.imagePullPolicy }} 40         livenessProbe: 41           tcpSocket: 42             port: 6379 43           initialDelaySeconds: 300 44           periodSeconds: 10 45         readinessProbe: 46           tcpSocket: 47             port: 6379 48           initialDelaySeconds: 1 49           periodSeconds: 103、如何解决问题把配置给加上（加上36到45行） 33       imagePullSecrets: 34         {{- toYaml . | nindent 8 }} 35       {{- end }} 36       initContainers: 37       - name: "change-permission-of-directory" 38         image: {{ .Values.database.internal.initContainerImage.repository }}:{{ .Values.database.internal.initContainerImage.tag }} 39         imagePullPolicy: {{ .Values.imagePullPolicy }} 40         command: <"/bin/sh"> 41         args: <"-c", "chown -R 999:999 /var/lib/redis"> 42         volumeMounts: 43         - name: data 44           mountPath: /var/lib/redis 45           subPath: {{ $redis.subPath }} 46       containers: 47       - name: redis 48         image: {{ .Values.redis.internal.image.repository }}:{{ .Values.redis.internal.image.tag     }}更新helm upgrade harbor -n harbor ./harbor-helm-1.4.1等待pod重启完毕$ kubectl -n harbor get podNAME                                         READY   STATUS    RESTARTS   AGEharbor-harbor-chartmuseum-6bbdd4578f-lr2p2   1/1     Running   0          4m28sharbor-harbor-core-5b449d7595-hgkqk          1/1     Running   0          4m28sharbor-harbor-database-0                     1/1     Running   0          34mharbor-harbor-jobservice-7cc8ff4b6d-vk7qx    1/1     Running   0          4m28sharbor-harbor-portal-59c779dd74-5ffg2        1/1     Running   0          34mharbor-harbor-redis-0                        1/1     Running   0          3m56sharbor-harbor-registry-587f46865b-b5wd5      2/2     Running   0          4m28s登录测试

　　

ok 登录成功

　　

2、踩坑二：registry组件的镜像存储目录权限导致镜像推送失败1、错误重现k8s-node节点 配置docker仓库cat >/etc/docker/daemon.json<<EOF{  "registry-mirrors" : <    "https://8xpk5wnt.mirror.aliyuncs.com"  >,  "exec-opts" : <"native.cgroupdriver=systemd">,  "insecure-registries":<"192.168.10.101:5000","core.harbor.domain">}EOF重启dockersystemctl restart dockerhosts解析echo '192.168.10.101 core.harbor.domain' >>/etc/hosts登录仓库$ docker login core.harbor.domainUsername: adminPassword: Harbor12345WARNING! Your password will be stored unencrypted in /root/.docker/config.json.Configure a credential helper to remove this warning. Seehttps://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded拉取镜像docker pull nginx:alpine打标签docker tag nginx:alpine core.harbor.domain/library/nginx:alpine推送失败# docker push core.harbor.domain/library/nginx:alpine2、解决registry的镜像存储目录，需要设置registry用户的用户及用户组，不然镜像推送失败

　　

修改配置加入（47到56行）$ vim harbor-helm-1.4.1/templates/registry/registry-dpl.yaml 45         {{- toYaml . | nindent 8 }} 46       {{- end }} 47       initContainers: 48       - name: "change-permission-of-directory" 49         image: {{ .Values.database.internal.initContainerImage.repository }}:{{ .Values.database.internal.initContainerImage.tag }} 50         imagePullPolicy: {{ .Values.imagePullPolicy }} 51         command: <"/bin/sh"> 52         args: <"-c", "chown -R 10000:10000 {{ .Values.persistence.imageChartStorage.filesystem.rootdirectory }}"> 53         volumeMounts: 54         - name: registry-data 55           mountPath: {{ .Values.persistence.imageChartStorage.filesystem.rootdirectory }} 56           subPath: {{ .Values.persistence.persistentVolumeClaim.registry.subPath }} 57       containers: 58       - name: registry更新helm upgrade harbor -n harbor ./harbor-helm-1.4.1再次推送成功$ docker push core.harbor.domain/library/nginx:alpineThe push refers to repository 075508cf8f04: Pushed5c865c78bc96: Pushed134e19b2fac5: Pushed83634f76e732: Pushed766fe2c3fc08: Pushed02c055ef67f5: Pushedalpine: digest: sha256:61191087790c31e43eb37caa10de1135b002f10c09fdda7fa8a5989db74033aa size: 15703、踩坑三：chartmuseum存储目录权限，导致chart推送失败修改配置(47到56行)

　　

$ vim harbor-helm-1.4.1/templates/chartmuseum/chartmuseum-dpl.yaml 44       imagePullSecrets: 45         {{- toYaml . | nindent 8 }} 46       {{- end }} 47       initContainers: 48       - name: "change-permission-of-directory" 49         image: {{ .Values.database.internal.initContainerImage.repository }}:{{ .Values.database.interna    l.initContainerImage.tag }} 50         imagePullPolicy: {{ .Values.imagePullPolicy }} 51         command: <"/bin/sh"> 52         args: <"-c", "chown -R 10000:10000 /chart_storage"> 53         volumeMounts: 54         - name: chartmuseum-data 55           mountPath: /chart_storage 56           subPath: {{ .Values.persistence.persistentVolumeClaim.chartmuseum.subPath }} 57       containers: 58       - name: chartmuseum更新helm upgrade harbor -n harbor ./harbor-helm-1.4.17、推送chart到Harbor仓库helm3默认没有安装helm push插件，需要手动安装。插件地址 https://github.com/chartmuseum/helm-push 安装插件$ helm plugin install https://github.com/chartmuseum/helm-push离线安装：（建议）# mkdir push# tar -xf helm-push_0.8.1_linux_amd64.tar.gz -C push/# helm plugin install ./push# helm plugin ls NAME VERSION DESCRIPTION                      push 0.8.1 Push chart package to ChartMuseum# helm push --helpHelm plugin to push chart package to ChartMuseum添加repo# helm repo add myharbor https://core.harbor.domain/chartrepo/library # x509错误# 添加证书信任，根证书为配置给ingress使用的证书# kubectl -n harbor get secretNAME                           TYPE                                  DATA   AGEdefault-token-qkh7r            kubernetes.io/service-account-token   3      4h9mharbor-harbor-chartmuseum      Opaque                                1      160mharbor-harbor-core             Opaque                                8      160mharbor-harbor-database         Opaque                                1      160mharbor-harbor-ingress          kubernetes.io/tls                     3      160mharbor-harbor-jobservice       Opaque                                2      160mharbor-harbor-registry         Opaque                                3      160msh.helm.release.v1.harbor.v1   helm.sh/release.v1                    1      160msh.helm.release.v1.harbor.v2   helm.sh/release.v1                    1      31m# kubectl -n harbor get secret harbor-harbor-ingress -oyaml# kubectl get secret harbor-harbor-ingress -n harbor -o jsonpath="{.data.ca\.crt}" | base64 -d >harbor.ca.crt# cp harbor.ca.crt /etc/pki/ca-trust/source/anchors$ update-ca-trust enable; update-ca-trust extract# 再次添加# helm repo add myharbor https://core.harbor.domain/chartrepo/library --ca-file=harbor.ca.crt# helm repo ls推送chart到仓库# helm push harbor myharbor --ca-file=harbor.ca.crt -u admin -p Harbor12345

　　

8、登录仓库遇到的问题Error response from daemon: Get https://core.harbor.domain/v2/: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "harbor-ca")解决

　　

$ vim /usr/lib/systemd/system/docker.serviceExecStart=/usr/bin/dockerd --insecure-registry core.harbor.domain#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock