转载请注明出处:k8s服务发布错误排查

K8s错误排查步骤是我们使用k8s的必经之路，可以参考如下步骤:

首先查看pod的情况，使用命令：

kubectl get pods

输出如下:

[zzq@localhost zzq]$ kubectl get pods
NAME                          READY     STATUS                  RESTARTS   AGE
report-api-57f64db6c7-6zksv   0/1       Init:0/1                0          1m
report-api-57f64db6c7-mqn7x   0/1       Init:CrashLoopBackOff   2          1m

拿到pod的name，查看详细的情况,使用命令:

kubectl describe pod report-api-57f64db6c7-mqn7x

这里report-api-57f64db6c7-mqn7x跟上面kubectl get pods中的name对应。
输出如下:

[zzq@localhost zzq]$ kubectl describe pod report-api-57f64db6c7-mqn7x
Name:           report-api-57f64db6c7-mqn7x
Namespace:      default
Node:           ip-10-10-133-37.cn-northwest-1.compute.internal/10.10.133.37
Start Time:     Thu, 13 Sep 2018 14:32:10 +0800
Labels:         app=report-apipod-template-hash=1392086273
Annotations:    kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container report-api; cpu request for init container pull-lib
Status:         Pending
IP:             100.96.6.2
Controlled By:  ReplicaSet/report-api-57f64db6c7
Init Containers:pull-lib:Container ID:  docker://41ada0ce00b3c724466abc3f2b945c7e85f59244ac52623ef235216b3adb64f6Image:         anigeo/awscli:latestImage ID:      docker-pullable://anigeo/awscli@sha256:910a18d43a9e936f38313b0dc44fbd7dc25303fab4ea89c4d7b082fefc654c8dPort:          <none>Host Port:     <none>Args:s3cps3://general-data-group/lib/report-api/report-api-1.0.0-SNAPSHOT.jar/jar/State:          WaitingReason:       CrashLoopBackOffLast State:     TerminatedReason:       ErrorExit Code:    1Started:      Thu, 13 Sep 2018 14:35:52 +0800Finished:     Thu, 13 Sep 2018 14:35:52 +0800Ready:          FalseRestart Count:  5Requests:cpu:  100mEnvironment:AWS_DEFAULT_REGION:  cn-northwest-1Mounts:/jar from workdir (rw)/var/run/secrets/kubernetes.io/serviceaccount from default-token-gs59h (ro)
Containers:report-api:Container ID:  Image:         java:8Image ID:      Port:          9999/TCPHost Port:     0/TCPCommand:javaArgs:-jar/jar/report-api-1.0.0-SNAPSHOT.jar--spring.profiles.active=prodState:          WaitingReason:       PodInitializingReady:          FalseRestart Count:  0Requests:cpu:        100mEnvironment:  <none>Mounts:/jar from workdir (rw)/var/run/secrets/kubernetes.io/serviceaccount from default-token-gs59h (ro)
Conditions:Type           StatusInitialized    False Ready          False PodScheduled   True 
Volumes:workdir:Type:    EmptyDir (a temporary directory that shares a pod's lifetime)Medium:  default-token-gs59h:Type:        Secret (a volume populated by a Secret)SecretName:  default-token-gs59hOptional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300snode.kubernetes.io/unreachable:NoExecute for 300s
Events:Type     Reason                 Age              From                                                      Message----     ------                 ----             ----                                                      -------Normal   Scheduled              4m               default-scheduler                                         Successfully assigned report-api-57f64db6c7-mqn7x to ip-10-10-133-37.cn-northwest-1.compute.internalNormal   SuccessfulMountVolume  4m               kubelet, ip-10-10-133-37.cn-northwest-1.compute.internal  MountVolume.SetUp succeeded for volume "workdir"Normal   SuccessfulMountVolume  4m               kubelet, ip-10-10-133-37.cn-northwest-1.compute.internal  MountVolume.SetUp succeeded for volume "default-token-gs59h"Normal   Pulling                3m (x4 over 4m)  kubelet, ip-10-10-133-37.cn-northwest-1.compute.internal  pulling image "anigeo/awscli:latest"Normal   Pulled                 3m (x4 over 4m)  kubelet, ip-10-10-133-37.cn-northwest-1.compute.internal  Successfully pulled image "anigeo/awscli:latest"Normal   Created                3m (x4 over 4m)  kubelet, ip-10-10-133-37.cn-northwest-1.compute.internal  Created containerNormal   Started                3m (x4 over 4m)  kubelet, ip-10-10-133-37.cn-northwest-1.compute.internal  Started containerWarning  BackOff                3m (x7 over 4m)  kubelet, ip-10-10-133-37.cn-northwest-1.compute.internal  Back-off restarting failed container
[zzq@localhost zzq]$ 

仔细看输出发现是在Init Containers的pull-lib阶段报错了 CrashLoopBackOff

 pull-lib:Container ID:  docker://41ada0ce00b3c724466abc3f2b945c7e85f59244ac52623ef235216b3adb64f6Image:         anigeo/awscli:latestImage ID:      docker-pullable://anigeo/awscli@sha256:910a18d43a9e936f38313b0dc44fbd7dc25303fab4ea89c4d7b082fefc654c8dPort:          <none>Host Port:     <none>Args:s3cps3://general-data-group/lib/report-api/report-api-1.0.0-SNAPSHOT.jar/jar/State:          WaitingReason:       CrashLoopBackOffLast State:     TerminatedReason:       ErrorExit Code:    1Started:      Thu, 13 Sep 2018 14:35:52 +0800Finished:     Thu, 13 Sep 2018 14:35:52 +0800

看看代码我们这里要做的从s3获取文件的操作，但是不知道为什么出错。

使用日志查看命令查看pull-lib阶段的情况，使用命令如下:

kubectl logs  report-api-57f64db6c7-mqn7x --container pull-lib

输出如下:

[zzq@localhost zzq]$ kubectl logs  report-api-57f64db6c7-mqn7x --container pull-lib
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
[zzq@localhost zzq]$ 

403禁止访问，说明我们没有s3的权限。

确认修复后重新发布即可。

程序正确启动后查看运行日志 使用命令

kubectl logs  report-api-57f64db6c7-mqn7x -f

进入容器操作shell

kubectl exec -ti report-api-797c974bff-bdxjf -- bash

转载请注明出处:k8s服务发布错误排查