Thursday, 2019-08-29

« Wednesday, 2019-08-28 Index Friday, 2019-08-30 »
openstackgerritMerged airship/treasuremap master: Updates on site authoring and deployment guide  https://review.opendev.org/67861700:23
*** michael-beaver has quit IRC01:19
openstackgerritKaspars Skels proposed airship/treasuremap master: Use MAAS VIP as DNS server for Seaworthy site  https://review.opendev.org/67865201:19
openstackgerritRahul Khiyani proposed airship/airship-in-a-bottle master: Adding calicoctl utility conatiner to airship-in-a-bottle  https://review.opendev.org/67917001:30
openstackgerritRahul Khiyani proposed airship/airship-in-a-bottle master: Adding calicoctl utility conatiner to airship-in-a-bottle  https://review.opendev.org/67917001:35
*** dpawlik has joined #airshipit01:39
*** dpawlik has quit IRC01:43
*** Kuirong has joined #airshipit01:48
*** Kuirong has left #airshipit01:55
*** Kuirong has joined #airshipit01:56
*** AlexNoskov has quit IRC02:27
*** openstackgerrit has quit IRC02:37
*** dpawlik has joined #airshipit03:40
*** dpawlik has quit IRC03:44
*** kskels has quit IRC03:50
*** gkadam has joined #airshipit03:50
*** gkadam has quit IRC03:50
*** rezroo has joined #airshipit04:49
*** dddpak has joined #airshipit05:17
*** jamesgu_ has quit IRC05:56
*** dpawlik has joined #airshipit06:00
*** dr_feelgood has joined #airshipit06:11
*** phasespace has quit IRC06:34
*** LoicL35 has joined #airshipit06:37
*** dr_feelgood has quit IRC06:49
*** dddpak has quit IRC06:49
*** dr_feelgood has joined #airshipit06:51
*** dr_feelgood has quit IRC06:53
*** dr_feelgood has joined #airshipit06:56
*** jsuchome has joined #airshipit06:57
*** kskels has joined #airshipit06:58
*** dr_feelgood has quit IRC07:31
*** dr_feelgood has joined #airshipit07:32
*** dr_feelgood has quit IRC07:52
*** dr_feelgood has joined #airshipit07:53
*** rezroo has quit IRC08:07
*** jamesgu has quit IRC08:23
*** jamesgu has joined #airshipit08:29
*** avolkov has joined #airshipit08:41
*** jithinjoshy has joined #airshipit08:54
jithinjoshyI have installed openstack with airship now unable to launch an instance, getting Error: Failed to perform requested operation on instance "test", the instance has an error status: Please try again later [Error: Build of instance b9cb3895-2226-41b1-978f-788279eb67b0 aborted: Volume 0143730b-0bda-4549-98e6-1046bff396f1 did not finish being created e08:58
jithinjoshyven after we waited 3 seconds or 2 attempts. And its status is error.].08:58
jithinjoshyany idea how to solve it?08:59
*** kskels has quit IRC09:08
dr_feelgoodjithinjoshy: I suggest take a look at cinder logs, seems like you have an issue with creating a volume09:28
jithinjoshyI have tried creating a volume and it too show error09:29
jithinjoshykubectl get pods -o wide --all-namespaces | grep cinderopenstack  cinder-volume-5cd4b79fbd-n6s56                            0/1    Init:CrashLoopBackOff   672    2d    10.97.212.46  airsloop-control-1   <none>09:30
*** openstackgerrit has joined #airshipit09:32
openstackgerritJenkins Uplifter proposed airship/treasuremap master: Auto chart/image uplift to latest  https://review.opendev.org/67921409:32
dr_feelgoodjithinjoshy: this two links below might be helpful09:34
dr_feelgoodhttps://docs.openstack.org/operations-guide/ops-logging.html09:34
dr_feelgoodhttps://airship-shipyard.readthedocs.io/en/latest/CLI.html#logs-commands09:34
jithinjoshyok. thanks for the docs, will check it09:44
jithinjoshyalso, any idea why cinder-volume is showing the status Init:CrashLoopBackOff09:45
jsuchomejithinjoshy: anything interesting under kubectl describe pod ? Or kubectl get events?09:48
jithinjoshyjsuchome: nothing is showing in it09:53
*** dpawlik has quit IRC10:19
*** dpawlik has joined #airshipit10:32
jithinjoshydr_feelgood: i see this error in the log /var/log/syslog10:34
jithinjoshyAug 29 05:30:58 airsloop-control-1 kubelet[2566]: E0829 05:30:58.428951    2566 pod_workers.go:186] Error syncing pod c095aeda-c773-11e9-982a-ac1f6b0efadc ("cinder-volume-5cd4b79fbd-n6s56_openstack(c095aeda-c773-11e9-982a-ac1f6b0efadc)"), skipping: failed to "StartContainer" for "init-cinder-conf" with CrashLoopBackOff: "Back-off 5m0s restarting fa10:34
jithinjoshyiled container=init-cinder-conf pod=cinder-volume-5cd4b79fbd-n6s56_openstack(c095aeda-c773-11e9-982a-ac1f6b0efadc)"10:34
dr_feelgoodjithinjoshy: which version of openstack did you deploy?10:36
dr_feelgoodalso, could you run this command below and share the output10:38
dr_feelgoodkubectl get pods -o wide --all-namespaces | grep CrashLoopBackOff10:38
jithinjoshyopenstack     cinder-volume-5cd4b79fbd-n6s56                                0/1       Init:CrashLoopBackOff   1001       3d        10.97.212.46     airsloop-control-1   <none>10:39
jithinjoshygot the above out put10:39
jithinjoshy# openstack --versionopenstack 3.19.010:39
jithinjoshyopenstack version seems to be 3.19.010:40
dr_feelgoodCrashLoopBackOff means "this means the pod is starting, then crashing, then starting again and crashing again"10:42
dr_feelgoodtry this two also10:42
dr_feelgoodkubectl -n <namespace-name> describe pod <pod name>10:43
dr_feelgoodkubectl -n <namespace-name> logs -p  <pod name>10:43
jithinjoshy# kubectl -n openstack describe pod cinder-volume-5cd4b79fbd-n6s5610:55
jithinjoshyName:               cinder-volume-5cd4b79fbd-n6s56Namespace:          openstackPriority:           0PriorityClassName:  <none>Node:               airsloop-control-1/10.185.167.181Start Time:         Sun, 25 Aug 2019 20:06:38 +0000Labels:             application=cinder                    component=volume                    pod-template-hash=1780635910:55
jithinjoshy68                    release_group=airship-cinderAnnotations:        configmap-bin-hash=c31ba36c37b2f876d0627433b863c80a095aee78bf6ad17e2ae5eb8aca76051c                    configmap-etc-hash=16b040cf0139b4a5c3c781eb50761aa51a62b750972c2aaac71261aa1c96861c                    openstackhelm.openstack.org/release_uuid=Status:             PendingIP:10:55
jithinjoshy              10.97.212.46Controlled By:      ReplicaSet/cinder-volume-5cd4b79fbdInit Containers:  init:    Container ID:  docker://e4e348f900bcb21a05a1e2551ac2fc4eb85b887c5263997c06f601ebd4f6f922    Image:         quay.io/stackanetes/kubernetes-entrypoint:v0.3.1    Image ID:      docker-pullable://quay.io/stackanetes/kubernetes-entrypoint@sha256:310:55
jithinjoshy2b1b657ee4bcc9cc7a1529e31d8e1a06376172373ee020f97f3e78168fde4b6    Port:          <none>    Host Port:     <none>    Command:      kubernetes-entrypoint    State:          Terminated      Reason:       Completed      Exit Code:    0      Started:      Sun, 25 Aug 2019 20:10:19 +0000      Finished:     Sun, 25 Aug 2019 20:14:20 +0000    Ready:10:55
jithinjoshy    True    Restart Count:  0    Environment:      POD_NAME:                    cinder-volume-5cd4b79fbd-n6s56 (v1:metadata.name)      NAMESPACE:                   openstack (v1:metadata.namespace)      INTERFACE_NAME:              eth0      PATH:                        /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/      DEPENDENCY_10:55
jithinjoshySERVICE:          openstack:keystone-api,openstack:cinder-api      DEPENDENCY_JOBS:             cinder-db-sync,cinder-ks-user,cinder-ks-endpoints,cinder-rabbit-init,cinder-storage-init      DEPENDENCY_DAEMONSET:              DEPENDENCY_CONTAINER:              DEPENDENCY_POD_JSON:               DEPENDENCY_CUSTOM_RESOURCE:        COMMAND:10:55
jithinjoshy         echo done    Mounts:      /var/run/secrets/kubernetes.io/serviceaccount from cinder-volume-token-jll4q (ro)  ceph-keyring-placement:    Container ID:  docker://49c81f8a82e88d5bec8e1d3d064ea35167304f4b2b3cc84c5bd1c0e571376a33    Image:         docker.io/openstackhelm/cinder:ocata-ubuntu_xenial    Image ID:      docker-pullable://openstackhe10:55
jithinjoshylm/cinder@sha256:81b4e230800271741d908aed99af82f03ae146bdee8688de2fae003431c51fd2    Port:          <none>    Host Port:     <none>    Command:      /tmp/ceph-keyring.sh    State:          Terminated      Reason:       Completed      Exit Code:    0      Started:      Sun, 25 Aug 2019 20:14:23 +0000      Finished:     Sun, 25 Aug 2019 20:14:23 +00010:55
jithinjoshy0    Ready:          True    Restart Count:  0    Environment:      RBD_USER:  cinder    Mounts:      /etc/ceph from etcceph (rw)      /tmp from pod-tmp (rw)      /tmp/ceph-keyring.sh from cinder-bin (ro)      /tmp/client-keyring from ceph-keyring (ro)      /var/run/secrets/kubernetes.io/serviceaccount from cinder-volume-token-jll4q (ro)  ceph-coor10:55
jithinjoshydination-volume-perms:    Container ID:  docker://4ad1cdd1d106db60d2f48a41da19956c7937a9bb6321993311e33a3b67095b71    Image:         docker.io/openstackhelm/cinder:ocata-ubuntu_xenial    Image ID:      docker-pullable://openstackhelm/cinder@sha256:81b4e230800271741d908aed99af82f03ae146bdee8688de2fae003431c51fd2    Port:          <none>    Host Port10:55
jithinjoshy:     <none>    Command:      chown      -R      cinder:      /var/lib/cinder/coordination    State:          Terminated      Reason:       Completed      Exit Code:    0      Started:      Sun, 25 Aug 2019 20:14:26 +0000      Finished:     Sun, 25 Aug 2019 20:14:26 +0000    Ready:          True    Restart Count:  0    Environment:    <none>    Mou10:55
jithinjoshynts:      /tmp from pod-tmp (rw)      /var/lib/cinder/coordination from cinder-coordination (rw)      /var/run/secrets/kubernetes.io/serviceaccount from cinder-volume-token-jll4q (ro)  init-cinder-conf:    Container ID:  docker://e4e0c5af4e808354105632c0597136059e23e4618bf9e115e57c74941df6245c    Image:         docker.io/openstackhelm/heat:ocata-ub10:55
jithinjoshyuntu_xenial    Image ID:      docker-pullable://openstackhelm/heat@sha256:5bd2cac1b8b9c1028829506b37f36034f94f00bb0fc5417e5e71e56d14fa0c1f    Port:          <none>    Host Port:     <none>    Command:      /tmp/retrieve-internal-tenant.sh    State:          Waiting      Reason:       CrashLoopBackOff    Last State:     Terminated      Reason:10:55
jithinjoshy Error      Exit Code:    1      Started:      Thu, 29 Aug 2019 10:47:38 +0000      Finished:     Thu, 29 Aug 2019 10:47:43 +0000    Ready:          False    Restart Count:  1003    Environment:      OS_IDENTITY_API_VERSION:         3      OS_AUTH_URL:                     <set to the key 'OS_AUTH_URL' in secret 'cinder-keystone-admin'>10:55
jithinjoshyOptional: false      OS_REGION_NAME:                  <set to the key 'OS_REGION_NAME' in secret 'cinder-keystone-admin'>          Optional: false      OS_INTERFACE:                    <set to the key 'OS_INTERFACE' in secret 'cinder-keystone-admin'>            Optional: false      OS_PROJECT_DOMAIN_NAME:          <set to the key 'OS_PROJECT_DOMAIN10:55
jithinjoshy_NAME' in secret 'cinder-keystone-admin'>  Optional: false      OS_PROJECT_NAME:                 <set to the key 'OS_PROJECT_NAME' in secret 'cinder-keystone-admin'>         Optional: false      OS_USER_DOMAIN_NAME:             <set to the key 'OS_USER_DOMAIN_NAME' in secret 'cinder-keystone-admin'>     Optional: false      OS_USERNAME:10:55
jithinjoshy         <set to the key 'OS_USERNAME' in secret 'cinder-keystone-admin'>             Optional: false      OS_PASSWORD:                     <set to the key 'OS_PASSWORD' in secret 'cinder-keystone-admin'>             Optional: false      INTERNAL_PROJECT_NAME:           internal_cinder      INTERNAL_USER_NAME:              internal_cinder      SERV10:55
jithinjoshyICE_OS_REGION_NAME:          <set to the key 'OS_REGION_NAME' in secret 'cinder-keystone-user'>          Optional: false      SERVICE_OS_PROJECT_DOMAIN_NAME:  <set to the key 'OS_PROJECT_DOMAIN_NAME' in secret 'cinder-keystone-user'>  Optional: false      SERVICE_OS_PROJECT_NAME:         <set to the key 'OS_PROJECT_NAME' in secret 'cinder-keystone-10:55
jithinjoshysorry, that was the output I get on running the first command10:58
jithinjoshy# kubectl -n openstack logs -p cinder-volume-5cd4b79fbd-n6s5610:58
jithinjoshyError from server (BadRequest): previous terminated container "cinder-volume" in pod "cinder-volume-5cd4b79fbd-n6s56" not found10:58
jithinjoshyand this is the log10:59
*** dddpak has joined #airshipit11:04
jithinjoshydr_feelgood: I have pasted the full output in11:06
jithinjoshyhttps://pastebin.com/wqZp4Jzj11:06
*** souradage has joined #airshipit12:02
*** Kuirong has quit IRC12:07
*** jithinjoshy has quit IRC12:14
*** dddpak has quit IRC12:31
openstackgerritJagan Mohan Kavva proposed airship/porthole master: Chart/Dockerfile for etcdctl Utility Container  https://review.opendev.org/67468012:45
*** LoicL35 has quit IRC12:54
*** bh526r has joined #airshipit12:55
*** mcriswell has joined #airshipit13:01
*** aaronsheffield has joined #airshipit13:04
dr_feelgoodjithinjoshy: the link doesn't work, at least for me. but you have an error says:  <set to the key 'OS_AUTH_URL' in secret 'cinder-keystone-admin'>13:07
dr_feelgoodI suggest check cinder options on your deployment config13:10
openstackgerritLuna Das proposed airship/porthole master: Add fixes to user logging in calicoctl utility container logs  https://review.opendev.org/67903513:17
*** AlexNoskov has joined #airshipit13:27
*** CobHead has quit IRC13:30
*** kranthikirang has joined #airshipit13:35
*** rezroo has joined #airshipit13:46
openstackgerritIan Pittwood proposed airship/spyglass master: Implement intermediary file validation  https://review.opendev.org/67864213:46
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: Implement intermediary validation  https://review.opendev.org/67864413:51
*** kskels has joined #airshipit13:59
*** nick_kar has quit IRC14:11
*** spilla has joined #airshipit14:20
*** dpawlik has quit IRC14:23
*** dddpak has joined #airshipit14:26
kskelsteam, please have a look at MAAS DNS improvement - this would allow commissioning/deploying nodes in MAAS/Drydock without requiring external/routed network access for PXE interface https://review.opendev.org/#/c/678652/15:02
*** jamesgu_ has joined #airshipit15:03
*** sthussey has joined #airshipit15:22
*** dpawlik has joined #airshipit15:24
*** dpawlik has quit IRC15:28
openstackgerritKaspars Skels proposed airship/treasuremap master: Uplift etcd and coredns images  https://review.opendev.org/67929215:33
openstackgerritKaspars Skels proposed airship/treasuremap master: Uplift etcd and coredns images  https://review.opendev.org/67929215:33
*** dddpak has quit IRC15:36
openstackgerritMerged airship/treasuremap master: Auto chart/image uplift to latest  https://review.opendev.org/67921415:49
openstackgerritKaspars Skels proposed airship/treasuremap master: Uplift etcd and coredns images  https://review.opendev.org/67929216:02
openstackgerritEvgeniy L proposed airship/shipyard master: CI: Update Airskiff full-site manifest location  https://review.opendev.org/67782316:20
openstackgerritMerged airship/shipyard master: Use a direct connectivity to RabbitMQ  https://review.opendev.org/67157516:23
openstackgerritEvgeniy L proposed airship/shipyard master: CI: Update Airskiff full-site manifest location  https://review.opendev.org/67782316:24
*** jsuchome has quit IRC16:26
*** bh526r has quit IRC16:34
*** AlexNoskov has quit IRC16:55
*** mbologna has quit IRC16:56
*** mbologna_ has joined #airshipit16:57
*** alexanderhughes has quit IRC17:01
*** mbologna_ has quit IRC17:03
*** obravo has joined #airshipit17:07
*** mbologna has joined #airshipit17:07
*** dr_feelgood has quit IRC17:45
*** AlexNoskov has joined #airshipit17:53
openstackgerritLuna Das proposed airship/porthole master: Add fixes to user logging in calicoctl utility container logs  https://review.opendev.org/67903518:07
openstackgerritRoy Tang proposed airship/treasuremap master: Updating seaworthy bonding  https://review.opendev.org/67931718:10
*** dpawlik has joined #airshipit18:17
openstackgerritRajeshwari Dharwadkar proposed airship/deckhand master: Fix for opensuse image build issue.  https://review.opendev.org/67870618:18
*** dpawlik has quit IRC18:22
*** dr_feelgood has joined #airshipit18:23
*** dr_feelgood has quit IRC18:27
openstackgerritScott Hussey proposed airship/airship-in-a-bottle master: (multinode) Add resource metric collection  https://review.opendev.org/67673718:40
*** phasespace has joined #airshipit18:58
*** KeithMnemonic has quit IRC19:09
*** obravo has quit IRC19:39
*** obravo has joined #airshipit19:40
openstackgerritRajeshwari Dharwadkar proposed airship/deckhand master: Fix for opensuse image build issue.  https://review.opendev.org/67870619:43
*** obravo has quit IRC19:44
*** souradage has quit IRC20:03
*** timClicks has joined #airshipit20:14
*** dpawlik has joined #airshipit20:18
*** dpawlik has quit IRC20:23
openstackgerritJagan Mohan Kavva proposed airship/airship-in-a-bottle master: Adding etcdctl utility conatiner to airship-in-a-bottle  https://review.opendev.org/67462620:28
openstackgerritJagan Mohan Kavva proposed airship/airship-in-a-bottle master: Adding etcdctl utility conatiner to airship-in-a-bottle  https://review.opendev.org/67462620:29
*** sgrasley has joined #airshipit20:54
*** timClicks has quit IRC21:08
*** spilla has quit IRC21:09
*** GoldenBear has joined #airshipit21:23
openstackgerritRoy Tang proposed airship/treasuremap master: Updating seaworthy bonding  https://review.opendev.org/67931721:26
*** kranthikirang has quit IRC21:35
openstackgerritRoy Tang proposed airship/treasuremap master: Updating seaworthy bonding  https://review.opendev.org/67931722:00
openstackgerritRoy Tang proposed airship/treasuremap master: Updating seaworthy bonding  https://review.opendev.org/67931722:10
*** rdharwadkar has joined #airshipit22:18
*** rdharwadkar has quit IRC22:34
openstackgerritJagan Mohan Kavva proposed airship/airship-in-a-bottle master: Update to work with new version of Promenade  https://review.opendev.org/66442623:08
*** aaronsheffield has quit IRC23:22
*** sthussey has quit IRC23:52
« Wednesday, 2019-08-28 Index Friday, 2019-08-30 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!