| *** ianychoi has quit IRC | 00:14 | |
| *** ianychoi has joined #openstack-helm | 00:17 | |
| openstackgerrit | Merged openstack/openstack-helm master: Rally: remove unused config template https://review.openstack.org/558057 | 00:20 |
|---|---|---|
| openstackgerrit | Merged openstack/openstack-helm master: Rally: Move all heat templates to be directly values driven. https://review.openstack.org/558058 | 00:22 |
| *** sawblade6 has joined #openstack-helm | 00:56 | |
| *** sawblade_ has quit IRC | 00:58 | |
| *** sawblade_ has joined #openstack-helm | 00:59 | |
| *** sawblade6 has quit IRC | 01:00 | |
| *** sawblade6 has joined #openstack-helm | 01:03 | |
| *** sawblade_ has quit IRC | 01:04 | |
| *** sawblade_ has joined #openstack-helm | 01:06 | |
| *** sawblade6 has quit IRC | 01:08 | |
| *** sawblade6 has joined #openstack-helm | 01:17 | |
| *** sawblade_ has quit IRC | 01:19 | |
| *** sawblade_ has joined #openstack-helm | 01:20 | |
| *** sawblade6 has quit IRC | 01:21 | |
| *** yangyapeng has joined #openstack-helm | 01:21 | |
| *** sawblade6 has joined #openstack-helm | 01:24 | |
| *** sawblade_ has quit IRC | 01:26 | |
| *** sawblad__ has joined #openstack-helm | 01:26 | |
| *** sawblade6 has quit IRC | 01:29 | |
| *** sawblade6 has joined #openstack-helm | 01:32 | |
| *** sawblad__ has quit IRC | 01:35 | |
| *** sawblade_ has joined #openstack-helm | 01:36 | |
| *** sawblade6 has quit IRC | 01:38 | |
| *** sawblade6 has joined #openstack-helm | 01:39 | |
| *** sawblade_ has quit IRC | 01:40 | |
| *** sawblade_ has joined #openstack-helm | 01:42 | |
| *** sawblade6 has quit IRC | 01:44 | |
| *** sawblade_ has quit IRC | 01:47 | |
| *** sawblade6 has joined #openstack-helm | 01:52 | |
| *** sawblade6 has quit IRC | 02:03 | |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Implement nginx sidecar https://review.openstack.org/555386 | 02:04 |
| *** sawblade6 has joined #openstack-helm | 02:05 | |
| *** sawblade_ has joined #openstack-helm | 02:07 | |
| *** sawblade6 has quit IRC | 02:10 | |
| *** sawblade6 has joined #openstack-helm | 02:14 | |
| *** sawblade_ has quit IRC | 02:16 | |
| *** sawblade_ has joined #openstack-helm | 02:18 | |
| *** sawblade6 has quit IRC | 02:19 | |
| *** sawblade6 has joined #openstack-helm | 02:36 | |
| *** sawblade_ has quit IRC | 02:36 | |
| *** sawblade_ has joined #openstack-helm | 02:41 | |
| *** sawblade6 has quit IRC | 02:43 | |
| *** ianychoi has quit IRC | 02:48 | |
| *** sawblade6 has joined #openstack-helm | 02:53 | |
| *** sawblade_ has quit IRC | 02:56 | |
| *** ianychoi has joined #openstack-helm | 02:58 | |
| *** sawblade_ has joined #openstack-helm | 02:59 | |
| *** sawblade6 has quit IRC | 03:01 | |
| *** ianychoi has quit IRC | 03:12 | |
| *** ianychoi has joined #openstack-helm | 03:13 | |
| *** yamamoto has joined #openstack-helm | 03:26 | |
| *** sawblade6 has joined #openstack-helm | 04:20 | |
| *** sawblade_ has quit IRC | 04:22 | |
| *** sawblade_ has joined #openstack-helm | 04:24 | |
| *** sawblade6 has quit IRC | 04:26 | |
| *** ianychoi has quit IRC | 04:35 | |
| *** ianychoi has joined #openstack-helm | 04:36 | |
| *** ianychoi has quit IRC | 05:00 | |
| *** ianychoi has joined #openstack-helm | 05:02 | |
| *** yangyapeng has quit IRC | 05:23 | |
| *** sawblade6 has joined #openstack-helm | 05:35 | |
| *** sawblade_ has quit IRC | 05:37 | |
| *** mdih has joined #openstack-helm | 05:39 | |
| *** sawblade_ has joined #openstack-helm | 05:41 | |
| *** sawblade6 has quit IRC | 05:44 | |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Fix document ref link https://review.openstack.org/558162 | 05:53 |
| *** mdih has quit IRC | 05:54 | |
| *** mdih has joined #openstack-helm | 06:00 | |
| *** yamamoto_ has joined #openstack-helm | 06:18 | |
| *** yamamoto has quit IRC | 06:21 | |
| *** mdih has quit IRC | 06:25 | |
| *** yamamoto has joined #openstack-helm | 06:29 | |
| *** mdih has joined #openstack-helm | 06:30 | |
| *** unicell has joined #openstack-helm | 06:30 | |
| *** yamamoto_ has quit IRC | 06:32 | |
| *** sawblade_ has quit IRC | 06:59 | |
| *** sawblade_ has joined #openstack-helm | 06:59 | |
| *** sawblade6 has joined #openstack-helm | 07:03 | |
| *** sawblade_ has quit IRC | 07:04 | |
| *** unicell has quit IRC | 07:25 | |
| *** sawblade_ has joined #openstack-helm | 07:25 | |
| *** sawblade6 has quit IRC | 07:27 | |
| *** sawblade6 has joined #openstack-helm | 07:48 | |
| *** sawblade_ has quit IRC | 07:51 | |
| *** sawblade_ has joined #openstack-helm | 08:09 | |
| *** sawblade6 has quit IRC | 08:11 | |
| *** yamamoto has quit IRC | 08:51 | |
| *** ianychoi has quit IRC | 08:52 | |
| *** ianychoi has joined #openstack-helm | 08:54 | |
| *** yamamoto has joined #openstack-helm | 09:00 | |
| *** sawblade6 has joined #openstack-helm | 09:04 | |
| *** sawblade_ has quit IRC | 09:06 | |
| *** ianychoi has quit IRC | 09:24 | |
| *** ianychoi has joined #openstack-helm | 09:25 | |
| *** mdih has quit IRC | 09:43 | |
| *** ianychoi has quit IRC | 09:44 | |
| *** ianychoi has joined #openstack-helm | 09:45 | |
| *** goutham has joined #openstack-helm | 10:07 | |
| *** ianychoi has quit IRC | 10:16 | |
| *** ianychoi has joined #openstack-helm | 10:17 | |
| *** ianychoi has quit IRC | 10:31 | |
| *** ianychoi has joined #openstack-helm | 10:38 | |
| *** ianychoi has quit IRC | 10:43 | |
| *** pbourke has quit IRC | 10:46 | |
| *** pbourke has joined #openstack-helm | 10:47 | |
| *** ianychoi has joined #openstack-helm | 10:59 | |
| *** ianychoi has quit IRC | 11:26 | |
| *** ianychoi has joined #openstack-helm | 11:27 | |
| *** sawblade_ has joined #openstack-helm | 11:27 | |
| *** sawblade6 has quit IRC | 11:30 | |
| *** sawblade6 has joined #openstack-helm | 11:52 | |
| *** sawblade_ has quit IRC | 11:56 | |
| *** goutham has quit IRC | 12:08 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Add proxy options to pods https://review.openstack.org/550651 | 12:18 |
| *** sawblade_ has joined #openstack-helm | 12:37 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Corp-Proxy: Add proxy options to helm-toolkit driven pods https://review.openstack.org/550651 | 12:39 |
| *** sawblade6 has quit IRC | 12:39 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Corp-Proxy: Add proxy options to helm-toolkit driven pods https://review.openstack.org/550651 | 12:46 |
| *** julim has quit IRC | 13:31 | |
| *** felipemonteiro__ has joined #openstack-helm | 13:32 | |
| *** julim has joined #openstack-helm | 13:32 | |
| *** yamamoto has quit IRC | 13:37 | |
| *** yamamoto has joined #openstack-helm | 13:37 | |
| *** felipemonteiro_ has joined #openstack-helm | 13:39 | |
| *** felipemonteiro__ has quit IRC | 13:42 | |
| *** sawblade6 has joined #openstack-helm | 13:50 | |
| *** evrardjp has quit IRC | 13:52 | |
| *** yangyapeng has joined #openstack-helm | 13:52 | |
| *** sawblade_ has quit IRC | 13:52 | |
| *** yangyapeng has quit IRC | 13:56 | |
| *** sawblade_ has joined #openstack-helm | 14:04 | |
| *** sawblade6 has quit IRC | 14:07 | |
| openstackgerrit | Merged openstack/openstack-helm master: Glance: Move all config to be directly values driven. https://review.openstack.org/557023 | 14:08 |
| openstackgerrit | Merged openstack/openstack-helm master: Gnocchi: move to use templater for apache config https://review.openstack.org/557881 | 14:08 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Implement nginx sidecar https://review.openstack.org/555386 | 14:23 |
| openstackgerrit | Merged openstack/openstack-helm master: Ironic: Move all config to be directly values driven. https://review.openstack.org/557882 | 14:28 |
| openstackgerrit | Merged openstack/openstack-helm master: Fix document ref link https://review.openstack.org/558162 | 14:29 |
| *** yangyapeng has joined #openstack-helm | 14:29 | |
| *** sawblade_ has quit IRC | 14:31 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Nagios: Configure ports with endpoint port lookups https://review.openstack.org/558212 | 14:31 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Nagios: Configure ports with endpoint port lookups https://review.openstack.org/558212 | 14:32 |
| *** sawblade6 has joined #openstack-helm | 14:34 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Ceph: Make mon deployment compatible with k8s >= 1.10 https://review.openstack.org/558214 | 14:39 |
| *** sawblade_ has joined #openstack-helm | 14:45 | |
| *** sawblade6 has quit IRC | 14:48 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: K8s: Move version in kubeadm gate to 1.9.4 https://review.openstack.org/552552 | 14:49 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Implement nginx sidecar https://review.openstack.org/555386 | 14:59 |
| *** sawblade6 has joined #openstack-helm | 15:04 | |
| *** sawblade_ has quit IRC | 15:06 | |
| openstackgerrit | Stacey Fletcher proposed openstack/openstack-helm-addons master: Jenkins chart https://review.openstack.org/544780 | 15:20 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Neutron: Update daemonset labels and mixins https://review.openstack.org/555461 | 15:31 |
| openstackgerrit | Stacey Fletcher proposed openstack/openstack-helm-addons master: Jenkins chart https://review.openstack.org/544780 | 15:33 |
| *** ianychoi has quit IRC | 15:34 | |
| openstackgerrit | Merged openstack/openstack-helm-infra master: Grafana dashboard for Nginx using nginx-vts metrics https://review.openstack.org/541396 | 15:36 |
| *** felipemonteiro_ has quit IRC | 15:50 | |
| *** felipemonteiro has joined #openstack-helm | 15:55 | |
| openstackgerrit | Stacey Fletcher proposed openstack/openstack-helm-addons master: Jenkins chart https://review.openstack.org/544780 | 15:56 |
| *** felipemonteiro_ has joined #openstack-helm | 15:57 | |
| *** yangyapeng has quit IRC | 15:58 | |
| *** ianychoi has joined #openstack-helm | 15:58 | |
| *** felipemonteiro has quit IRC | 16:01 | |
| *** ianychoi has quit IRC | 16:06 | |
| *** ianychoi has joined #openstack-helm | 16:07 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Ceph: Make mon deployment compatible with k8s >= 1.10 https://review.openstack.org/558214 | 16:12 |
| *** felipemonteiro_ has quit IRC | 16:29 | |
| *** felipemonteiro__ has joined #openstack-helm | 16:29 | |
| *** yamamoto has quit IRC | 16:33 | |
| openstackgerrit | Rakesh Patnaik proposed openstack/openstack-helm-infra master: Alert rules in prometheus to support nagios based monitoring via alert metric queries https://review.openstack.org/555700 | 16:33 |
| openstackgerrit | Rakesh Patnaik proposed openstack/openstack-helm-infra master: Alert rules in prometheus to support nagios based monitoring via alert metric queries https://review.openstack.org/555700 | 16:35 |
| *** yamamoto has joined #openstack-helm | 16:42 | |
| openstackgerrit | Stacey Fletcher proposed openstack/openstack-helm-addons master: Jenkins chart https://review.openstack.org/544780 | 16:57 |
| openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implement Ceph-Mon component chart https://review.openstack.org/558011 | 16:58 |
| *** yamamoto has quit IRC | 16:59 | |
| openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implement Ceph-Mon component chart https://review.openstack.org/558011 | 17:10 |
| *** felipemonteiro_ has joined #openstack-helm | 17:18 | |
| *** felipemonteiro__ has quit IRC | 17:22 | |
| *** sawblade_ has joined #openstack-helm | 17:36 | |
| *** sawblade6 has quit IRC | 17:39 | |
| *** felipemonteiro_ has quit IRC | 17:41 | |
| *** felipemonteiro_ has joined #openstack-helm | 17:41 | |
| *** sawblade6 has joined #openstack-helm | 17:44 | |
| *** sawblade_ has quit IRC | 17:46 | |
| openstackgerrit | Rakesh Patnaik proposed openstack/openstack-helm-infra master: Nagios chart modifications to use prometheus alert metric for monitoring https://review.openstack.org/555731 | 17:46 |
| *** yamamoto has joined #openstack-helm | 18:00 | |
| *** yamamoto has quit IRC | 18:06 | |
| openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implement Ceph-Mon component chart https://review.openstack.org/558011 | 18:09 |
| *** unicell has joined #openstack-helm | 18:13 | |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: Adding TLS support for sidecar https://review.openstack.org/558248 | 18:30 |
| *** yamamoto has joined #openstack-helm | 18:37 | |
| *** yamamoto has quit IRC | 18:45 | |
| openstackgerrit | Chris Wedgwood proposed openstack/openstack-helm master: RabbitMQ: only request 256Mi for PVC by default https://review.openstack.org/558258 | 18:59 |
| *** yamamoto has joined #openstack-helm | 19:45 | |
| *** yamamoto has quit IRC | 19:51 | |
| *** clarkb has joined #openstack-helm | 20:23 | |
| clarkb | hello, I'm debugging a new cloud that the infra team has turned on for test resources and the helm multinode fedora job has failed a couple times. Looks like it is failing in other clouds consistently too so I don't think it is the new cloud at fault. However I notice that it uses 5 nodes per run and is failing consistently. Maybe this is a good candidate for being an experimental job so that it is | 20:24 |
| clarkb | only run on demand while correcting the job? then can move back to check when it is reliable | 20:24 |
| v1k0d3n | mattmceuen ^^ | 20:28 |
| clarkb | also after a quick digging it appears that nova db sync is failing due to unsafe migrations? http://logs.openstack.org/86/555386/19/check/openstack-helm-multinode-fedora/869cd12/primary/pod-logs/openstack/nova-db-sync-9n4kz/nova-db-sync.txt | 20:32 |
| clarkb | but maybe thats a non issue and its elsewhere | 20:33 |
| portdirect | clarkb: non-issue | 20:33 |
| portdirect | but we do need to spend some time resolving whats going on here | 20:34 |
| openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implement Ceph-Mon component chart https://review.openstack.org/558011 | 20:34 |
| portdirect | many of our multinode gates are suffering from sporadic failures | 20:34 |
| portdirect | I'm tempted to blame the vm's we provision on - as they seem non-deterministic in nature :( | 20:34 |
| portdirect | but I'd be up for moving some of them to check-experimental untill we determine root cause | 20:35 |
| portdirect | would it be possible to spend some time with you to triage? | 20:35 |
| portdirect | as I'm sure your insight would be invaluable here | 20:36 |
| clarkb | portdirect: can you provide more info on what sort of non determinism you see? they are not all identical because they come from different cloud providers across the world | 20:37 |
| clarkb | but they should be relatively deterministic within each provider and the image we boot on all of them is the same | 20:37 |
| clarkb | but the cpus and disks and partition sizes and so on are all going to vary | 20:37 |
| portdirect | I suspect iops? | 20:38 |
| portdirect | as most come from timeouts of various sorts | 20:38 |
| portdirect | but I'll compile a summary and ping you if thats ok? | 20:38 |
| clarkb | sure | 20:38 |
| mattmceuen | anticw was also looking specifically into the fedora failures and may have data to contribute too | 20:39 |
| mattmceuen | rather that was centos not fedora - nm anticw :) | 20:40 |
| portdirect | yeah - I'm less worried about fedora - but more all the gates :( | 20:40 |
| portdirect | we recently disabled ubuntu as it was consistently failing on the gate, though passed the check everytime... | 20:40 |
| *** sawblade_ has joined #openstack-helm | 20:41 | |
| clarkb | looking at your logs http://logs.openstack.org/14/558214/2/check/openstack-helm-multinode-fedora/090b77f/job-output.txt.gz#_2018-04-02_18_12_30_414649 says that nova placement api didn't come up. Reading its logs it was blocking on nova db sync for another 4 minutes after that before it completed | 20:42 |
| clarkb | so at a high level its things not completing quickly enough? | 20:42 |
| portdirect | yes | 20:42 |
| *** sawblade6 has quit IRC | 20:43 | |
| clarkb | diggin further http://logs.openstack.org/14/558214/2/check/openstack-helm-multinode-fedora/090b77f/primary/pod-logs/openstack/nova-db-sync-d7lwm/nova-db-sync.txt starts 18 minutes after its dep said it was done in http://logs.openstack.org/14/558214/2/check/openstack-helm-multinode-fedora/090b77f/primary/pod-logs/openstack/nova-db-sync-d7lwm/init.txt is there any way to account for that time? | 20:44 |
| portdirect | that was to bring up the sync | 20:45 |
| portdirect | here we see the error you reported: http://logs.openstack.org/14/558214/2/check/openstack-helm-multinode-fedora/090b77f/primary/pod-logs/openstack/nova-db-sync-d7lwm/nova-db-sync.txt | 20:45 |
| clarkb | portdirect: right but if you look at the timestamps the sync itself doesn't start for another 18 minutes later | 20:45 |
| clarkb | portdirect: and by then its past where the job has failed | 20:46 |
| clarkb | so I'm trying to understand where that 18 minutes went | 20:46 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: Adding TLS support for sidecar https://review.openstack.org/558248 | 20:46 |
| portdirect | hmm | 20:46 |
| portdirect | I think getting the kublet logs would help here | 20:47 |
| *** yamamoto has joined #openstack-helm | 20:47 | |
| portdirect | as i see your point | 20:47 |
| portdirect | http://logs.openstack.org/14/558214/2/check/openstack-helm-multinode-fedora/090b77f/primary/objects/namespaced/openstack/pods/nova-db-sync-d7lwm.txt | 20:48 |
| portdirect | looks like it may have failed a few times | 20:48 |
| portdirect | I'll try and get what i can here | 20:48 |
| portdirect | brb - getting called away | 20:48 |
| clarkb | no rush, but ya looks liek it ate most of that 18 minutes then failed, then tried again and succeeded much more quickly but by then job was over | 20:50 |
| clarkb | fwiw we currently cache docker images from docker hub but I don't think we do that for quay.io so that might be another thing to look into as far as making this more reliable | 20:50 |
| clarkb | and I'm guessing we don't grab the logs for the failed half? I'll let you figure that out and help debug from there | 20:51 |
| *** yamamoto has quit IRC | 20:52 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: Gate: Add heat template to bring up a KubeADM cluster with OSH-Infra https://review.openstack.org/550684 | 21:01 |
| anticw | mattmceuen: sorry, i just saw this now ... i didn't look closely at fedora but can later on | 21:09 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: Adding TLS support for sidecar https://review.openstack.org/558248 | 21:27 |
| openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implement Ceph-Mon component chart https://review.openstack.org/558011 | 21:31 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: Gate: Add heat template to bring up a KubeADM cluster with OSH-Infra https://review.openstack.org/550684 | 21:47 |
| *** yamamoto has joined #openstack-helm | 21:48 | |
| *** yamamoto has quit IRC | 21:54 | |
| openstackgerrit | Renis Makadia proposed openstack/openstack-helm master: WIP: Implement Ceph-Mon component chart https://review.openstack.org/558011 | 22:04 |
| *** felipemonteiro_ has quit IRC | 22:39 | |
| *** yamamoto has joined #openstack-helm | 22:50 | |
| *** yamamoto has quit IRC | 22:56 | |
| *** yamamoto has joined #openstack-helm | 23:52 | |
| *** yamamoto has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!